Parallel compression?

[mitar]

What about support for parallel compression for bzip2?

[cosnicolaou]

the linux pbzip2 utility already supports parallel compression/decompression using multiple streams. I created this package to deal with the case when pbzip2 is not used and there is a single large stream. This turns out to be the common case for downloads since most people don't think to use pbzip2. Another snag is that the go standard library lacks a bzip2 compressor and there seems to be no urgency to adding one. All in all I personally don't have any plans to work on this in the immediate or mid term future. Is your use case for minio or something else?

[mitar]

The issue is that it would be nice to be able to generate bzip2 using Go itself, not using an external tool (e.g., if you are producing data using Go itself). Ideally in parallel way.

No, I am not affiliated with minio. I am working on some Wikidata/Wikipedia data processing.

[cosnicolaou]

I see, unfortunately there don't appear to be any native golang implementations of bzip2 and little prospect of one appearing in the standard library any time soon judging by this: golang/go#4828. There is this https://github.com/larzconwell/bzip2 but I've not looked at it to know if it's usable or not.

[klauspost]

https://github.com/dsnet/compress/tree/master/bzip2 also implements a writer.