Getting error when reading LZ4 compressed files from R

[xiaodaigh]

I use R's fst to compress vectors into LZ4.

But I get the below when I try to fnl_res = transcode(LZ4Decompressor, compressed_chunk) reading the README and this thread python-lz4/python-lz4#143 indicates that CodecLz4.jl uses the Frame format but not the standard LZ4 or LZ4_HC formats.

Is there any possibility of adding support for them?

ERROR: LZ4F_decompress: ERROR_frameType_unknown                       
Stacktrace:                                                           
 [1] transcode(::LZ4Decompressor, ::Array{UInt8,1}) at C:\Users\RTX2080\.julia\packages\TranscodingStreams\MsN8d\src\transcode.jl:121       
 [2] transcode(::Type{LZ4Decompressor}, ::Array{UInt8,1}) at C:\Users\RTX2080\.julia\packages\TranscodingStreams\MsN8d\src\transcode.jl:34  
 [3] uncompress_cfst(::String) at c:\scratch\fst.jl:32                
 [4] top-level scope at REPL[60]:1   

[morris25]

I believe LZ4 and LZ4_HC formats are supported through Blosc.jl.

We can consider adding support for those formats here as future work if there is demand for it.

[xiaodaigh]

Just tested it. It doesn't work. Not straight out of the box anyway.

https://discourse.julialang.org/t/can-blosc-jl-only-decompress-content-that-it-has-compressed/31141

[morris25]

I might be able to find some time next week to look into implementing the other formats here. I'll get back to you then.

[morris25]

Sorry I didn't get back to you sooner. I've resurrected the header files from an earlier version of this package, but I still need to test them and get them wrapped up as TranscodingStreams. Hopefully I'll have a working version up in the next week or so.

[morris25]

I've got a couple of the other functions working #26. If I've read the R code correctly, lz4_decompress should work for decoding the compressed vectors.

Let me know if that works for you.

[xiaodaigh]

I still the same error. To produce

1. Run the R code to generate data

xc = sample(10:20, 1e6, replace = TRUE)
x <- writeBin(xc, raw())

xc1= qs::lz4_compress_raw(x, 1)
writeBin(xc1, "c:/data/ok.io")

# R works
qs::lz4_decompress_raw(xc1)

and then in 2) Julia do and all of the options I tried failed

a = open("c:/data/ok.io", "r")
ar = read(a)
hehe = transcode(LZ4FrameDecompressor, ar) # fails
hehe = transcode(LZ4SafeDecompressor, ar) # fails
hehe = lz4_decompress(ar) #fails

Any ideas?