Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Data larger than 4194304 Bytes is not decompressed correctly #203

Open
mgrundie-r7 opened this issue Jan 17, 2023 · 2 comments
Open

Data larger than 4194304 Bytes is not decompressed correctly #203

mgrundie-r7 opened this issue Jan 17, 2023 · 2 comments

Comments

@mgrundie-r7
Copy link

    Random random = new Random();
    data = new byte[4194304];
    random.nextBytes(data);

    ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();

    LZ4FrameOutputStream lz4FrameOutputStream = new LZ4FrameOutputStream(byteArrayOutputStream);
    lz4FrameOutputStream.write(data);
    lz4FrameOutputStream.close();

    byte[] compressedBytes = byteArrayOutputStream.toByteArray();
    LZ4FrameInputStream compressedInputStream = new LZ4FrameInputStream(new ByteArrayInputStream(compressedBytes));

    byte[] decompressedData = new byte[data.length];
    compressedInputStream.read(decompressedData);
    compressedInputStream.close();

    assertTrue(Arrays.equals(data, decompressedData));

The above assertion passes. If you update the code to data = new byte[4194305]; then the assertion fails and the 4194305th element of decompressedData (decompressedData[4194304]) will equal 0

@petoncle
Copy link

petoncle commented Feb 13, 2023

That's because you are decompressing only one block (you should call compressedInputStream .read() more than once).
Instead of:

byte[] decompressedData = new byte[data.length];
compressedInputStream.read(decompressedData);

Try:

byte[] decompressedData = compressedInputStream.readAllBytes();

@mgrundie-r7
Copy link
Author

I can't control what the callers do and the callers have no specific knowledge which type of InputStream this is. I ended up writing an adapter that extends InputStream and delegates to LZ4FrameInputStream to add some reset capability and to chunk read when read is asked to read > 4194304

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants