Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

random access/seek support #18

Open
jrevels opened this issue Dec 26, 2019 · 1 comment
Open

random access/seek support #18

jrevels opened this issue Dec 26, 2019 · 1 comment

Comments

@jrevels
Copy link

jrevels commented Dec 26, 2019

ref facebook/zstd#395 (comment)
ref https://discourse.julialang.org/t/ann-onda-jl-a-format-for-multi-sensor-multi-channel-lpcm-encodable-recordings/32650/3

I'm not sure if the upstream zstd's seek support is considered stable enough to support here, or what the most desirable interface would be from a TranscodingStreams perspective, but it would definitely useful if this feature was useable with CodecZstd.

Even if the upstream seekable format is still considered experimental, it may still be worthwhile to support here; the referenced thread makes it seem like more downstream usage of the experimental API would motivate further development/codification of the seekable format.

@mkitti
Copy link
Member

mkitti commented Jun 2, 2024

My current thinking here is the level to implement seekability on frames rather than blocks.

Through a combination of ZSTD_findDecompressedSize and ZSTD_getFrameContentSize we can get the total decompressed size and the decompressed size of individual frames, if saved such as via our ZstdFrameCompressor.

To seek, we iterate through frames. For each frame

  1. Check if the frame is skippable or not via ZSTD_isSkippableFrame
  2. Get the decompressed size via ZSTD_getFrameContentSize
  3. Get the size of the frame ZSTD_findFrameCompressedSize
  4. If the data we seek is not within the decompressed size, continue to the next frame by advancing by the frame compressed size

To better support random access, especially remotely, we could encode a directory into a skippable frame at the beginning or end of the stream.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants