Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support reading from other file systems like HDFS or S3 #51

Open
mzyoussef opened this issue Apr 5, 2020 · 1 comment
Open

Support reading from other file systems like HDFS or S3 #51

mzyoussef opened this issue Apr 5, 2020 · 1 comment

Comments

@mzyoussef
Copy link

is there any chance that BagFile.getChannel return ReadableByteChannel instead of FileChannel to support reading bag files from HDFS or S3 direct?

Currently, we have to download the bag file to the local file system or cache it into memory, so it is an overhead, any idea that we can read from the stream directly?

@pjreed
Copy link
Contributor

pjreed commented Apr 6, 2020

This would be a little more complex than just returning a ReadableByteChannel; the bag reader expects to have access to a SeekableByteChannel interface so that it can have random access to different parts of the file in order to avoid reading the entire file into memory at once.

I'm not very familiar with using HDFS or S3, do you know if there are any examples or libraries for doing that kind of thing?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants