Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Does streaming not work with compressed files? #19

Open
synthesizerpatel opened this issue Apr 25, 2024 · 0 comments
Open

Does streaming not work with compressed files? #19

synthesizerpatel opened this issue Apr 25, 2024 · 0 comments

Comments

@synthesizerpatel
Copy link

First off, wonderful module, thank you for your work!

I'm wondering if the C code nature of this module makes it too low level to work with compressed stream objects?

With the latest version of your code I try

def parser(file):
    with bz2.open(file, 'rb') as bzfh:
        for jstonstr in splitfile(bzfh, format="json"):
            print(jsonstr)
            yield json.loads(jsonstr)

When I run it, I would expect jsonstr to be a string of one complete JSON statement {...} , but it looks like it the compressed stream of data?

I'm was hoping to open bz2 compressed compact JSON files with many records back to back {..}{..}{..}{..} and process them.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant