Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use LazyRead instead of read_bytes #185

Closed
wants to merge 3 commits into from
Closed

Conversation

blegat
Copy link
Contributor

@blegat blegat commented Aug 27, 2021

When reading andora.osm.pbf with OpenStreetMapX (using pszufe/OpenStreetMapX.jl#52), the timing is:
Before this PR:

julia> @time OpenStreetMapX.parsePBF("/home/blegat/Downloads/andorra-latest.osm.pbf");
  0.266550 seconds (578.92 k allocations: 134.372 MiB, 16.38% gc time)

After this PR:

julia> @time OpenStreetMapX.parsePBF("/home/blegat/Downloads/andorra-latest.osm.pbf");
  0.226363 seconds (439.89 k allocations: 111.927 MiB)

read_bytes makes a copy of the bytes, even nested ones if an object contains another object or a packed vector, ...
With this PR, we make instead a wrapper type that virtually limits the size of the IO.
As this can be called recursively, this could create a stack of LazyRead wrappers so we take care of avoiding this and just update the LazyRead instead.

@Drvi
Copy link
Member

Drvi commented Aug 17, 2022

Closing as we overhauled the whole package. I invite you to try the new version out, it should be faster.

@Drvi Drvi closed this Aug 17, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants