Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use concrete types instead of dictionaries #52

Merged
merged 3 commits into from
Aug 29, 2021

Conversation

blegat
Copy link
Contributor

@blegat blegat commented Aug 27, 2021

As discussed in #44 (comment) and JuliaIO/ProtoBuf.jl#179, creating the small dictionaries can be a bottleneck when reading many small structs.
I assume that for DenseNodes, it it's ok since it's one bit struct but for reading all the small Ways and Relations, it can be quite noticeable.
As the timings depends on the % of the GC time, I give a few different runs (I know, I should use BenchmarkTools ^^)

Currently:

julia> @time OpenStreetMapX.parsePBF("/home/blegat/Downloads/andorra-latest.osm.pbf");
  0.819850 seconds (1.58 M allocations: 183.134 MiB, 17.31% gc time)

After JuliaIO/ProtoBuf.jl#184:

julia> @time OpenStreetMapX.parsePBF("/home/blegat/Downloads/andorra-latest.osm.pbf");
  0.358973 seconds (1.14 M allocations: 176.507 MiB, 16.71% gc time)

julia> @time OpenStreetMapX.parsePBF("/home/blegat/Downloads/andorra-latest.osm.pbf");
  0.312807 seconds (1.14 M allocations: 176.506 MiB, 5.27% gc time)

After this PR:

julia> @time OpenStreetMapX.parsePBF("/home/blegat/Downloads/andorra-latest.osm.pbf");
  0.743238 seconds (1.01 M allocations: 141.000 MiB, 20.39% gc time)

julia> @time OpenStreetMapX.parsePBF("/home/blegat/Downloads/andorra-latest.osm.pbf");
  0.669973 seconds (1.01 M allocations: 141.000 MiB, 12.62% gc time)

julia> @time OpenStreetMapX.parsePBF("/home/blegat/Downloads/andorra-latest.osm.pbf");
  0.626401 seconds (1.01 M allocations: 141.000 MiB, 6.59% gc time)

After both PR:

julia> @time OpenStreetMapX.parsePBF("/home/blegat/Downloads/andorra-latest.osm.pbf");
  0.266550 seconds (578.92 k allocations: 134.372 MiB, 16.38% gc time)

src/OSMPBF.jl Outdated Show resolved Hide resolved
@blegat blegat force-pushed the concrete_protobuf branch from 5909097 to 289bd53 Compare August 28, 2021 08:44
@pszufe pszufe merged commit 87ce7e4 into pszufe:master Aug 29, 2021
@pszufe
Copy link
Owner

pszufe commented Aug 29, 2021

Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants