Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

validate should check indexes in parallel #144

Closed
sourcefrog opened this issue Aug 11, 2020 · 0 comments
Closed

validate should check indexes in parallel #144

sourcefrog opened this issue Aug 11, 2020 · 0 comments

Comments

@sourcefrog
Copy link
Owner

sourcefrog commented Aug 11, 2020

Validate currently checks indexes purely sequentially, but it would be faster to parallelize:

  1. check multiple indexes in parallel
  2. check multiple hunks from inside each index in parallel

In principle we could also parallelize checking different entries from inside a hunk, but this is probably not useful. Most of the time is spent getting the hunk from disk, decompressing, and deserializing it, and the per-entry work is very small.

Probably this is better to do after #141, the fix for which involves changing index validation to return a set of referenced addresses.


Checking multiple hunks inside an index in parallel is possible, but a little more complicated because we also want to check ordering constraints between hunks, so I deferred that to #145.

sourcefrog pushed a commit that referenced this issue Aug 13, 2020
Don't stop early if there are problems in the index. Just count the
problems.

Fixes #144
sourcefrog added a commit that referenced this issue Aug 13, 2020
Don't stop early if there are problems in the index. Just count the
problems.

Fixes #144
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant