Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: infer unknown lengths from context in from_buffers #2732

Merged
merged 7 commits into from
Oct 3, 2023

Conversation

agoose77
Copy link
Collaborator

@agoose77 agoose77 commented Oct 2, 2023

This PR follows up from #2718 by relaxing some rules:

  • Placeholder arrays no longer need known lengths if from_buffers is able to deduce the length.
    This requirement was originally made for safety reasons — if something went wrong during rehydration etc., we would see placeholders without known lengths in contexts where they should be known. However, by imposing this restriction, the caller would need to recreate much of from_buffers to compute lengths from offset buffers. Thus, we relax this rule, and permit from_buffers to update placeholders (via a new placeholder) with new known lengths.
  • Non-placeholder arrays can be encountered in contexts where the outer length is not known.
    Originally we wanted this to be an error so that we could clearly identify strange touching behaviour. However, in general it should be possible to refer to the same buffer multiple times within a form, so we should remove this rule.

@agoose77 agoose77 temporarily deployed to docs-preview October 2, 2023 12:53 — with GitHub Actions Inactive
@codecov
Copy link

codecov bot commented Oct 2, 2023

Codecov Report

Merging #2732 (d3038d2) into main (6dc52cc) will increase coverage by 0.00%.
Report is 1 commits behind head on main.
The diff coverage is 72.41%.

Additional details and impacted files
Files Coverage Δ
src/awkward/_nplikes/typetracer.py 76.60% <ø> (-0.03%) ⬇️
src/awkward/contents/bytemaskedarray.py 89.07% <ø> (ø)
src/awkward/contents/listarray.py 88.38% <100.00%> (ø)
src/awkward/contents/listoffsetarray.py 82.77% <100.00%> (ø)
src/awkward/operations/ak_from_buffers.py 94.07% <100.00%> (-0.08%) ⬇️
src/awkward/contents/bitmaskedarray.py 69.15% <50.00%> (-0.10%) ⬇️
src/awkward/contents/unionarray.py 85.57% <50.00%> (ø)
src/awkward/contents/regulararray.py 85.14% <50.00%> (+0.09%) ⬆️
src/awkward/contents/recordarray.py 84.55% <76.47%> (-0.28%) ⬇️

... and 1 file with indirect coverage changes

@agoose77 agoose77 temporarily deployed to docs-preview October 2, 2023 13:38 — with GitHub Actions Inactive
@agoose77 agoose77 marked this pull request as ready for review October 3, 2023 14:26
@agoose77 agoose77 requested a review from jpivarski October 3, 2023 14:35
@agoose77 agoose77 temporarily deployed to docs-preview October 3, 2023 14:52 — with GitHub Actions Inactive
Copy link
Member

@jpivarski jpivarski left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks very sensible. And it's a good procedure for strict rules during early development to be relaxed later as the system gets more understood.

@agoose77 agoose77 merged commit 626ff08 into main Oct 3, 2023
37 of 38 checks passed
@agoose77 agoose77 deleted the agoose77/fix-from-buffers-placeholders-part-2 branch October 3, 2023 16:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants