Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Chunks do not align in dimension 1 #35

Open
rafaqz opened this issue Jun 5, 2021 · 2 comments
Open

Chunks do not align in dimension 1 #35

rafaqz opened this issue Jun 5, 2021 · 2 comments

Comments

@rafaqz
Copy link
Collaborator

rafaqz commented Jun 5, 2021

Can we work around this? This being an error is a problem for using chunked broadcast as a default.

I'm not totally across the underlying mechanisms yet, but I can imagine 2 possibilities:

  • force chunk alignment by loading multiple chunks to a temporary chunk of the right size
  • load the whole array to memory and ignore chunking, especially if the array is small

Then the error could be a warning.

@meggart
Copy link
Collaborator

meggart commented Jan 13, 2022

There are certainly ways to work around this, but I don't know how to do it with the current broadcast syntax. Ideally one would force the broadcast operation to use the chunks of the array with the highest access latency, and usually the user should know which one this is. We could say that in case where chunks do not align, we just give preference to the first array in the broadcast operation and document that users should always do: largearray .+ smallarray instead of smallarray .+ largearray?

@rafaqz
Copy link
Collaborator Author

rafaqz commented Jan 13, 2022

Yeah something like that could work. And we just use the largest possible chunks of the second? So some duplication of chunk loads but maybe not terrible

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants