Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sort out domains #20

Open
3 tasks
DPeterK opened this issue Feb 26, 2020 · 0 comments
Open
3 tasks

Sort out domains #20

DPeterK opened this issue Feb 26, 2020 · 0 comments

Comments

@DPeterK
Copy link
Contributor

DPeterK commented Feb 26, 2020

The multi-attr tiledb writer currently uses a different set of domains to what is calculated by the data model. The data model provides only the 'super-domains' (the minimum set of highest-dimensionality domains that enclose the maximum number of input datasets), but the multi-attr writer makes one domain for each unique set of dimensions and writes all the datasets that match that set of dimensions.

For example, taking the following datasets:

a --> [x, y, z, t]
b --> [x, y, t]
c --> [x, y, t]
d --> [x, y, t1]
e --> [x1, y1, z, t1]
f --> [x1, y1, t1]
g --> [x, y, z, t]

This is the set of domains that would be made by the data model:

x,y,z,t --> domain_0
x1,y1,z,t1 --> domain_1
x,y,t1 --> domain_2
domain_0 --> a, b, c, g
domain_1 --> e, f
domain_2 --> d

And this is the set of domains that would be made by the multi-attr writer:

x,y,z,t --> a, g
x,y,t --> b, c
x,y,t1 --> d
x1,y1,z,t1 --> e
x1,y1,t1 --> f

Some time we should tidy this discrepancy. Assuming that multi-attr append goes in (see #19) then the multi-attr case should become the default, and the data model domain assignation algorithm should just be updated to match what the multi-attr writer is doing.


Here's the TODO list:

  • decide on a single writing strategy - potentially prefer multi-attr as it seems to be the best approach for storing multiple data vars
  • commonalise the domain algorithm between the data model and multi-attr writer
  • commonalise to a single writer
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant