You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
arrow-rs doesn't support converting Duration arrays to/from Parquet format. To get around this, Synapse uses a hack where Duration arrays are cast to int64 before writing to disk, and when reading from disk a Cast node is inserted into the execution plan to convert back to duration.
This works for scalar arrays, but not for Duration tensors, which are represented as fixed-size lists, since the arrow-rscast method doesn't support fixed-sized lists.
This should be fixed upstream in the near future since there is significant work being done in datafusion to better support fixed-size lists: apache/datafusion#6560
In the meantime only scalar Duration fields are supported.
The text was updated successfully, but these errors were encountered:
arrow-rs
doesn't support converting Duration arrays to/from Parquet format. To get around this, Synapse uses a hack where Duration arrays are cast toint64
before writing to disk, and when reading from disk a Cast node is inserted into the execution plan to convert back to duration.This works for scalar arrays, but not for Duration tensors, which are represented as fixed-size lists, since the
arrow-rs
cast
method doesn't support fixed-sized lists.This should be fixed upstream in the near future since there is significant work being done in
datafusion
to better support fixed-size lists:apache/datafusion#6560
In the meantime only scalar Duration fields are supported.
The text was updated successfully, but these errors were encountered: