You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
At the moment, csp fails to read parquet files that have the "large string" arrow type. This is particularly problematic because polars uses large string by default and doesn't plan to change. More info about polars string types here. This means that parquet files written by polars are not directly readable by csp.
Describe the solution you'd like
The csp parquet reader natively supports large string type.
Describe alternatives you've considered
We have utility functions now when writing polars parquet files to convert to arrow, identify large types, cast to small types, and write using pyarrow.parquet, but this is tedious/non-standard and this approach doesn't support other polars functionality (like the streaming engine) that might generate such parquet files.
Additional context
N/A
The text was updated successfully, but these errors were encountered:
Is your feature request related to a problem? Please describe.
At the moment, csp fails to read parquet files that have the "large string" arrow type. This is particularly problematic because polars uses large string by default and doesn't plan to change. More info about polars string types here. This means that parquet files written by polars are not directly readable by csp.
Describe the solution you'd like
The csp parquet reader natively supports large string type.
Describe alternatives you've considered
We have utility functions now when writing polars parquet files to convert to arrow, identify large types, cast to small types, and write using
pyarrow.parquet
, but this is tedious/non-standard and this approach doesn't support other polars functionality (like the streaming engine) that might generate such parquet files.Additional context
N/A
The text was updated successfully, but these errors were encountered: