Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

chore: use polars-compute in polars-parquet statistics #16687

Closed
coastalwhite opened this issue Jun 3, 2024 · 1 comment · Fixed by #17153 or #17158
Closed

chore: use polars-compute in polars-parquet statistics #16687

coastalwhite opened this issue Jun 3, 2024 · 1 comment · Fixed by #17153 or #17158
Assignees
@coastalwhite
Labels
accepted Ready for implementation enhancement New feature or an improvement of an existing feature

Comments

@coastalwhite
Copy link
Collaborator

Description

We would like to use polars-compute functions in the polars-parquet crate to calculate some of the metadata
@coastalwhite coastalwhite added the enhancement New feature or an improvement of an existing feature label Jun 3, 2024
@coastalwhite coastalwhite self-assigned this Jun 3, 2024
@coastalwhite
Copy link
Collaborator Author

Part of this is addressed in #16788. But the most important part, which is the PrimitiveArray, is not done yet. There are some trait bounds that are difficult to circumvent.

@coastalwhite coastalwhite mentioned this issue Jun 20, 2024
Open
5 tasks
coastalwhite added a commit to coastalwhite/polars that referenced this issue Jun 24, 2024
@coastalwhite
refactor(rust): MinMaxKernel in prim parquet stats
8570914 
Fixes pola-rs#16687.

This PR now fully utilizes the MinMaxKernel to calculate the Parquet statistics. Specifically this PR adds this for `PrimitiveArray` which is the majority of values.
@coastalwhite coastalwhite mentioned this issue Jun 24, 2024
Merged
coastalwhite added a commit to coastalwhite/polars that referenced this issue Jun 24, 2024
@coastalwhite
refactor(rust): MinMaxKernel in prim parquet stats
b82b54d 
Fixes pola-rs#16687.

This PR now fully utilizes the MinMaxKernel to calculate the Parquet statistics. Specifically this PR adds this for `PrimitiveArray` and `BinaryArray` which is the majority of values.
@coastalwhite coastalwhite mentioned this issue Jun 24, 2024
Merged
@c-peters c-peters added the accepted Ready for implementation label Jul 1, 2024
@c-peters c-peters added this to Backlog Jul 1, 2024
@c-peters c-peters moved this to Done in Backlog Jul 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@coastalwhite coastalwhite

Labels
accepted Ready for implementation enhancement New feature or an improvement of an existing feature
Projects
Backlog
Archived in project
Milestone
No milestone
2 participants
@coastalwhite @c-peters