Tracking Issue: utilize and track array metadata/statistics

[coastalwhite]

This issue follows the effort of incorporating computationally relevant metadata into the compute engine.

There are 4 components to this effort:

1. Importing metadata during DataFrame initialization (IO and in-memory)
2. Exporting metadata when writing DataFrame
3. Utilize available metadata to accelerate computation
4. Propagate and set metadata when available

Current Status

This metadata is now kept on ChunkedArray and can be retrieved from a Series with the SeriesTrait::get_metadata. The plan is to initially add a lot of functionally between the MetadataEnv::experimental_enabled (which can be enabled with POLARS_METADATA_USE=experimental), but to
later make this enabled by default.

Fields

We keep the following fields:

• Sorted: ascending, descending, none,
• Fast Explode
• Minimum Value
• Maximum Value
• Distinct Count

IO

We should properly read and write statistics when reading from and when writing into files. For file formats, that provide the possibility of saving metadata or statistics these should be used to save and load these metadata fields.

File Format	Reading	Writing
Parquet	🗸	🗸1
JSON	✗	N/A
NDJson	✗	N/A
Avro	?	?
CSV	✗	N/A
IPC	?	?
Excel	?	?
Arrow	?	?
Numpy	?	?
Pandas	?	?

Future

List of metadata fields that could be included:

• parted: All duplicates are clustered together (e.g. [1, 1, 1, 0, 0, 3, 2, 2])

List of operations where metadata can be used or set:

• unique_count can just use a bitmask to keep track of existence if the max_value - min_value < 128
• expand_from_offset can set the distinct_count to 1
• A distinct_count=1 can immediatly make a group slice as one group (in the group-by)
• A group_by can set the distinct_count per key/

Footnotes

1. We don't fully support writing parquet statistics (Support writing Parquet distinct_count statistics for all types #17087) and this writing should be sped up with polars-compute (chore: use polars-compute in polars-parquet statistics #16687) ↩