Add EDDTableFromParquet reader?

[MathewBiddle]

I'm continually seeing more and more groups using the Parquet Data File format (https://parquet.apache.org/docs/). Looking through the docs, I see there is an apache/parquet-java package (https://github.com/apache/parquet-java) which could lend itself to the java backend of ERDDAP. I'm curious if others have explored Parquet more in depth and if there is a need for ERDDAP to have the capacity to read from that format.

I haven't seen any discussion on the topic yet.
https://github.com/search?q=org%3AERDDAP%20parquet&type=code
https://groups.google.com/g/erddap/search?q=parquet

[MathewBiddle]

Also, maybe a writer is part of the conversation too??

[srstsavage]

+1, very performant format for certain use cases, would be great to add support for it

[rmendels]

Parquet JDBC - https://www.cdata.com/drivers/parquet/jdbc/

[rmendels]

see also https://cdn.cdata.com/help/HIK/jdbc/

[kevin-obrien]

I'll add another +1 for this.

[rmendels]

Does anyone have sample Parquet files?

[MathewBiddle]

😨 https://obis-datasets.ams3.digitaloceanspaces.com/exports/obis_20240723.parquet

[MathewBiddle]

It's big tho! GBs

[srstsavage]

glider.parquet.gz

[srstsavage]

FWIW duckdb is very powerful/useful for exploring and analyzing parquet files, and also provides a JDBC interface:

https://duckdb.org/docs/api/java.html

ClickHouse is another great analysis platform that uses parquet. Lots of options!

[rmendels]

Thanks for the links. Yes I already use duckdb for similar and knew it could do Parquet. In R there is duckdbplyr which ties duckdb to the tidyverse. I want to test if the JDBC already in ERDDAP can be made to work as is. But that only gives you read, not write, a brief look at the Apache Parquet Java suggests it may not be so simple. Also how much actual demand is there for this - at least in our immediate communities. Also, though it is not my favorite organization, https://geoparquet.org. This is related to GeoArrow, see https://geoarrow.org - scroll down for the discussion on the relationship with GeoParquet

[ChrisJohnNOAA]

I made an issue to track work on this. #196

[ChrisJohnNOAA]

I think directly using parquet-java in a new dataset type is the best plan for this. My guess is it will likely be the most performant option.