Replies: 1 comment 1 reply
-
|
@sergiimk Thanks for creating an issue. I hadn't seen Spark Connect before, looks interesting! Livy alternatives are much needed. There aren't plans to support it, but at a glance, it seems very feasible. Would you be open to contributing? |
Beta Was this translation helpful? Give feedback.
1 reply
-
|
@devstein I would be interested. Any open discussions for reference? |
Beta Was this translation helpful? Give feedback.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
I've been a user of Sparkmagic <-> Livy combo for a long time and while upgrading spark I stumbled upon Spark Connect.
Since Livy is essentially unmaintained, I was wondering if there are plans to support this interface instead.
It seems to be based on Apache Arrow, which should provide an top-notch performance in combination with Pandas.
Our company is also heavily invested in DataFusion and Arrow FlightSQL (SQL API over Arrow) so I would really like to get to a state where Sparkmagic works with both Spark and our DataFusion+FlightSQL engines.
Beta Was this translation helpful? Give feedback.
All reactions