How to use functions from ML?

[mrericrichter]

Could you please provide an example (or even an implementation) of how to use functions from the ML package, e.g. the Microsoft.Spark.ML.Feature.Bucketizer class?

[GoEddie]

I was looking at how PySpark implements the ML functions, it seems that they use numpy and do some of the work on the client but I'm not 100% - will carry on trying to figure it out!

[mrericrichter]

Typically, these functions run on worker nodes. However, it seems that Spark Connect currently supports SQL functions only. All supported functions contain a flag in their documentation that states 'Supports Spark Connect', e.g.

Functions from the MLLib do not contain this flag, e.g.

This seems to be a major limitation of Spark Connect as of today. The documentation says that more functions will be added in future versions of Spark.

[GoEddie]

i'll keep this open and add them in when they are available

[GoEddie]

There is an open pr for adding ml functions to spark connect, once this is merged in we should be able to implement the functions:

wbo4958/spark#5