Need to deploy on Synapse environment in order to track lineage for Synapse notebooks

[talhagithub02]

I have a requirement where I would need to scan pyspark notebook but within Azure synapse environment so whenever we run a synapse notebook (pyspark) the lineage for it gets tracked by spline, I am not sure how to achieve this, I have tested spline on local pyspark scripts which is good to go but what and where to deploy on synapse environment is confusing.

Your assistance required and your time is appreciated.

[wajda]

Putting it simply, all you need to do boils down to two simple steps:

1. You need to add the Spline agent bundle JAR to the Spark driver, one way or another.
2. Register the Spline agent listener in the Spark listener manager (either via config or programmatically) to let Spark know that it needs to kick the agent.

I've never used Synapse, but a quick googling gave me this - https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/apache-spark-manage-pool-packages

Hope it helps.