Spark Actions that trigger Spline Lineage harvesting

[ianmark72]

I’m working with a library that writes a custom dispatcher that does some transformations to the Spline output and submits to an endpoint. However, I can only get it to trigger on a Spark write, and I’ve tried a few other actions such as count, show, map, head. I’m trying to get this to a point where I can capture lineage information based on REST submit. That being said, are there any other ways to trigger Spline besides a write?

[cerveada]

Spline understands the lineage as description of flow of data from one data source to another, so if there is no write to a data source it's not something Spline Agent is supposed to capture.

So there are no other ways to trigger Spline other than write and this is intentional.

[ianmark72]

That makes sense. I’m publishing my to my datasource via REST, so I’ll just write to an intermediary location beforehand to track the lineage information. Just out of curiosity as I wasn’t able to find it myself, where in the code does Spline listen for the Query Execution events and access the execution plan?

[cerveada]

Now I think I understand the question better. Your Spark job is writing the data to rest endpoint. Right?

In that case that should be considered write by Spline. The problem is the agent sees only the operations represented in Spark Logical Plan, if it's UDF we don't see inside the function.

There is already ticket for the UDF problem:
#181

In case it's a write "hidden" in UDF. I think Spark will not recognise it and will not call the listener we listen to at all. Not sure about this though.

[cerveada]

... where in the code does Spline listen for the Query Execution events and access the execution plan?

SplineQueryExecutionListener is the actual Spark listener implementation
LineageHarvester is the place where the actual transformation from Spark Logical Plan to spline model happens

[wajda]

I’ve tried a few other actions such as count, show, map, head

Just for the reference, there is a feature request to enable capturing lineage from memory only actions - #361