SplineInitializationException Error

[SP-ARUN]

Hi Alex,

I'm encountering an error as follow while trying to get the lineage from Databricks and store it in Azure Functions. SplineInitializationException : Spark Agent was not able to establish connection to Spline Gateway. HTTP Response: 404 while trying to enable the spline lineage.

The below is the code i used:

System.setProperty("spline.mode", "REQUIRED")
System.setProperty("spark.spline.lineageDispatcher", "http") 
System.setProperty("spark.sql.queryExecutionListeners", "za.co.absa.spline.harvester.listener.SplineQueryExecutionListener")
System.setProperty("spline.producer.url","https://xxxxxxxx.azurewebsites.net/api/func/producer")

-- This runs fine

import za.co.absa.spline.harvester.SparkLineageInitializer._
spark.enableLineageTracking()

-- this throws an error - SplineInitializationException: Spark Agent was not able to establish connection to Spline Gateway. HTTP Response: 404

Could you help me here ?

[wajda]

I guess it's the same issue as #469. Check the Azure function exposed API as described in the mentioned topic.

[SP-ARUN]

I could still not overcome the issue/error i'm encountering.

I am using spline with Azure Databricks and receive the lineage to a Azure function. Can you share with me the best practices for this ?

I got this method from Spline Portal's Article. I used the below link to get this method done

https://github.com/intellishore/data-lineage-databricks-to-purview.

[wajda]

Unfortunately I don't have enough personal capacity to dig into 3rd party implementations. The above error message occurs when the HTTPLineageDispatcher doesn't get a response from the HEAD /status request. The response code makes it clear - 404. Check your Azure function behaviour with curl or any general purpose REST client.

[SP-ARUN]

Hi Alex,

I have included the the HTTPLineageDIspatcher to the code, Correct me if Im worng.

System.setProperty("spline.mode", "REQUIRED")
System.setProperty("spark.spline.lineageDispatcher", "http")
System.setProperty("spark.sql.queryExecutionListeners", "za.co.absa.spline.harvester.dispatcher.HttpLineageDispatcher")
System.setProperty("spline.lineageDispatcher.http.apiVersion", "LATEST")
System.setProperty("spline.producer.url","https://xxxxxxxx.azurewebsites.net/api/func/producer")

import za.co.absa.spline.harvester.SparkLineageInitializer._
spark.enableLineageTracking()

I'm still encountering the 404 error.

Is there something, I should be reconsidering here ?? Do I need to check with my Azure Functions permission? do we have any documentation/ plan to check what level of access/permission we need wrt Spline gateway to connect with Functions app ?

[ganeshnikumbh]

Hi @SP-ARUN , you can also try the codeless installation steps I created in attached document. as far as the error is concerned, you will have to check your Azure function, network (vnet) config, ssl config etc.. try debugging with postman..

Install_Spline_Codeless.docx

[wajda]

Examine the HTTP lineage dispatcher source code to get a better understanding of what's happening inside of it - https://github.com/AbsaOSS/spline-spark-agent/blob/release/0.7.10/core/src/main/scala/za/co/absa/spline/harvester/dispatcher/HttpLineageDispatcher.scala