Is there a way I connect to Synapse Spark Pool using dbt-spark? #1164
Unanswered
MohammedSarfarazuddin
asked this question in
Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
I want to read and write delta tables from the ADLS Gen 2 storage account and for the transformation part, I would like to use dbt.
Our team of analytics engineers use dbt with Azure SQL server right now but we would like to switch to the lakehouse approach with Delta Lake.
Is this possible with Synapse Spark pools? (i.e., the data transformations using dbt-spark connected to a synapse spark pool) If yes, can anyone share their approach on how they set it up because my initial attempts have not succeeded in even establishing a connection.
Extra info: We do not use Azure Databricks but we will be moving to Fabric in few months. So if is more feasible to set up dbt for lakehouse using dbt-fabric, please enlighten me.
Thanks!
Below is the profiles.yml I have used.
deltatest: outputs: dev: type: spark method: http host: {sparkpoolname}.dev.azuresynapse.net # The Synapse workspace hostname port: 443 schema: default token: "" # EntraID token target: dev
Beta Was this translation helpful? Give feedback.
All reactions