can't connect Databricks to Synapse

Question

i'm trying to read a table created in synapse, this is my configuration

spark.conf.set("fs.azure.account.auth.type", "OAuth")
spark.conf.set("fs.azure.account.oauth.provider.type",  "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider")
spark.conf.set("fs.azure.account.oauth2.client.id", "<<APPLICATION CLIENT ID>>")
spark.conf.set("fs.azure.account.oauth2.client.secret", "<<CLIENT SECRET VALUE>>")
spark.conf.set("fs.azure.account.oauth2.client.endpoint", "https://login.microsoftonline.com/<<DIRECTORY TENANT ID>>/oauth2/token")

# Defining a separate set of service principal credentials for Azure Synapse Analytics (If not defined, the connector will use the Azure storage account credentials)
spark.conf.set("spark.databricks.sqldw.jdbc.service.principal.client.id", "<<APPLICATION CLIENT ID>>")
spark.conf.set("spark.databricks.sqldw.jdbc.service.principal.client.secret", "<<CLIENT SECRET VALUE>>")

storageAccount = "<<STORAGE-NAME>>"
containerName = "<<CONTAINER-NAME>>"
accessKey = "<<KEY>>"
jdbcURI = "<<jdbc:sqlserver://...>>"

spark.conf.set(f"fs.azure.account.key.{storageAccount}.blob.core.windows.net", accessKey)

tableDF = (spark.read.format("com.databricks.spark.sqldw").option("url", jdbcURI).option("tempDir", "abfss://CONTAINER@WORKSPACE.dfs.core.windows.net/temp").option("enableServicePrincipalAuth", "true").option("dbTable", tableName).load())

tableDF.createOrReplaceTempView("TBL")

But, when I run the code, I have this error: com.databricks.spark.sqldw.SqlDWConnectorException: Exception encountered in Azure Synapse Analytics connector code

Are there any settings I am missing?

Answer 1

Hello @Luis Cespedes ,
Thanks for the ask and using Microsoft Q&A platform .
You have not mentioned as to if you are trying to connected for dedicted pool or on demand pool , but the below reply from @PRADEEPCHEEKATLA does call out all the steps .

https://learn.microsoft.com/en-us/answers/questions/327270/azure-databricks-to-azure-synapse-service-principa.html

Please do let me know how it goes .
Thanks
Himanshu