can't connect Databricks to Synapse

Luis Cespedes 41 Reputation points
2022-01-19T14:27:36.917+00:00

i'm trying to read a table created in synapse, this is my configuration

spark.conf.set("fs.azure.account.auth.type", "OAuth")
spark.conf.set("fs.azure.account.oauth.provider.type",  "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider")
spark.conf.set("fs.azure.account.oauth2.client.id", "<<APPLICATION CLIENT ID>>")
spark.conf.set("fs.azure.account.oauth2.client.secret", "<<CLIENT SECRET VALUE>>")
spark.conf.set("fs.azure.account.oauth2.client.endpoint", "https://login.microsoftonline.com/<<DIRECTORY TENANT ID>>/oauth2/token")

# Defining a separate set of service principal credentials for Azure Synapse Analytics (If not defined, the connector will use the Azure storage account credentials)
spark.conf.set("spark.databricks.sqldw.jdbc.service.principal.client.id", "<<APPLICATION CLIENT ID>>")
spark.conf.set("spark.databricks.sqldw.jdbc.service.principal.client.secret", "<<CLIENT SECRET VALUE>>")

storageAccount = "<<STORAGE-NAME>>"
containerName = "<<CONTAINER-NAME>>"
accessKey = "<<KEY>>"
jdbcURI = "<<jdbc:sqlserver://...>>"

spark.conf.set(f"fs.azure.account.key.{storageAccount}.blob.core.windows.net", accessKey)

tableDF = (spark.read.format("com.databricks.spark.sqldw").option("url", jdbcURI).option("tempDir", "abfss://CONTAINER@WORKSPACE.dfs.core.windows.net/temp").option("enableServicePrincipalAuth", "true").option("dbTable", tableName).load())

tableDF.createOrReplaceTempView("TBL")

But, when I run the code, I have this error: com.databricks.spark.sqldw.SqlDWConnectorException: Exception encountered in Azure Synapse Analytics connector code

Are there any settings I am missing?

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,443 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,955 questions
0 comments No comments
{count} votes

Accepted answer
  1. HimanshuSinha-msft 19,381 Reputation points Microsoft Employee
    2022-01-21T01:05:31.827+00:00

    Hello @Luis Cespedes ,
    Thanks for the ask and using Microsoft Q&A platform .
    You have not mentioned as to if you are trying to connected for dedicted pool or on demand pool , but the below reply from @PRADEEPCHEEKATLA-MSFT does call out all the steps .

    https://learn.microsoft.com/en-us/answers/questions/327270/azure-databricks-to-azure-synapse-service-principa.html

    Please do let me know how it goes .
    Thanks
    Himanshu


0 additional answers

Sort by: Most helpful