Databricks Synapse connector using certificate

Question

Hi,

We a large number of records to Synapse dedicated pool using the synapse connector. We are able to do this using userid and password and client id and secret. However, we wanted to check if this is possible using client id and certificate and/or private key.
We are using the below code to set the spark.conf for the Synapse connector. he code we have looks like this :
# Defining the service principal credentials for the Azure storage account
spark.conf.set("fs.azure.account.auth.type", "OAuth")
spark.conf.set("fs.azure.account.oauth.provider.type", "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider")
spark.conf.set("fs.azure.account.oauth2.client.id", "<application-id>")
spark.conf.set("fs.azure.account.oauth2.client.secret", "<service-credential>")
spark.conf.set("fs.azure.account.oauth2.client.endpoint", "https://login.microsoftonline.com/<directory-id>/oauth2/token")

\# Defining a separate set of service principal credentials for Azure Synapse Analytics (If not defined, the connector will use the Azure storage account credentials)
spark.conf.set("spark.databricks.sqldw.jdbc.service.principal.client.id", "<application-id>")
spark.conf.set("spark.databricks.sqldw.jdbc.service.principal.client.secret", "<service-credential>")

We're aware of MSAL and we have successfully used this with secret and certificates. However, this is quite slow compared to the Spark Synapse connector for ingestion of large volumes of data.
Is there a way to use Spark Synapse connector with SPN+ certificate?

Answer 1

Hello @Chand, Anupam SBOBNG-ITA/RX ,

Thanks for the question and using MS Q&A platform.

Unfortunately, it is not possible to use Spark Synapse connector with SPN+ certificate.

Azure Databricks only support SPN/secret when accessing the ADLS gen2 storage. As described in the document, both ADB and synapse need authenticate with the storage account. For the private key you mentioned, does it mean storage key? If yes, then it shall support when accessing blob storage.

Appreciate if you could share the feedback on our feedback channel. Which would be open for the user community to upvote & comment on. This allows our product teams to effectively prioritize your request against our existing feature backlog and gives insight into the potential impact of implementing the suggested feature.

Hope this will help. Please let us know if any further queries.

------------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators