Running ADF notebooks on SQL endpoint instead of interactive cluster

Question

Hi,

We have a ADF pipeline that runs several notebooks. Currently all notebooks run on interactive databricks cluster. Is it possible to run these notebooks on databricks SQL endpoint instead? If so could you please share steps to create linked service.

Regards,

_arc

Answer 1

Hi _arc,

Thank you for using Microsoft Q&A forum and posting your query.

Unfortunately, the current Azure Databricks connector cannot be used to execute your notebooks on databricks SQL endpoint clusters.

Here is an existing feature request logged by other user in IDEAS forum for the same requirement: connect Azure Databricks SQL endpoints with Azure Databricks using ADF Please feel free to upvote it and comment as it would help increase the priority of the feature request.

Workaround:

Since Databricks support ODBC connection, as an alternative you may try exploring Azure Data Factory ODBC connection to connect to Databricks SQL endpoint and execute queries.

ADF ODBC Connector documentation: Copy data from and to ODBC data stores using Azure Data Factory or Synapse Analytics

Here is an article by a community volunteer where it has been explained in detail on how to establish connection to Databricks SQL endpoint from Azure Data factory using ODBC connector: How to Connect to Databricks SQL Endpoint from Azure Data Factory?

Hope this info helps.

---

Please don’t forget to Accept Answer and Up-Vote wherever the information provided helps you, this can be beneficial to other community members.