Azure Databricks authentication to Azure SQL database with service principal

Question

Hello Team,

I have created below sql connection python file in my ADO repo. But while executing databricks notebook I am facing dq_library module not found and the error is pointing to sql connection python file.

Also am authenticating Azure Databricks to Azure SQL authentication via service principal below are all the snippets. All the relevant details regarding sql and databricks is stored in secrets and setup config files.

sql.py file in repo which is in turn referred in databricks notebook

And below is my notebook, I have installed the required wheel files on the cluster.

Can you please let me know how this issue could be fixed? the error was pointing to line number 10 in notebook. Do I need to import msal/adal library in sql connection python file or any other recommendations to resolve this issue.

Regards,

Ashwini G

Answer 1

@Ashwini Gaikwad - I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others ", I'll repost your solution in case you'd like to accept the answer .

Ask: Azure Databricks authentication to Azure SQL database with service principal

Solution: We figured out that the issue is with version of dq_library wheel file which we are building and not with sql connection file. Service principal authentication works fine.

If I missed anything please let me know and I'd be happy to add it to my answer, or feel free to comment below with any additional information.

If you have any other questions, please let me know. Thank you again for your time and patience throughout this issue.

---

Please don’t forget to Accept Answer and Yes for "was this answer helpful" wherever the information provided helps you, this can be beneficial to other community members.

Answer 2

Hi Ashwini,

It seems like you are trying to run something that isn't native to databricks

Try running the following in the first code block:

%pip install dq

dbutils.library.restartPython()

Refer to https://pypi.org/project/dq/ for more information on DataQuery.If this is helpful please accept answer.