Query serverless SQL pool external view/ tables from an Apache Spark python notebook using AAD authentication

Question

Query serverless SQL pool external view/ tables from an Apache Spark python notebook using AAD authentication

Srinivasarao Doma 0

Serverless SQL pool which contains external tables/Views write against ADLS Gen2 parquet files.

Want to query external table / views from synapse spark notebook using python.

Server less SQL Pool is using Azure Active directory.

Can you please let me know how can achieve this.

Thank you,

Sri

2 answers

Your answer

Answer 1

Vinodh247 34,491 MVP Volunteer Moderator

Hi,

Thanks for reaching out to Microsoft Q&A.

You can use pyspark to query the data from sql pool., let me know if this is what you are looking for?

https://vinsdata.wordpress.com/2022/03/03/create-synapse-notebook-and-run-python-and-sql-under-spark-pool/

https://learn.microsoft.com/en-us/azure/synapse-analytics/get-started-analyze-spark#main

Please Upvote and Accept as answer if the reply was helpful, this will be benefitting the other community members who go through the same issue.

Srinivasarao Doma 0 Reputation points

2023-06-14T02:46:33.6966667+00:00

Srinivasarao Doma 0

Thanks for your response. I already written view against ADLS gen2.
 CREATE VIEW TaxiView 
AS SELECT * FROM    OPENROWSET(BULK 'parquet/taxi/year=*/month=*/*.parquet',
       DATA_SOURCE = 'sqlondemanddemo', 
       FORMAT='PARQUET) AS nyc


I want to execute this view in synapse python notebook using AAD.

Thank you,
Sri

Answer 2

Bhargava-MSFT 31,256 Microsoft Employee Moderator

Hello Srinivas Doma,

To connect Serverless SQL pool with Synapse python notebook with Azure AD authentication, You will need to install ADAL4j library on your spark pool.

See this stackoverflow thread for more details.

Please try the solution and let us know if you have any further questions.

Bhargava-MSFT 31,256 Reputation points Microsoft Employee Moderator

2023-06-20T16:53:34.3466667+00:00

Hello Srinivas Doma

I am checking to see if you got a chance to look into my above answer. Please let me know if you have any further questions.

Share via

Query serverless SQL pool external view/ tables from an Apache Spark python notebook using AAD authentication

2 answers

Your answer