Share via

Can Synapse Spark Clusters query dedicated SQL pools in the workspace?

Anonymous
2023-05-03T21:21:45.1066667+00:00

I have a Synapse workspace that has both a dedicated SQL pool and a spark cluster. Is there a way to query the SQL pool from the spark cluster using PySpark? I.e. is there a way to connect PySpark SQL to the entire SQL pool?

Pandas on Spark separately allows connection to the SQL pool and has a query function, but internally the queries are used as subqueries which makes some things difficult.

Azure SQL Database
Azure
Azure
A cloud computing platform and infrastructure for building, deploying and managing applications and services through a worldwide network of Microsoft-managed datacenters.
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
0 comments

2 answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. Anonymous
    2023-05-03T21:59:54.9866667+00:00
    0 comments
  2. Oury Ba-MSFT 21,126 Reputation points Microsoft Employee Moderator
    2023-05-05T12:57:08.1166667+00:00

    Dance, Cody R. (ALT) Thank you for reaching out.

    I am glad that you were able to find the answer here https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/synapse-spark-sql-pool-import-export?tabs=python%2Cpython1%2Cpython2%2Cscala3%2Cscala4%2Cscala5.

    Yes, you are right, you can query a dedicated SQL pool in Azure synapse workspace using pyspark.

    From your dedicated SQL pool, you can right click and see "New Notebook."

    From here, you can use pyspark code to query the tables. 

    User's image

    Regards,

    Oury

Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.