Databricks integration to Azure Synapse Serveless SQL Pool

Vishnu Tupe 20 Reputation points
2024-05-21T16:50:55.78+00:00

Hi,

What is the way to connect to Databricks to Azure Synapse serverless SQL pool

when I am trying to connect using serverless SQL endpoint getting an error as -

Py4JJavaError: An error occurred while calling o388.load. : com.microsoft.sqlserver.jdbc.SQLServerException: Cannot find the CREDENTIAL 'https://xxxxxxx.dfs.core.windows.net/xxxxxxxxx/deltalake/custgroup_partitioned', because it does not exist or you do not have permission.

Note: xxxx represents actual values.

Below is the Python code used.

Thanks.

# Create a Spark session
from pyspark.sql import SparkSession

spark = SparkSession.builder \
    .appName("Synapse SQL Connection") \
    .getOrCreate()

# Set the Synapse SQL endpoint and credentials
url = "jdbc:sqlserver://xxxxxxxxxxxxxxx-ondemand.sql.azuresynapse.net:1433;database=xxxxxxxxxxxxxxxxxxxxxxxx"
user = "xxxxxxxxxx"
password = "xxxxxxxx"

df = spark.read.format("jdbc") \
    .option("url", url) \
    .option("user", user) \
    .option("password", password) \
    .option("database", "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx") \
    .option("dbtable", "dbo.custgroup") \
    .load()
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,566 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,030 questions
{count} votes

Accepted answer
  1. BhargavaGunnam-MSFT 28,271 Reputation points Microsoft Employee
    2024-05-29T17:47:56.5933333+00:00

    Hello Vishnu Tupe,

    Thank you for the confirmation. I will post this answer as a solution, so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others

    I'll repost your solution in case you'd like to accept the answer

    Issue:

    Unable to connect to Databricks to Azure Synapse serverless SQL pool/Lake databases

    Error:

    Operation CREATE EXTERNAL DATA SOURCE is not allowed for a replicated database.

    Solution:

    Acess to Lake Database directly from the Databricks is not possible.

    Lake databases doesn't support creating external data source as they are replicated database.

    This is only supported for serverless SQL db.

    Please remember to "Accept Answer" if any answer/reply helped, so that others in the community facing similar issues can easily find the solution.

    0 comments No comments

1 additional answer

Sort by: Most helpful
  1. Deleted

    This answer has been deleted due to a violation of our Code of Conduct. The answer was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.


    Comments have been turned off. Learn more