Error connecting databricks to azure synapse

Question

I'm new working with cloud services and I'm trying to make a connection between databricks and azure synapse. I have notebooks in databricks that generate data frames and I want to populate a Dedicated SQL pool inside synapse with them.

After looking at what the microsoft documentation recommends do and follow the steps, I came across this error.

code

df.write\  
  .format("com.databricks.spark.sqldw") \  
  .option("url", <the-rest-of-the-connection-string>") \  
  .option("forwardSparkAzureStorageCredentials", "true") \  
  .option("dbTable", "Table") \  
  .option("tempDir", "wasbs://<your-container-name>@<your-storage-account-name>.blob.core.windows.net/<your-directory-name>") \  
  .load()  

error

Py4JJavaError: An error occurred while calling o1509.save.  
: com.databricks.spark.sqldw.SqlDWConnectorException: Exception encountered in Azure Synapse Analytics connector code.  
  
Caused by: java.lang.StringIndexOutOfBoundsException: String index out of range: 14  

Some considerations

• I created a simple data frame for testing purposes assuming the problem could be the data frame.
• An empty table was not previously generated in synapse, I expect it to be created automatically

Could someone please help me understand this problem?

Answer 1

Hi @Guilherme Lima

It seems you want to populate a table in synapse dedicated sql pool with data from a databricks dataframe. I see you are using spark.read for that, instead you should be using the spark "write" method.

Here is a sample code that will allow you to write the data in dataframe "df" into a synapse dedicated sql pool. If the table "table01" doesn't reside in synapse dedicated pool, it will be created for you.

df.write\
.mode('append')\
.format(com.databricks.spark.sqldw)\
.option("url","complete connection string URL")\
.option("forwardSparkAzureStorageCredentials","true")\
.option("dbTable","table01")\
.option("tempDir","staging_folder_path in your storage account")\
.save()

Hope this helps!