Databricks 7.0 load to Azure Synapse Analytics fails when using useAzureMSI = true and writeSemantics = copy

Tom Smith 1 Reputation point
2020-06-29T21:51:55.107+00:00

When I try to execute a script on Databricks 7.0 to write data to a table in Azure Synapse Analytics, I get an error: Parse error at line: 7, column: 30: Incorrect syntax near ''Managed Service Identity''.

I have useAzureMSI option equal to true. Using the Managed Service Identity for our storage account access is a hard requirement for us.

I worked around this by setting the spark conf to use polybase - spark.conf.set("spark.databricks.sqldw.writeSemantics", "polybase")

but I'd like to try out the new COPY command to load the data, if there is a workaround available for that. Anyone have any ideas or sample code to get the write command configured properly?

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,222 questions
{count} votes

1 answer

Sort by: Most helpful
  1. Nolan Walker 1 Reputation point
    2020-08-11T15:21:53.627+00:00

    Hi @HimanshuSinha-msft !

    I can confirm that this is still happening. Everything except writing to Synapse works. It looks like the underlying SQL that is sent to the cluster via the JDBC has a syntax error. The fix the OP wrote was the only way I could get it to write after I changed cluster version.

    0 comments No comments

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.