Share via

Need to have more clearance spark synapse copy write options

Girisankar Rajendran 0 Reputation points
2024-03-22T07:34:45.9733333+00:00

Hi All,

For my business requirements, my spark code in databricks were used to load a gigabytes of data into Azure synapse database.

The actual issue was by using COPY Into I can able to use option errorfile used to store error records.

But in spark I am unable to achieve it. Could any one help us to achieve this.

Spark code:

spark.conf.set("fs.azure.account.auth.type", "OAuth")

spark.conf.set("fs.azure.account.oauth.provider.type", "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider")

spark.conf.set("fs.azure.account.oauth2.client.id", "

Azure Synapse Analytics
Azure Synapse Analytics

An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.

Azure Databricks
Azure Databricks

An Apache Spark-based analytics platform optimized for Azure.

Azure Data Factory
Azure Data Factory

An Azure service for ingesting, preparing, and transforming data at scale.

Azure Dedicated Host
Azure Dedicated Host

An Azure service that provides a dedicated physical server to host Azure virtual machines for Windows and Linux.


Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.