there's a similar question that has been marked as answer, can you check if this helps?
Error while writing Delta table to ADLS from Azure Synapse analytics using serverless
DeGuy
40
Reputation points
I'm tying to write delta formatted table to ADLS Gen2 from Azure Synapse Pyspark notebook with Serverless SQL pool.
It is throwing me error, while writing to ADLS Gen2 as below.
Py4JJavaError: An error occurred while calling o3898.save.
: Operation failed: "An HTTP header that's mandatory for this request is not specified.", 400, PUT,
Below is the code which I am using to write to ADLS.
if (DeltaTable.isDeltaTable(spark, source_path)):
print('Existing delta table')
# Read the existing Delta Table
delta_table = DeltaTable.forPath(spark, source_path)
# Merge new data into existing table
delta_table.alias("existing").merge(
source = df_eventLog.alias("updates"),
condition = " AND ".join(conditions_list)
).whenMatchedUpdateAll(
).whenNotMatchedInsertAll(
).execute()
else:
print('New delta table')
# Create new delta table with new data
df_eventLog.write.format('delta').save(source_path)
In my case, Delta table isn't available initially, So, else part is running. df_eventLog is loading fine without errors.
Can someone help me where I am going wrong?
Thank you in advance.
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,378 questions
1 answer
Sort by: Most helpful
-
Vinodh247 34,741 Reputation points MVP Volunteer Moderator
2023-07-25T04:56:34.59+00:00