Error while writing Delta table to ADLS from Azure Synapse analytics using serverless

DeGuy 40 Reputation points
2023-07-25T00:51:30.91+00:00

I'm tying to write delta formatted table to ADLS Gen2 from Azure Synapse Pyspark notebook with Serverless SQL pool.

It is throwing me error, while writing to ADLS Gen2 as below.

User's image

Py4JJavaError: An error occurred while calling o3898.save.
: Operation failed: "An HTTP header that's mandatory for this request is not specified.", 400, PUT,

Below is the code which I am using to write to ADLS.

if (DeltaTable.isDeltaTable(spark, source_path)):
    print('Existing delta table')
    # Read the existing Delta Table
    delta_table = DeltaTable.forPath(spark, source_path)
 
    # Merge new data into existing table
    delta_table.alias("existing").merge(
        source = df_eventLog.alias("updates"),
        condition = " AND ".join(conditions_list)
         
    ).whenMatchedUpdateAll(
    ).whenNotMatchedInsertAll(
    ).execute() 
else:
    print('New delta table')
    # Create new delta table with new data
    df_eventLog.write.format('delta').save(source_path)

In my case, Delta table isn't available initially, So, else part is running. df_eventLog is loading fine without errors.

Can someone help me where I am going wrong?

Thank you in advance.

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,378 questions
{count} votes

1 answer

Sort by: Most helpful
  1. Vinodh247 34,741 Reputation points MVP Volunteer Moderator
    2023-07-25T04:56:34.59+00:00

    there's a similar question that has been marked as answer, can you check if this helps?

    https://learn.microsoft.com/en-us/answers/questions/124197/writing-parquet-file-throws-an-http-header-thats-m

    1 person found this answer helpful.

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.