How to write to datalakegen2 storage using databricks in delta format when connected using SAS tocken

Question

How to write to datalakegen2 storage using databricks in delta format when connected using SAS tocken

Sai Sunny Kothuri 5

I converted the data from parquet form to data format. Now I want to write the data to blob storage - datalakegen2. But facing below error while writing. I useed the below command to write my data:

output_path = '/mnt/silver/SalesLT/Address.parquet'
df.write.format('delta').mode('overwrite').save(output_path)

org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 3.0 failed 4 times, most recent failure: Lost task 0.3 in stage 3.0 (TID 6) (10.139.64.4 executor driver): org.apache.spark.SparkException: [TASK_WRITE_FAILED] Task failed while writing rows to dbfs:/mnt/silver/SalesLT/Address.parquet.

I established a connection between datalakegen2 and databricks using SAS tokens.

Sai Sunny Kothuri 5 Reputation points

2024-04-10T01:22:54.4533333+00:00

Hello @KarishmaTiwari-MSFT

In the above mentioned output path, there is only silver container initially in the storage container. When I run the above code, it created _$azuretmpfolder$ and SalesLT folders and in the SalesLT folder it created Address.parquet folder, within it, _delta_log folder and 4 blob files are created all dynamically. Intially only empty 'silver' container is present.
KarishmaTiwari-MSFT 20,857 Reputation points Microsoft Employee Moderator

2024-04-24T05:00:35.0166667+00:00

@Sai Sunny Kothuri I have followed up with you via private message.

1 answer

Your answer

Sai Sunny Kothuri 5 Reputation points

2024-04-10T01:22:54.4533333+00:00

Hello @KarishmaTiwari-MSFT

In the above mentioned output path, there is only silver container initially in the storage container. When I run the above code, it created _$azuretmpfolder$ and SalesLT folders and in the SalesLT folder it created Address.parquet folder, within it, _delta_log folder and 4 blob files are created all dynamically. Intially only empty 'silver' container is present.
KarishmaTiwari-MSFT 20,857 Reputation points Microsoft Employee Moderator

2024-04-24T05:00:35.0166667+00:00

@Sai Sunny Kothuri I have followed up with you via private message.

Answer 1

KarishmaTiwari-MSFT 20,857 Microsoft Employee Moderator

This error is usually seen when the file is corrupted/problematic.

Ensure that the data types of the columns in your DataFrame (df) are compatible with the data types of the columns in the table you are writing to. Mismatched data types can cause write failures.
Verify that there are no incompatible data types or corruption in your data.
Make sure that all the files are in the same format.

0 comments

Share via

How to write to datalakegen2 storage using databricks in delta format when connected using SAS tocken

1 answer

Your answer