Can't write to blob storage from AzureML Spark Cluster

Question

Can't write to blob storage from AzureML Spark Cluster

Anonymous

Using the Azure ML Spark compute (serverless or attached), it is not possible to write to gen2 datalake blob storage.

The code below produces the error 'Caused by: org.apache.hadoop.fs.azure.AzureException: com.microsoft.azure.storage.StorageException: This operation is not permitted on a non-empty directory.'

This greatly reduces the usefulness of the spark integration. Any help would be appreciated.

df = #pyspark.pandas or pyspark.sql dataframe
df.to_parquet('azureml://[blah]) #or df.write.parquet('azureml://[blah])

PRADEEPCHEEKATLA 90,651 Reputation points Moderator

2023-05-11T11:01:27.1366667+00:00

The error message you provided indicates that the operation is not permitted on a non-empty directory. This error can occur when you try to write to a directory that already contains data.

To resolve this issue, you can try the following steps:

Check if the directory already contains data: You can check if the directory already contains data by using the Azure Storage Explorer or the Azure Portal. If the directory already contains data, you may want to delete the data before writing new data to the directory.

Use overwrite mode: You can use overwrite mode to overwrite the existing data in the directory. To do this, you can set the "mode" parameter to "overwrite" when writing data to the directory.

Use a different directory: If the directory already contains data and you do not want to overwrite it, you can use a different directory to write the new data.

I hope this helps! Let me know if you have any further questions.
Anonymous

2023-05-11T22:03:06.7033333+00:00

Hi Pradeep, sorry for the lack of clarity. The 'non-empty' directory is the _temporary directory that df.write.parquet creates as part of its process.

Adding the mode=overwrite parameter does not help.
PRADEEPCHEEKATLA 90,651 Reputation points Moderator

2023-05-12T04:34:33.6833333+00:00

@Anonymous - Have you tried using different directory?

Could you please share the code snippet used and complete stack trace of the error message?
PRADEEPCHEEKATLA 90,651 Reputation points Moderator

2023-05-17T05:01:49.08+00:00

@Dance, Cody R. (ALT) - We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.

Your answer

PRADEEPCHEEKATLA 90,651 Reputation points Moderator

2023-05-11T11:01:27.1366667+00:00

The error message you provided indicates that the operation is not permitted on a non-empty directory. This error can occur when you try to write to a directory that already contains data.

To resolve this issue, you can try the following steps:

Check if the directory already contains data: You can check if the directory already contains data by using the Azure Storage Explorer or the Azure Portal. If the directory already contains data, you may want to delete the data before writing new data to the directory.

Use overwrite mode: You can use overwrite mode to overwrite the existing data in the directory. To do this, you can set the "mode" parameter to "overwrite" when writing data to the directory.

Use a different directory: If the directory already contains data and you do not want to overwrite it, you can use a different directory to write the new data.

I hope this helps! Let me know if you have any further questions.
Anonymous

2023-05-11T22:03:06.7033333+00:00

Hi Pradeep, sorry for the lack of clarity. The 'non-empty' directory is the _temporary directory that df.write.parquet creates as part of its process.

Adding the mode=overwrite parameter does not help.
PRADEEPCHEEKATLA 90,651 Reputation points Moderator

2023-05-12T04:34:33.6833333+00:00

@Anonymous - Have you tried using different directory?

Could you please share the code snippet used and complete stack trace of the error message?
PRADEEPCHEEKATLA 90,651 Reputation points Moderator

2023-05-17T05:01:49.08+00:00

@Dance, Cody R. (ALT) - We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.

Share via

Can't write to blob storage from AzureML Spark Cluster

Your answer