Can't write to blob storage from Azure Spark Cluster

Question

I am working on a notebook in azure synapse using pyspark for creating a parquet file.

I'm using the code

df.write.mode('overwrite').parquet(path)

When running i am getting the following error

Caused by: org.apache.hadoop.fs.azure.AzureException: com.microsoft.azure.storage.StorageException: This operation is not permitted on a non-empty directory.

My directory is empty, I went to check the folder and some files are written in it, but after few minutes the error appears.

Thank you for your attention.

Answer 1

Hi Rafael Martins de Mendonça,

Thank you for posting query in Microsoft Q&A Platform.

This can happen if there are files or folders in the directory that you are trying to overwrite.

To resolve this issue, you can try the following steps:

1. Delete the existing files or folders in the directory that you are trying to overwrite. You can do this using the Azure Storage Explorer or the Azure Portal.
2. Run your code again to write the parquet file to the directory.

If you are still seeing the error message after deleting the existing files or folders, you can try specifying a new directory path for the parquet file. This will ensure that you are not overwriting any existing files or folders.

Another thing to check is whether there are any other processes or applications that are accessing the directory at the same time. This can cause conflicts and prevent your code from writing to the directory.

I hope this helps! Let me know how it goes. Thank you.