NativeAzureFileSystem.delete with recursive = true failed on non-empty folder

Question

NativeAzureFileSystem.delete function with recursive = true failed on a non-empty folder.
My configuration is working with many other operations including an empty folder. So it is not a config issue.
I'm running the latest version hadoop-azure-3.3.6.jar
Any suggestions on how to delete a non-empty folder?
Implementing the delete recursion version myself is not an option, as this delete method is also called internally from many Spark functions.
I'm showing here a simple example of the problem. This code deletes successfully an empty folder and it fails if it has any sub-folder with error: This operation is not permitted on a non-empty directory

JavaSparkContext jsc = getSparkContext();

//<container-name>,<storage-account>,<folder-path> replaced with my values
Path src = new Path("wasbs://<container-name>@<storage-account>.blob.core.windows.net/<folder-path>");
FileSystem fs = src.getFileSystem(jsc.hadoopConfiguration()); // fs is an object of NativeAzureFileSystem
fs.delete(src, true);

This failed with error:
org.apache.hadoop.fs.azure.AzureException: com.microsoft.azure.storage.StorageException: This operation is not permitted on a non-empty directory.
	at org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.delete(AzureNativeFileSystemStore.java:2720)
	at org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.delete(AzureNativeFileSystemStore.java:2730)
	at org.apache.hadoop.fs.azure.NativeAzureFileSystem.deleteFile(NativeAzureFileSystem.java:2670)
	at org.apache.hadoop.fs.azure.NativeAzureFileSystem$2.execute(NativeAzureFileSystem.java:2421)
	at org.apache.hadoop.fs.azure.AzureFileSystemThreadPoolExecutor.executeParallel(AzureFileSystemThreadPoolExecutor.java:223)
	at org.apache.hadoop.fs.azure.NativeAzureFileSystem.deleteWithoutAuth(NativeAzureFileSystem.java:2433)
	at org.apache.hadoop.fs.azure.NativeAzureFileSystem.delete(NativeAzureFileSystem.java:2483)
	at org.apache.hadoop.fs.azure.NativeAzureFileSystem.delete(NativeAzureFileSystem.java:1977) 

Many people are asking about the same error but with different scenarios with no answer.

I believe it may be a bug in the library as it is internally calling NativeAzureFileSystem.deleteFile which doesn't take the recursive param. So I believe the delete recursively is not implemented correctly in this library.

Answer 1

Hello Sameh Ayoub,

Thank you for posting your query here!

Please note that the NativeAzureFileSystem.delete function with recursive = true failed on a non-empty folder because the folder is not empty. The delete function with recursive = true only works on empty folders.

To work around this limitation, you can recursively delete the contents of the directory before deleting the directory itself. Unfortunately, as you mentioned, implementing this recursive delete method yourself might not be an option.

However, you can use AzCopy with azcopy remove command and --recursive parameter:
https://learn.microsoft.com/en-us/azure/storage/common/storage-ref-azcopy-remove

Also, you can try the Delete Activity in Azure Data Factory to delete files or folders:
https://learn.microsoft.com/en-us/azure/data-factory/delete-activity

Please let us know if you have any further queries. I’m happy to assist you further.

---

Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.