NativeAzureFileSystem.rename fails with Error: One of the request inputs is not valid

Question

NativeAzureFileSystem.rename fails with Error: One of the request inputs is not valid
My configuration is working with many other operations (createFolder, upload file, delete file, ...). So it is not a config issue.
I'm running the latest version: hadoop-azure-3.3.6.jar
The error message is not helpful. I'm not sure which input is not valid.

Any suggestions on what could be the issue?
Are there logs in Azure storage that I can see for more information on what is wrong?
Here is a simple code that shows the problem.
The error also happens when Spark internal functions try to rename a file after uploading data to Azure Storage Blob (trying to rename the temp folder to the target folder).

JavaSparkContext jsc = getSparkContext(); // Get valid Spark context

//<container-name>,<storage-account> replaced with my values
String parent = "wasbs://<container-name>@<storage-account>.blob.core.windows.net/";
Path source = new Path(parent  + "sourceFolder");
Path target = new Path(parent  + "targetFolder");
FileSystem fs = source.getFileSystem(jsc.hadoopConfiguration()); // fs is an object of NativeAzureFileSystem
fs.rename(source, target); 

This failed with the error:

org.apache.hadoop.fs.azure.AzureException: com.microsoft.azure.storage.StorageException: One of the request inputs is not valid.
	at org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.rename(AzureNativeFileSystemStore.java:2898)
	at org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.rename(AzureNativeFileSystemStore.java:2770)
	at org.apache.hadoop.fs.azure.NativeAzureFileSystem$FolderRenamePending.execute(NativeAzureFileSystem.java:468)
	at org.apache.hadoop.fs.azure.NativeAzureFileSystem.rename(NativeAzureFileSystem.java:3337)
	

Answer 1

Welcome to the Microsoft Q&A Platform. Thank you for reaching out, and I hope you are doing well.

Azure Storage Analytics provides logging and metrics capabilities that help you understand how applications are using your storage accounts. To enable storage analytics logs for a storage account, you can leverage the article https://learn.microsoft.com/en-us/azure/storage/common/manage-storage-analytics-logs?tabs=azure-portal

Once storage analytics logs are enabled, you can access/download them using tool Azure Storage Explorer/ Storage browser in portal in $logs folder. You can leverage the following tool https://github.com/nunogabrielmonteiro/AzureStorageLogReader to read the logs in a formatted way.

Please refer https://learn.microsoft.com/en-us/rest/api/storageservices/blob-service-error-codes to corelate the findings and errors.

Thanks

Answer 2

Hello Sameh Ayoub,

Thank you for posting your query here!

The error "One of the request inputs is not valid," is a generic error message from Azure Storage, it often occurs when there is an issue with the parameters provided in the request. However, I would request you to check the following points:

• Please ensure that the paths and container names you're providing are correct and follow the expected format.
• Azure Storage has specific naming rules for containers and blobs. Please ensure that you are following these rules (e.g., lowercase letters, no special characters except "-", etc.).

You may enable logging for the Azure Storage library to get more detailed information about the request and response. You can enable diagnostic logging for your Azure Storage account, and these logs can be helpful in troubleshooting issues. The diagnostic logs capture information about various operations, including successful and failed operations. https://learn.microsoft.com/en-us/azure/storage/common/manage-storage-analytics-logs?tabs=azure-portal#enable-logs

If you chose to store logs in Azure Storage, you can find them in a container named $logs within your storage account.

Please let us know if you have any further queries. I’m happy to assist you further.

---

Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.

Answer 3

I enabled the logs, but there are no logs for the rename APIs. It only logs for read, write and delete operations.

Regarding the two concerned of names of the path and storage, they are good, it is not the issue.

Because I can do other operations successfully. The folder that is want to rename in this code was just created immediately by another API call.

Can you please try the above simple code and see if it ever works on your side?
The error blocking Spark internal functions of saving data to Azure Storage Blob (trying to rename the temp folder to the target folder).