While running databricks migrate pipeline facing an issue with invalid configuration for storage account key

Question

Hello Team,

I am trying to run the yaml pipeline for azure databricks migration from Non UC workspace to UC workspace for reference this is the rep https://github.com/databrickslabs/migrate so while exporting the hive metastore, I am running into error :

2024-06-17,15:30:19;INFO;Logging failure

2024-06-17,15:30:19;ERROR;Failure to initialize configurationInvalid configuration value detected for fs.azure.account.key

2024-06-17,15:30:19;ERROR;{"resultType": "error", "summary": "Failure to initialize configurationInvalid configuration value detected for fs.azure.account.key", "cause": "---------------------------------------------------------------------------\nPy4JJavaError

Also I followed this article to solve the issue https://stackoverflow.com/questions/69825487/error-invalid-configuration-value-detected-for-fs-azure-account-key

https://medium.com/@kyle.hale/troubleshooting-invalid-configuration-value-detected-for-fs-azure-account-key-6c6fcc67b217but it didn't help.

I am adding below spark configuration in the Databricks cluster but still somehow the pipeline throws me an error for the invalid configuration detected for storage account key.

spark.f"fs.azure.account.key.testadls01.dfs.core.windows.net {{secrets/DevOps/d-01-sa}}

Note: Storage account key is not expired and also has been rotated for testing

Request you to please let me know how to solve the issue or what spark configuration should I add in the cluster so this error will be mitigated.

Regards,

Ashwini Gaikwad

Answer 1

Looks like only the following config is supported when mounting blob storage using wasbs driver:

• fs.azure.account.key.<storage-account-name>.blob.core.windows.net or 
• fs.azure.sas.<container-name>.<storage-account-name>.blob.core.windows.net

The config that resulted in an error will only work for abfss driver (storage hierarchy must be enabled ie ADLS-G2) and not wasbs driver.