ADF Delta table in copy activity fails

Question

Hi.

I need help debugging my error.
I have ADF copy activity from Delta lake dataset to CSV (and other pipeline other way around).
Delta lake dataset works fine in design mode. Meaning that schem import and data preview both works. However when I run either of my pipelines I get this error:

ErrorCode=AzureDatabricksCommandError,Hit an error when running the command in Azure Databricks. Error details: Failure to initialize configurationInvalid configuration value detected for fs.azure.account.key Caused by: Invalid configuration value detected for fs.azure.account.key.

I have located both my delta tables and CSV's / folders in same storage account (gen2 with hierarchical namespace enabled ) in same container.
From databricks cluster I can access all locations and files used. My cluster does have my storage accounts primary key in Spark-configs. All out of ideas why I am getting this error.

Please help :)

Answer 1

Hello @,
Thanks for the ask and using Microsoft Q&A platform .
You will have to create a Application id and then pass the details in the config as shown below

fs.azure.account.auth.type.<storage-account-name>.dfs.core.windows.net OAuth
fs.azure.account.oauth.provider.type.<storage-account-name>.dfs.core.windows.net org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider
fs.azure.account.oauth2.client.id.<storage-account-name>.dfs.core.windows.net <application-id>
fs.azure.account.oauth2.client.secret.<storage-account-name>.dfs.core.windows.net <service-credential>
fs.azure.account.oauth2.client.endpoint.<storage-account-name>.dfs.core.windows.net https://login.microsoftonline.com/<tenant-id>/oauth2/token

Please do let me know how it goes .
Thanks
Himanshu

-------------------------------------------------------------------------------------------------------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators