ADF copy activity from ADLS to delta lake fails due to invalid configuration value detected

Question

ADF copy activity from ADLS to delta lake fails due to invalid configuration value detected

Aswathi Kallinkeel 0

Hi, I have a copy activity in ADF to copy data from ADLS to Delta lake. I created linked services with authentication method as Access token. The test connection for linked service is successful. But when I try to execute my pipeline I am getting an error message mentioned in the screenshot.

we have mount points created in Databricks cluster to read and write data from/to ADLS.

Kindly help to resolve this issue.

Aswathi Kallinkeel 0 Reputation points

2025-04-30T11:57:31.3633333+00:00

Adding the screenshot with error message here
Smaran Thoomu 23,260 Reputation points Microsoft External Staff Moderator

2025-05-05T08:03:09.89+00:00

@Aswathi Kallinkeel Following up to see if the below answer was helpful. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

1 answer

Your answer

Aswathi Kallinkeel 0 Reputation points

2025-04-30T11:57:31.3633333+00:00

Adding the screenshot with error message here
Smaran Thoomu 23,260 Reputation points Microsoft External Staff Moderator

2025-05-05T08:03:09.89+00:00

@Aswathi Kallinkeel Following up to see if the below answer was helpful. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Answer 1

Hi @Aswathi Kallinkeel
Based on the error message you're encountering:

Invalid configuration value detected for fs.azure.account.key usually indicates that the Databricks mount point is attempting to authenticate using a storage account key, but the configuration provided is either missing, incorrect, or conflicting with the linked service authentication method.

From your description, you are using:

ADF with Access Token authentication for the linked service.
Databricks mount points to ADLS, likely configured using fs.azure.account.key.

Root Cause

Databricks mount points created using fs.azure.account.key rely on storage account keys, but your ADF linked service is configured with Access Token (OAuth). When ADF tries to interact with the mounted directory via Databricks, it does not have access to the expected key-based configuration, leading to the error.

Option 1: Use OAuth-based mount configuration in Databricks

If your organization prefers token-based authentication, consider re-mounting the ADLS location in Databricks using Azure Active Directory passthrough or service principal (OAuth), which aligns with ADF’s Access Token method.

Sample (OAuth) mount code:

configs = {
  "fs.azure.account.auth.type": "OAuth",
  "fs.azure.account.oauth.provider.type": "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider",
  "fs.azure.account.oauth2.client.id": "<app-id>",
  "fs.azure.account.oauth2.client.secret": "<secret>",
  "fs.azure.account.oauth2.client.endpoint": "https://login.microsoftonline.com/<tenant-id>/oauth2/token"
}
dbutils.fs.mount(
  source = "abfss://<container>@<storageaccount>.dfs.core.windows.net/",
  mount_point = "/mnt/<mount-name>",
  extra_configs = configs)

Option 2: Update the mount point to use the correct storage account key

If you're committed to using storage account keys, ensure:

The correct fs.azure.account.key.<storageaccount>.dfs.core.windows.net is configured in the cluster or passed in as part of the Spark configuration.
The key used is valid and active.

The test connection success in ADF only confirms the linked service setup, not the downstream Spark mount access. ADF does not manage Databricks cluster mount credentials - those are controlled within the Databricks environment.

For more information on the error please refer this: https://medium.com/@kyle.hale/troubleshooting-invalid-configuration-value-detected-for-fs-azure-account-key-6c6fcc67b217

I hope this information helps. Please do let us know if you have any further queries.

Kindly consider upvoting the comment if the information provided is helpful. This can assist other community members in resolving similar issues.

Aswathi Kallinkeel 0 Reputation points

2025-04-30T13:50:49.3233333+00:00

Hi Smaran, Thank you for your response. I am passing the same secret details used in Databricks as well.

Can I change the connectivity using system assigned managed identity? Will this issue resolve. If yes for ADF what kind off access required in Databricks.
Smaran Thoomu 23,260 Reputation points Microsoft External Staff Moderator

2025-05-02T20:45:19.6166667+00:00

Hi @Aswathi Kallinkeel

Yes, you can switch to System Assigned Managed Identity for authentication — and doing so can help resolve the mismatch between ADF's authentication method and the one used in your Databricks mount configuration.

However, to make this work end-to-end, you'll need to make sure the Databricks cluster and mount point are configured to support OAuth-based authentication using a service principal or managed identity, instead of using fs.azure.account.key. Steps to proceed:

In Azure Data Factory

Continue using System Assigned Managed Identity in the linked service to ADLS.

Ensure that this identity has the right RBAC roles:

At a minimum, assign Storage Blob Data Contributor on the ADLS Gen2 container or filesystem.

In Azure Databricks

Unfortunately, Databricks does not directly support mounting ADLS using a Managed Identity (system-assigned or user-assigned) today.

Instead, you can:

Use a Service Principal (client ID/secret) to configure the mount with OAuth.

Or, avoid mounts altogether, and access ADLS paths directly via abfss://... using the Spark config with OAuth.

Alternative (Recommended for alignment)

To fully align ADF and Databricks authentication models:

Don't use mounts.

Use direct ABFSS path access in your Databricks notebooks or jobs, passing the same OAuth config as Spark settings.

That way, ADF (using Managed Identity or Access Token) and Databricks (using SPN or token) work independently but securely.

Hope this clarifies things!

Kindly consider upvoting the comment if the information provided is helpful. This can assist other community members in resolving similar issues.

Share via

ADF copy activity from ADLS to delta lake fails due to invalid configuration value detected

1 answer

Your answer