Unable to access ADLS Gen2 through Spark while manually running a notebook in Synapse Analytics Studio

Question

Unable to access ADLS Gen2 through Spark while manually running a notebook in Synapse Analytics Studio

Ashwin R 0

I'm trying to read a csv file from my ADLS container through Spark using a linked service but when I run my notebook I'm getting java.nio.file.AccessDeniedException: Operation failed: "This request is not authorized to perform this operation.", 403, HEAD but when I run the same notebook through a pipeline it seems work just fine, so much so that I don't even need to set any spark.conf. My synapse workspace has Blob Storage Data Contributor access and my account has User access to the storage account. My linked service uses System Managed Identity.

I have attached the code below and the error

Py4JJavaError: An error occurred while calling o4538.csv.
: java.nio.file.AccessDeniedException: Operation failed: "This request is not authorized to perform this operation.", 403, HEAD, https://myaccount.dfs.core.windows.net/fsglasdp/?upn=false&action=getAccessControl&timeout=90

from notebookutils.mssparkutils import fs
from notebookutils import mssparkutils
from pyspark.sql import SparkSession


source_full_storage_account_name = "myaccount.dfs.core.windows.net"

linked_service_name = "mylinkedservice"

spark.conf.set(
    f"spark.storage.synapse.{source_full_storage_account_name}.linkedServiceName",
    linked_service_name
)

sc._jsc.hadoopConfiguration().set(
    f"fs.azure.account.oauth.provider.type.{source_full_storage_account_name}",
    "com.microsoft.azure.synapse.tokenlibrary.LinkedServiceBasedTokenProvider"
)


df = spark.read.csv('abfss://<container>@<myaccount>.dfs.core.windows.net/path/to/folder/mycsv.csv')

Smaran Thoomu 35,045 Reputation points Microsoft External Staff Moderator

2025-07-21T08:25:41.93+00:00
Hi Ashwin R

Thanks for sharing the detailed error and context.

What you're experiencing is expected behavior due to how identity and permissions work differently when running notebooks manually (interactively) vs. through pipelines in Synapse.

Why it works in pipeline:

When the pipeline runs, it uses the System Managed Identity (SMI) of the Synapse workspace - which has the correct permissions (Blob Storage Data Contributor) on your ADLS Gen2.

Why it fails manually in notebook:

When you manually run a notebook via Synapse Studio, it uses your personal user identity (your Azure AD account), not the workspace SMI. If your personal account only has Reader or User access, you won’t be able to read from the ADLS container.

How to fix It:

You can resolve this in one of two ways:

Option 1: Use Workspace MSI for manual notebook runs

Manually specify the linked service in the Spark config (you already tried this). Make sure you run this block before accessing the file:

source_full_storage_account_name = "myaccount.dfs.core.windows.net" linked_service_name = "mylinkedservice" spark.conf.set( f"spark.storage.synapse.{source_full_storage_account_name}.linkedServiceName", linked_service_name ) sc._jsc.hadoopConfiguration().set( f"fs.azure.account.oauth.provider.type.{source_full_storage_account_name}", "com.microsoft.azure.synapse.tokenlibrary.LinkedServiceBasedTokenProvider" )

Also ensure:

The linked service name is correct.

The System Managed Identity of Synapse has at least Storage Blob Data Contributor role.

You may need to restart the session or kernel before retrying.

Option 2: Grant your user account permissions on ADLS

If you prefer to run notebooks using your identity:

Go to the Storage Account in the Azure Portal.

Under “Access control (IAM)”, assign your user Storage Blob Data Contributor role at the container or account level.

Let me know which option you'd prefer to go with, or if you'd like help checking permissions. Once this is set, your manual notebook runs should work just like they do in pipelines.

I hope this information helps. Please do let us know if you have any further queries.

Kindly consider upvoting the comment if the information provided is helpful. This can assist other community members in resolving similar issues.
Ashwin R 0 Reputation points

2025-07-21T08:48:25.36+00:00

Thanks for your reply @Smaran Thoomu but my issue isn't resolved. Yes, I have followed all your steps for option 1 which is what I want. I have ensured the linked service name is correct, it has the correct access (system-assigned manged identity), I also ran it before accessing the files and also my workspace also has Storage Blob Data Contributor role. I have also ensured my linked service is working by trying to mount my ADLS using this linked service which worked perfectly fine. I was able to read and write. But I'm unable to access my ADLS using full path like abfss://
Smaran Thoomu 35,045 Reputation points Microsoft External Staff Moderator

2025-07-22T14:03:29.88+00:00
Hi Ashwin R

Thanks for confirming that you have already implemented Option 1 and verified the linked service, identity roles, and Spark configs - that's helpful context.

Since mounting via the linked service works fine, but you're facing issues specifically when using full abfss:// paths, here are a few points to check further:

Things to Recheck

Ensure Spark Config Block Runs Before File Access

Make sure the config block (spark.conf.set(...)) is executed in the same session and before the spark.read.csv(...) call.

If you're testing in different cells, ensure there's no session restart in between.

Correct Linked Service Name Mapping

Double-check that the full ADLS account domain (myaccount.dfs.core.windows.net) exactly matches how it’s defined in the Spark config.

Any mismatch (even a typo or trailing slash) may break the identity token flow.

Try Using mssparkutils.fs.head()

As a quick test, run:
mssparkutils.fs.head("abfss://<container>@<account>.dfs.core.windows.net/path/to/file.csv")
If this also fails, it confirms that the identity binding to the storage account is not being honored for direct access.

Recommended Next Step

To isolate if it’s an identity binding issue or a Spark resolution problem, try the following:

Temporarily assign yourself (your user identity) the Storage Blob Data Contributor role on the same container.

Then retry the notebook read using abfss://....

If this succeeds, it confirms that Spark is defaulting to user identity, not honoring the linked service identity despite the config.

This may happen if the linked service token binding isn't correctly applied - often due to workspace or notebook session state.

Let me know if you'd prefer help checking this or want to explore using mssparkutils.fs.mount() as a workaround in the interim.

I hope this information helps. Please do let us know if you have any further queries.
Smaran Thoomu 35,045 Reputation points Microsoft External Staff Moderator

2025-07-23T19:30:49.08+00:00

Ashwin R We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

1 answer

Your answer

Ashwin R 0 Reputation points

2025-07-21T08:48:25.36+00:00

Thanks for your reply @Smaran Thoomu but my issue isn't resolved. Yes, I have followed all your steps for option 1 which is what I want. I have ensured the linked service name is correct, it has the correct access (system-assigned manged identity), I also ran it before accessing the files and also my workspace also has Storage Blob Data Contributor role. I have also ensured my linked service is working by trying to mount my ADLS using this linked service which worked perfectly fine. I was able to read and write. But I'm unable to access my ADLS using full path like abfss://
Smaran Thoomu 35,045 Reputation points Microsoft External Staff Moderator

2025-07-23T19:30:49.08+00:00

Ashwin R We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

Answer 1

Hey, I know it’s been a while since I posted this question, but I finally had some time to revisit it and found a solution.

Config:

My Synapse workspace is not in a Managed VNet, so there shouldn’t be any restriction on outbound network access. I also verified that the “Allow Azure services and resources to access this workspace” option is enabled in the networking tab.

On my ADLS Gen2 storage account, I enabled “Allow trusted Microsoft services to access this resource” under the networking settings. Additionally, I added my Synapse workspace as a resource instance with access to the storage account. The workspace’s managed identity has Storage Blob Data Owner, Contributor, and Reader roles assigned. I also granted full rwx ACL permissions to this managed identity for the entire container.

Despite all these configurations, I was still getting the following error when trying to read or write files using Spark from Synapse:

Operation failed: "This request is not authorized to perform this operation.", 403, HEAD

What made this confusing was that the issue only occurred when running the notebook interactively (using the managed identity in the Spark session), and not when running it through a pipeline.

Solution:

After some investigation, I discovered that adding a private endpoint to the ADLS Gen2 storage account resolves the issue. You can do this from the Synapse Manage tab under the Security → Managed private endpoints section. Once created, make sure to approve the private endpoint in the storage account.

This solution worked in my case.

TL;DR: Add a private endpoint from Synapse to your ADLS Gen2 storage account.

Share via

Unable to access ADLS Gen2 through Spark while manually running a notebook in Synapse Analytics Studio

1 answer

Your answer