Why does fsspec always connects to blob?

Robin Kühling 6

I am currently working in an Azure Synapse Notebook. I use fsspec (https://learn.microsoft.com/en-us/azure/synapse-analytics/machine-learning/tutorial-spark-pool-filesystem-spec) to connect to a datalake storage to read and write json files.

Although everything is configured to connect to datalake storage, fsspec seems to try to build a connection to a blob storage which is not accessable:

ServiceRequestError: Cannot connect to host accountname.blob.core.windows.net:443 ssl:True [Name or service not known]

What am I doing wrong? My code looks like this:

import fsspec
import pandas

adls_account_name = 'accountname' #Provide exact ADLS account name
sas_key = TokenLibrary.getConnectionString('Staging')

fsspec_handle = fsspec.open('abfs://sandbox/marketing/schema.json', account_name = adls_account_name, sas_token=sas_key)

with fsspec_handle.open() as f:
df = pandas.read_json(f)

df.head()

PRADEEPCHEEKATLA-MSFT 76,511 Reputation points Microsoft Employee

2021-10-27T06:33:20.563+00:00
Hello @Robin Kühling ,

Thanks for the question and using MS Q&A platform.

Could you please share more details on where does this error message shows up: ServiceRequestError: Cannot connect to host accountname.blob.core.windows.net:443 ssl:True [Name or service not known]? And also please do share the complete stacktrace of the error message which you are experiencing?

As per the repro, I'm able to connect and read the data from ADLS Gen2 using fsspec:

FSSPEC can read/write ADLS data by specifying the linked service name.

In Synapse studio, open Data > Linked > Azure Data Lake Storage GenUpload data to the default storage account.

Run the following code. PYSPARK CODE: import fsspec
import pandas adls_account_name = '' #Provide exact ADLS account name
sas_key = TokenLibrary.getConnectionString(<LinkedServiceName>) fsspec_handle = fsspec.open('abfs://<container>/<path-to-file>', account_name = adls_account_name, sas_token=sas_key) with fsspec_handle.open() as f:
df = pandas.read_csv(f)
Robin Kühling 6 Reputation points

2021-10-28T06:53:39.747+00:00

The error started to occur when private endpoints where implemented in the project.
I already veryfied that the account name is correct and that the sas_key corresponds to the desired linked service. The URL in the linked Service is https://<account_name>.dfs.core.windows.net. Though fsspec always raises an error that it cant connect to ...BLOB.core.windows.net.

I think the code you suggest is from the MS documentation right? This is the same what I do I think.

The complete stacktrace:

144445-fsspec-stacktrace.txt
PRADEEPCHEEKATLA-MSFT 76,511 Reputation points Microsoft Employee

2021-10-28T08:14:16.463+00:00
Hello @Robin Kühling ,

Have you tried the above method like just passing the storage account name?

adls_account_name = 'chepragen2' #Provide exact ADLS account name sas_key = TokenLibrary.getConnectionString("ADLSGen2_chepra") #Provide exact ADLS Linked Service
Robin Kühling 6 Reputation points

2021-10-28T09:14:24.487+00:00

I don't really get what you mean? Your code example is exactly the same as mine.
In the actual code I did not write 'accountname', I just used it to anonymize a company project.
PRADEEPCHEEKATLA-MSFT 76,511 Reputation points Microsoft Employee

2021-10-28T09:37:41.437+00:00

Hello @Robin Kühling ,

Could you please share the screenshot of the code which you are trying to execute?
Robin Kühling 6 Reputation points

2021-10-28T10:12:48.337+00:00

The same as in my original question
PRADEEPCHEEKATLA-MSFT 76,511 Reputation points Microsoft Employee

2021-11-01T11:35:17.117+00:00

Hello @Robin Kühling ,

Did you get a chance to accept your own answer?

1 answer

Robin Kühling 6 Reputation points

2021-10-28T10:22:49.83+00:00

It seems like fsspec always routes over blob.core even if the private endpoints specifies the url as dfs.core. After creating a private endpoints for the data lake which explioctly routes over blob.core the code works again.
Please sign in to rate this answer.

1 person found this answer helpful.
PRADEEPCHEEKATLA-MSFT 76,511 Reputation points Microsoft Employee

2021-10-29T06:10:28.7+00:00

Hello @Robin Kühling ,

Glad to know that your issue has resolved. You can accept it as answer(). This can be beneficial to other community members. Thank you.

Kamal Sharma 1 Reputation point

2021-11-23T10:18:13.143+00:00

Hi @Robin Kühling

How did you create private endpoints that explicitly route over blob.core? I created private endpoint with target sub-resource as blob but the error still persists.

WZIAFP 157 Reputation points

2022-09-28T12:02:13.367+00:00

@Robin Kühling @PRADEEPCHEEKATLA-MSFT Would like to know how you resolved this also. I have the same problem as @Kamal Sharma

David Beavon 971 Reputation points

2023-03-01T17:12:16.65+00:00

Did you happen to open a support case? This seems like buggy behavior.
Sign in to comment