Error "[Errno 5] Input/output error" when opening ADLS Blob from mounted file system

David Leach 1

Hi,

Using a Synapse Pyspark Notebook I have mounted an ADLS container and checked this is working ok with the following command:

"jobid=mssparkutils.env.getJobId()"
"mssparkutils.fs.ls(f"synfs:/{jobid}/synapse-ws01")"

This shows underlying folders and files as expected :

"[FileInfo(path=synfs:/1/synapse-ws01/app, name=app, size=0),
FileInfo(path=synfs:/1/synapse-ws01/client_secrets.json, name=client_secrets.json, size=441),
FileInfo(path=synfs:/1/synapse-ws01/mnt, name=mnt, size=0),
FileInfo(path=synfs:/1/synapse-ws01/synapse, name=synapse, size=0)]"

However when I attempt to open the file "client_secrets.json" using the command:

"f = os.open(f"/synfs/{jobid}/synapse-ws01/client_secrets.json",os.O_RDONLY)"

The execution hangs for about 6 minutes and finally returns:

"OSError: [Errno 5] Input/output error: '/synfs/1/synapse-ws01/client_secrets.json'"

The logs do not provide any clues as to what the issue is. The Synapse Workspace is a Storage Blob Data Contributor on the Storage Account. I have also repeated this test against a newly created test Storage Account with no firewall restrictions and the same result occurred.

Can anyone help or advise as to how I might get some more detail as to the error?

Thanks,

Dave

ShaikMaheer-MSFT 38,401 Reputation points Microsoft Employee

2022-10-20T16:52:00.963+00:00

Hi @David Leach ,

Thank you for posting query in Microsoft Q&A Platform.

Could you please check once mount point exists or not using below sample code? test1 is mount point name here. Replace it accordingly.
import os
jobID=mssparkutils.env.getJobId()
os.listdir("/synfs/" + jobID + "/test1")

If mount point exists, then try to try below format of path strings and see if that helps.
'file:///synfs/' + jobID + '/test/data/'
OR
'/synfs/'+jobID+'/test1/data/'

Please let me know how it goes. Thank you.
ShaikMaheer-MSFT 38,401 Reputation points Microsoft Employee

2022-10-21T13:55:44.503+00:00

Hi @David Leach , Just checking if you get chance to share updates? Thank you.
David Leach 1 Reputation point

2022-11-25T08:29:37.623+00:00

Hi

Many thanks for responding, I didn't realise someone had responded and I am still no further in finding the answer. Your suggestions did not work, when I run "listdir" this returns the folders under the mounted container as expected, I can also list files in sub folders but the opening a file using "f = open("/synfs/" + jobid + "/......", this fails with: "OSError: [Errno 5] Input/output error: '/synfs/............json'". I can however use "mssparkutils.fs.head......" and this works fine!

I have investigated further by creating another storage account from scratch and the code works fine on the new storage account but I have replicated all the settings so it does not make sense. My feeling is that this is somehow file permissions as I can list the files but not open them. I have checked all the file permissions on the storage accounts and they are identical so it makes no sense.

This is becoming quite urgent now so your help would me much appreciated.

Dave
David Leach 1 Reputation point

2022-11-25T09:08:43.14+00:00

Another thought, the storage accounts are accessed via Private Endpoints on Virtual Network, could it be something to do with these although both the one that works and the one that doesn't work are both on VN's and both using PE's.
David Leach 1 Reputation point

2022-11-25T10:10:23.293+00:00

Further feedback, I discovered your feedback in this article: https://stackoverflow.com/questions/73188733/how-to-mount-file-as-a-file-object-using-pyspark-in-azure-synap. I have created a Blob Storage private endpoint instead of a ADLS Gen2 endpoint and it has worked. What I cannot understand is the Synapse Linked Service is still using the ADLS Gen2 Private Endpoint so how is the Blob Storage Private Endpoint making the difference?
Eman Abela 0 Reputation points

2023-03-06T10:50:43.0433333+00:00

Having same issue myself with same set up in terms of MPEs. Was there every any explanation why this works correctly through the Blob MPE even despite using the DFS endpoint and driver when mounting?
David Beavon 976 Reputation points

2023-03-22T23:23:50.1166667+00:00

Hi all. I opened a support case with Microsoft, and I suggest you do too. There is some sort of weird incompatibility that seems to be happening when using python in synapse to access an ADLS gen2 storage container with MPE's.

The confusing thing is the need to create an additional blob endpoint MPE. They added an explanation of this well-known bug in a "Tutorial" Use Pandas to read/write ADLS data in serverless Apache Spark pool in Synapse Analytics - Azure Synapse Analytics

https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/tutorial-use-pandas-spark-pool

Please note that the problem is specific to python's internal fsspec implementation. We do not need the "blob" endpoint (nor do we encounter errors) when using a regular programming language like scala, java, or c#.

Share via

Error "[Errno 5] Input/output error" when opening ADLS Blob from mounted file system