I am trying to use the mssparkutils package in azure synapse notebook. Im following the process as shown here . I am getting this error I was using the exact same process before and it was working. It started giving this issue on saturday 12 AM. I have the same issue on different synapse accounts with different storage accounts. I have also used different spark pools with different versions of spark. Tried with scala, got the same issue Got the same issue using these methods blob Storage account and Data Lake Storage Gen2 account Also Tried with account key and linked service. Our previous notebook was working fine before but after Saturday it started having the issue above. It does work using local file system API e.g. mssparkutils.fs.ls("file:///synfs/<jobId>/test") or using glob on /synfs/<jobId>/test will work. Sample Code

@PRADEEPCHEEKATLA-MSFT After 2,3 days i ran the same code, changed nothing and it started working again.

Hi 544cl, Thanks for reaching out to Microsoft Q&A. Please refer to the updated answer by pradeep below. I could see that this is the same error you are facing. https://learn.microsoft.com/en-us/answers/questions/565405/mount-azure-storage-account-to-file-system.html Please Upvote and Accept as answer if the reply was helpful.

Microsoft Spark Utilities Issue

M Saad 36

I am trying to use the mssparkutils package in azure synapse notebook. Im following the process as shown here . I am getting this error

I was using the exact same process before and it was working. It started giving this issue on saturday 12 AM.
I have the same issue on different synapse accounts with different storage accounts.
I have also used different spark pools with different versions of spark.
Tried with scala, got the same issue
Got the same issue using these methods blob Storage account and Data Lake Storage Gen2 account
Also Tried with account key and linked service.
Our previous notebook was working fine before but after Saturday it started having the issue above.
It does work using local file system API e.g. mssparkutils.fs.ls("file:///synfs/<jobId>/test") or using glob on /synfs/<jobId>/test will work.

Sample Code

2 answers

M Saad 36 Reputation points

2022-09-16T05:11:34.807+00:00

@PRADEEPCHEEKATLA-MSFT
After 2,3 days i ran the same code, changed nothing and it started working again.
Please sign in to rate this answer.

1 person found this answer helpful.
PRADEEPCHEEKATLA-MSFT 88,791 Reputation points Microsoft Employee

2022-09-19T07:01:02.207+00:00

Hello @M Saad ,

Glad to know issue has been resolved.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.
Vinodh247 18,501 Reputation points

2022-09-13T06:20:15.427+00:00

Hi 544cl,

Thanks for reaching out to Microsoft Q&A.

Please refer to the updated answer by pradeep below. I could see that this is the same error you are facing.

https://learn.microsoft.com/en-us/answers/questions/565405/mount-azure-storage-account-to-file-system.html

Please Upvote and Accept as answer if the reply was helpful.
Please sign in to rate this answer.
M Saad 36 Reputation points

2022-09-13T06:43:08.887+00:00

Hello Vinodh247-1375,
Thanks for answering. I am using the exact same process as shown above. This issue started recently it was working for me before.

PRADEEPCHEEKATLA-MSFT 88,791 Reputation points Microsoft Employee

2022-09-13T09:31:26.467+00:00

Hello @M Saad ,

Could you please share the stack trace of the error message which you are experiencing?

M Saad 36 Reputation points

2022-09-13T09:53:00.77+00:00

Recent Logs from driver

22/09/13 09:39:37 INFO notebookUtils: createNewBlob - /mounts/514/mount.json start execute request.
22/09/13 09:39:37 INFO notebookUtils: createNewBlob - /mounts/514/mount.json execute request successfully.
22/09/13 09:39:37 INFO notebookUtils: Try to release lock for mount info config file
22/09/13 09:39:37 INFO notebookUtils: checkBlobIfExists - /mounts/514/Lock start execute request.
22/09/13 09:39:37 INFO notebookUtils: checkBlobIfExists - /mounts/514/Lock execute request successfully.
22/09/13 09:39:37 INFO notebookUtils: deleteBlob - /mounts/514/Lock start execute request.
22/09/13 09:39:38 INFO notebookUtils: deleteBlob - /mounts/514/Lock execute request successfully.
22/09/13 09:39:38 INFO notebookUtils: successfully release lock for mountinfo config file.
22/09/13 09:39:38 INFO notebookUtils: [telemetry][info][funcName:mount|cost:29394|traceId:c329bfbd-905e-4ce0-a91f-ca6b807ec025|storageType:dfs] exec success
22/09/13 09:39:38 INFO env: Getting jobId (mssparkutils)

PRADEEPCHEEKATLA-MSFT 88,791 Reputation points Microsoft Employee

2022-09-13T10:02:08.707+00:00

Hello @M Saad ,

Thanks for sharing additional details.

Currently, I'm working with the internal team and I will update you with my findings.

PRADEEPCHEEKATLA-MSFT 88,791 Reputation points Microsoft Employee

2022-09-13T10:27:28.323+00:00

Hello @M Saad ,

As per the repro, I was able to get the desired output. Could you please re-try and do let us know the status.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Microsoft Spark Utilities Issue

2 answers

Your answer