Share via

Microsoft Spark Utilities Issue

M Saad 36 Reputation points
2022-09-12T13:25:21.257+00:00

I am trying to use the mssparkutils package in azure synapse notebook. Im following the process as shown here . I am getting this error
240114-image.png

  • I was using the exact same process before and it was working. It started giving this issue on saturday 12 AM.
  • I have the same issue on different synapse accounts with different storage accounts.
  • I have also used different spark pools with different versions of spark.
  • Tried with scala, got the same issue
  • Got the same issue using these methods blob Storage account and Data Lake Storage Gen2 account
  • Also Tried with account key and linked service.
  • Our previous notebook was working fine before but after Saturday it started having the issue above.
  • It does work using local file system API e.g. mssparkutils.fs.ls("file:///synfs/<jobId>/test") or using glob on /synfs/<jobId>/test will work.

Sample Code
240161-image.png

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
0 comments

2 answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. M Saad 36 Reputation points
    2022-09-16T05:11:34.807+00:00

    @PRADEEPCHEEKATLA
    After 2,3 days i ran the same code, changed nothing and it started working again.

    1 person found this answer helpful.
  2. Vinodh247 40,296 Reputation points MVP Volunteer Moderator
    2022-09-13T06:20:15.427+00:00

    Hi 544cl,

    Thanks for reaching out to Microsoft Q&A.

    Please refer to the updated answer by pradeep below. I could see that this is the same error you are facing.

    https://learn.microsoft.com/en-us/answers/questions/565405/mount-azure-storage-account-to-file-system.html

    240401-test.png

    Please Upvote and Accept as answer if the reply was helpful.

Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.