question

FelixLampDE-6873 avatar image
0 Votes"
FelixLampDE-6873 asked $$ANON_USER$$ commented

Mount Azure Storage Account to File System

Hello,

I am using Synapse. I want to use my own Java Programm. I can import it into a scala workbook. My Java program uses java.io.File that needs a path to a file. Therefore I want to mount the Azure Storage Account to my file system.

I have located mssparkutils.fs.mountToDriverNode. However, this method does not seem to be documented online.

Can you help me?

Best,

Felix

azure-synapse-analytics
· 5
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @FelixLampDE-6873,

Thanks for using Microsoft Q&A !!
Can you please let us know where are you seeing this mounttoDriverNode method. When using Synapse Spark pool available methods as listed in the document are -

  • cp

  • mv

  • ls

  • mkdirs

  • put

  • head

  • append

  • rm

You can also check available methods on by executing mssparkutils.fs.help() in your Scala notebook. (See screenshot below)
135105-image.png

If you want to access file from your Blob Storage or ADLS Gen2 then you can follow above shared documentation which will help you configure access and use the mssparkutils package to access the files.
Please let me know if you have any other questions.

Thanks
Saurabh


0 Votes 0 ·
image.png (56.1 KiB)
image.png (56.6 KiB)

Hi Saurabh,

thank you for your answer. I can see mounttoDriverNode from the completion in the IDE and I can also import it without getting errors:

import mssparkutils.fs.mountToDriverNode

From the methods you have provided I cannot see how would I mount a ADLS Gen2 directory to the file system. Databricks has a mount method:

https://docs.microsoft.com/en-us/azure/databricks/dev-tools/databricks-utils#--mounts-command-dbutilsfsmounts

Is there something equivalent for Synapse?

Best,

Felix

0 Votes 0 ·

Hi @felixlampde-6873,
Which IDE you are seeing this method available. Can you please provide a screenshot if possible.

Thanks
Saurabh

0 Votes 0 ·
Show more comments

how to get all mount points and how can we access that mount?

getting this error , could you plz help

AttributeError: module 'notebookutils.mssparkutils.fs' has no attribute 'mounts'
Traceback (most recent call last):

AttributeError: module 'notebookutils.mssparkutils.fs' has no attribute 'mounts'

and i mounted adls to /mnt/mdpdevfs/ , but when i run below code , im getting error

mssparkutils.fs.ls("/mnt/mdpdevfs/")



Py4JJavaError: An error occurred while calling z:mssparkutils.fs.ls.
: java.io.FileNotFoundException: Operation failed: "The specified path does not exist.", 404, GET, https://adlskgltraininggen2.dfs.core.windows.net/fs-training?upn=false&resource=filesystem&maxResults=500&directory=mnt/mdpdevfs&timeout=90&recursive=false, PathNotFound, "The specified path does not exist. RequestId:f532d193-101f-0040-77c1-cf6bdf000000 Time:2021-11-02T08:16:41.8990033Z"

0 Votes 0 ·

1 Answer

PRADEEPCHEEKATLA-MSFT avatar image
1 Vote"
PRADEEPCHEEKATLA-MSFT answered PRADEEPCHEEKATLA-MSFT commented

Hello @FelixLampDE-6873,

(UPDATE: 1/11/2021): Currently these are the three authentication types are supported when trigger mount operation, via LinkeService, via accountKey and via sastoken.

145491-image.png


Method1: Using Linked Service

 mssparkutils.fs.mount(
     "abfss://<FILESYSTEMNAME>@<STORAGENAME>.dfs.core.windows.net/", #ADLS GEN 2 PATH
     "</MOUNTNAME>", #Mount Point Name
     { "linkedService" : "<REPLACE LINKED SERVICE NAME>"}
 )

Method2: Using SAS Token

 mssparkutils.fs.mount(
     "abfss://<FILESYSTEMNAME>@<STORAGENAME>.dfs.core.windows.net/", #ADLS GEN 2 PATH
     "</MOUNTNAME>", #Mount Point Name
     { "sasToken" : "<REPLACE SAS TOKEN KEY>"}
 )

Method3: Using Account Key

 mssparkutils.fs.mount(
     "abfss://<FILESYSTEMNAME>@<STORAGENAME>.dfs.core.windows.net/", #ADLS GEN 2 PATH
     "</MOUNTNAME>", #Mount Point Name
     { "accountKey" : "<REPLACE AACOUNT KEY>"}
 )



Welcome to the Microsoft Q&A platform.

Currently, Azure Synapse Analytics doesn't allow mounting of Azure Storage accounts in Spark Pool.

As per the conversation with product team, they are working on this feature and it will be available soon.

I will update the thread once it goes live.

Hope this will help. Please let us know if any further queries.


  • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how

  • Want a reminder to come back and check responses? Here is how to subscribe to a notification

  • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators




image.png (60.4 KiB)
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi pradeep ,
mssparkutils.fs.mount( source = "abfss://container@adls.dfs.core.windows.net/",
mountPoint = "/mnt/mdpdevfs",
extraConfigs = configs)

i tried giving all ways of authorization in configs , still im getting
"com.microsoft.spark.notebook.msutils.InvalidCredentialsException: Please provide at least one of LinkedService name, accountKey or sasToken for authorization." Error , how can i solve this

0 Votes 0 ·

Hello anonymous user,

Please check the updated answer.

0 Votes 0 ·