How can I read a .dcm file stored in ADLS gen2 from pyspark notebook in Azure Synapse?

Question

How can I read a .dcm file stored in ADLS gen2 from pyspark notebook in Azure Synapse?

Prateek Narula 41

.dcm is an image format for medical images. I've stored the file on ADLS gen2 and want to access those files in a notebook,

and this is how I'm trying to access it, I'm using pydicom and notebook is unable to resolve the file destination for dcm file. Although csv file is fine.

data['dicom'] is the path to the file in the ADLS gen 2 and the path is correct. I've verified and and cross checked it multiple times.

What I want to achieve is select and display the image based on the id that I have (pid).
I want to use Azure Synapse Analytics and ADLS gen2, I can use blob storage as well but the problem is the same.

The error I get is:

both the directions (blob storage and ADLS gen2 ) give me the same file not found error.
Seems like the library is not able to access the ADLS path as is.

Accepted answer

0 additional answers

Your answer

Answer 1

MartinJaffer-MSFT 26,236

Hello @Prateek Narula and welcome to Microsoft Q&A. Thank you for providing such detailed background information and context. I had never heard of dicom or pydicom before.

So here is my hunch.

The lowest level of your error, the filereader.py is using low level operating system methods for opening a file. This is expecting to run on a normal environment on a computer where the file is stored on the local disk drive. The URI you have given is sharing the https protocol, as opposed to a local disk like C:\ .

I think we should try downloading the file to the cluster, and then running on that.

Another option is to mount the storage account, so it can then be referred to by a more operating system like designation. Actually, this
second option is what you probably want more. Mounting makes it so spark can pretend the storage account is actually an attached disk drive, as opposed to some web URL.

Does this make sense? You were trying to use a local file system operation on a website resource. Like trying to get to a campsite using a passenger train instead of a car. Both car and train are modes of transportation, but trains only go to train stops. The train can't get to the campsite.

Prateek Narula 41 Reputation points

2022-06-03T07:56:13.437+00:00

hello @MartinJaffer-MSFT , thanks for the reply.
Yes your hunch is correct the library is using os.path to get the file. And the idea is also is the right direction on mounting the storage so it can be treated as local stortage.

But now the question is how to mount the storage on Synapse analytics?
Is there any documentation that can help, I know i can do that on DataBricks but how to mount the storage in Azure Synapse Analytics?

MartinJaffer-MSFT 26,236

@Prateek Narula

https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/synapse-file-mount-api#how-to-mount-gen2blob-storage

If you have created a linked service for that storage account, you can then:

mssparkutils.fs.mount(   
    "abfss://mycontainer@<accountname>.dfs.core.windows.net",   
    "/test",   
    {"linkedService":"mygen2account"}   
)

If you want to use account key, and you have stored account key in a key vault, you can:

from notebookutils import mssparkutils    
  
accountKey = mssparkutils.credentials.getSecret("MountKV","mySecret")    
mssparkutils.fs.mount(    
    "abfss://mycontainer@<accountname>.dfs.core.windows.net",    
    "/test",    
    {"accountKey":accountKey}  
)

Share via

How can I read a .dcm file stored in ADLS gen2 from pyspark notebook in Azure Synapse?

0 additional answers

Your answer