AzureML Compute Instance Mount Data (SDK V2)

Question

I am trying to access my blob data store in an AzureML Compute Instance (VM) in a Jupyter Notebook.

For this, I want to avoid downloading the data to disk, but rather mount the blob data storage and then access the data as with native file access. I am using Identity based access in AzureML+Storage Account with disabled shared access keys.

What is the recommended way to achieve this with the AzureML V2 SDK?

I am able to achieve this via a workaround with BlobClient and io streaming, however, I would prefer to find a simpler way using mount and then passing native file paths to read.

Answer 1

Hi rsaw

Permissions for users/computes/workspace need to use managed identity are

1. Storage blob data contributor.
2. Storage file system privileged data contributor.

Reference thread - https://learn.microsoft.com/en-us/answers/questions/5685206/cant-access-aml-notebooks

Please note that all artifacts like model, training dataset are storage containers only not in compute storage.

Reference - https://learn.microsoft.com/en-us/azure/machine-learning/concept-compute-instance?view=azureml-api-2#accessing-files

Below documentation suggested to access storage using file system

1. https://learn.microsoft.com/en-us/azure/machine-learning/how-to-access-data-interactive?view=azureml-api-2&tabs=adls#access-data-from-a-datastore-uri,-like-a-filesystem
2. https://learn.microsoft.com/en-us/azure/machine-learning/how-to-read-write-data-v2?view=azureml-api-2&tabs=python

Simplest to get usage code is to locate the file from "explore" option and click "Copy usage code"

You can set a CRON job to clean the Cache built in computes (to avoid out of memory issues)

https://learn.microsoft.com/en-us/azure/machine-learning/how-to-schedule-pipeline-job?view=azureml-api-2&tabs=cliv2

Please let me know if you want clarity on these pointers.

Thank you