Azure Machine Learning Cluster Compute Data Vaniches

Question

Azure Machine Learning Cluster Compute Data Vaniches

Haseeb Sultan 20

I have configured GPU based Compute Cluster in Azure Machine Learning Workspace. After completing some work , when I turn that off and restart all my data vanishes. Any suggestions?

Also how do I attach a Azure blob storage to the compute cluster so that all my work gets stored there automatically?

VasaviLankipalle-MSFT 18,681 Reputation points Moderator

2023-05-02T22:53:23.83+00:00

Hi @Haseeb Sultan , Thanks for using Microsoft Q&A Platform.

Yes, when the compute cluster is no longer in use, it can be deallocated. Any data stored on the cluster is lost when it is deallocated. To avoid this, you can save your data in Azure Blob Storage.

You can create an Azure Blob Storage container and attach it to your compute cluster as a mounted folder.

Using Azure Machine Learning datastores we can link the existing storage account for Azure Machine Learning use.

Here is the documentation for your reference: https://learn.microsoft.com/en-us/azure/machine-learning/how-to-datastore?view=azureml-api-2&tabs=cli-identity-based-access%2Ccli-adls-identity-based-access%2Ccli-azfiles-account-key%2Ccli-adlsgen1-identity-based-access#attach-a-datastore-to-a-compute-target
Haseeb Sultan 20 Reputation points

2023-05-03T13:19:21.3866667+00:00

I created a datastore and in the workspace under "Data" tab , I have linked the container of blob storage.
Now when I use compute instance, it still stores data under its own directory not the blob storage
VasaviLankipalle-MSFT 18,681 Reputation points Moderator

2023-05-03T20:27:46.02+00:00

@Haseeb Sultan , check to see if the Azure Blob Storage container is the default storage location for your compute instance. If not, then you are required to mount the Blob Storage container to the compute instance.
Haseeb Sultan 20 Reputation points

2023-05-03T20:30:16.82+00:00

In the datastore, I made the azure blob storage as default. Hope that's the way to do it.

Also Can you guide me how to mount the drive?

Your answer

VasaviLankipalle-MSFT 18,681 Reputation points Moderator

2023-05-02T22:53:23.83+00:00

Hi @Haseeb Sultan , Thanks for using Microsoft Q&A Platform.

Yes, when the compute cluster is no longer in use, it can be deallocated. Any data stored on the cluster is lost when it is deallocated. To avoid this, you can save your data in Azure Blob Storage.

You can create an Azure Blob Storage container and attach it to your compute cluster as a mounted folder.

Using Azure Machine Learning datastores we can link the existing storage account for Azure Machine Learning use.

Here is the documentation for your reference: https://learn.microsoft.com/en-us/azure/machine-learning/how-to-datastore?view=azureml-api-2&tabs=cli-identity-based-access%2Ccli-adls-identity-based-access%2Ccli-azfiles-account-key%2Ccli-adlsgen1-identity-based-access#attach-a-datastore-to-a-compute-target
Haseeb Sultan 20 Reputation points

2023-05-03T13:19:21.3866667+00:00

I created a datastore and in the workspace under "Data" tab , I have linked the container of blob storage.
Now when I use compute instance, it still stores data under its own directory not the blob storage
VasaviLankipalle-MSFT 18,681 Reputation points Moderator

2023-05-03T20:27:46.02+00:00

@Haseeb Sultan , check to see if the Azure Blob Storage container is the default storage location for your compute instance. If not, then you are required to mount the Blob Storage container to the compute instance.
Haseeb Sultan 20 Reputation points

2023-05-03T20:30:16.82+00:00

In the datastore, I made the azure blob storage as default. Hope that's the way to do it.

Also Can you guide me how to mount the drive?

Share via

Azure Machine Learning Cluster Compute Data Vaniches

Your answer