Can I attach NVMe to an AzureML cluster node?

Question

In my Azure Machine Learning workspace, I am using a managed compute cluster with VM size Standard_NC80adis_H100_v5.

According to the official VM documentation, this SKU includes multiple ephemeral local NVMe disks, which are usable on a “raw” Azure VM (where they can be formatted and mounted during VM initialization).

My question is specifically about AzureML managed compute clusters:

Is there a supported way to format and mount these local NVMe disks during the node preparation / provisioning phase of an AzureML compute cluster, so that they are available to AzureML jobs for local data storage or caching during job execution?

So far, attempts to mount /dev/nvme* from inside the job container fail with filesystem / superblock errors, which suggests that:

• the disks are either unformatted, or
• mounting them at job runtime is too late / not supported

I am trying to understand whether:

1. Using local NVMe disks on Standard_NC80adis_H100_v5 is supported at all on AzureML managed compute clusters, and if so, how
2. Or whether access to NVMe is only possible when using:
   • attached VMs / compute instances, or
   • non-AzureML-managed virtual machines

Ultimately, the goal is to use the NVMe disks as fast local scratch storage (e.g. for dataset caching) during training jobs.

Any clarification on the supported and unsupported scenarios would be appreciated.

Answer 1

Using local NVMe disks on Azure Machine Learning (AzureML) managed compute clusters, such as the Standard_NC80adis_H100_v5, is not explicitly supported for formatting and mounting during the node preparation or provisioning phase. The documentation indicates that while these VMs do include ephemeral local NVMe disks, the ability to format and mount these disks for AzureML jobs may not be feasible within the managed compute environment.

1. Support for Local NVMe Disks: The local NVMe disks are primarily designed for use in raw Azure VMs where they can be formatted and mounted. In the context of AzureML managed compute clusters, there is no clear guidance on whether these disks can be utilized in the same manner. The typical usage scenario for AzureML managed clusters does not include direct access to local NVMe disks for job execution.
2. Access Limitations: Access to NVMe disks is generally more straightforward when using attached VMs or non-AzureML-managed virtual machines. In managed compute clusters, the environment is abstracted, and direct manipulation of the underlying VM disks is limited, which may lead to the filesystem or superblock errors you encountered when attempting to mount the NVMe disks during job execution.

Ultimately, if your goal is to use NVMe disks for fast local scratch storage during training jobs, you may need to consider using non-managed compute instances where you have full control over the VM and can format and mount the NVMe disks as needed.