Extra disk space in Azure Machine Learning Online Endpoint

Question

I need to deploy an LLM via AML Managed Online Endpoint but the machine that i want to use (standard_nc40ads_h100_v5) apparently has only 128 GB of disk space

even if the documentation says that it should have 3576 GiB https://learn.microsoft.com/en-us/azure/virtual-machines/ncads-h100-v5

How can i increase the disk space, maybe via mounting some external volume?

Answer 1

Hello Ferrero Alberto,

Thank you for sharing the details. I understand you’re deploying an LLM on an Azure Machine Learning managed online endpoint using the Standard_NC40ads_H100_v5 SKU, but you only see ~128 GB disk available, even though the VM-size documentation lists 3576 GiB local storage.

Why you see this difference

• The 3576 GiB shown for Standard_NC40ads_H100_v5 is the VM’s local temporary (host) storage when you deploy that SKU as an IaaS VM. This is documented under Local (temp) storage for the NCads H100 v5 series.
  Link - (NCads_H100_v5 size series - Azure Virtual Machines)
• In Azure ML managed online endpoints, the model is hosted inside a managed containerized environment, and the service does not expose the full VM local NVMe/temp disk to your container.

**Can the disk be increased or can an external volume be mounted?

** For managed online endpoints, there is no supported option today to increase that local disk size or attach/mount an additional Azure managed disk into the managed endpoint container environment.

Recommended options (supported paths)

1. If your model artifacts/weights fit within the ~128 GB limit:
   • Keep large artifacts in Azure Storage (e.g., Blob) and download required files at container startup, ensuring the total footprint on local disk remains under the limit. (General managed online endpoint concept:
     Link - https://learn.microsoft.com/en-us/azure/machine-learning/concept-endpoints-online?view=azureml-api-2
2. If your model weights are larger than ~128 GB (or you need persistent >128 GB storage):
   • A managed online endpoint is not the right hosting option for that storage requirement. Use one of the following instead:
   • Kubernetes online endpoint (AKS) so you control the cluster and can use persistent storage patterns, or
   • Deploy directly on an H100 VM / VMSS where you can attach/size disks as needed and leverage the SKU’s capabilities. (Endpoint type choices,
     VM local storage spec for the SKU:
     Link - https://learn.microsoft.com/en-us/azure/virtual-machines/sizes/gpu-accelerated/ncadsh100v5-series?tabs=sizebasic

Thanks,
Manish

Answer 2

hi Ferrero Alberto,

In a managed online endpoint in Azure Machine Learning you always get only 128 GB of disk. That limit is fixed. The 3.5 TB shown in the VM documentation is local NVMe on the physical host, but the managed service does not expose it to your container. You cannot increase the disk size and you cannot mount an extra volume.

If your model is larger than 128 GB, store the weights in Azure Blob Storage and download them when the container starts. This is the standard and supported approach. If the model is very large and you do not want to pay the cold start cost, then a managed online endpoint is the wrong tool. In that case use a Kubernetes online endpoint with a persistent volume, or run the model directly on a regular H100 VM where you control the disks.

lmk if its works for you,

rgds,

Alex