Share via

Troubleshoot NFS Azure Files persisten volume mount for Azure Container App using Consumption-GPU-NC24-A100 workload profile

Daniel 0 Reputation points
2025-06-26T10:04:39.8733333+00:00

I'm currently working on deploying an Azure Container App that uses Azure Files (NFS) as a persistent volume mount. My final goal is to deploy a custom large image generation model and expose an inference endpoint. I want to use NFS to store the model weights and optimize the startup time (currently I download them from blob at each start up).

To test this setup, I'm using the following Docker container:

FROM python:3.10-slim-bookworm

WORKDIR /app

COPY basic_server.py /app/basic_server.py

RUN pip install flask

EXPOSE 8080

ENTRYPOINT ["python", "-u", "basic_server.py"]
from flask import Flask, jsonify

app = Flask(__name__)


@app.route("/liveness", methods=["GET"])
def liveness():
    return jsonify({"status": "alive"}), 200


@app.route("/startup", methods=["GET"])
def startup():
    return jsonify({"status": "started"}), 200


@app.route("/readiness", methods=["GET"])
def readiness():
    return jsonify({"status": "ready"}), 200


@app.route("/an", methods=["GET"])
def an():
    return jsonify({"message": "Hello from the server!"}), 200


if __name__ == "__main__":
    app.run(host="0.0.0.0", port=8080)

The application has been deployed with identical configurations across two container apps, differing only in the workload profile used:

  1. test-mount using the standard Consumption workload profile – ✅ Works as expected
  2. test-mount-on-gpu using the Consumption-GPU-NC24-A100 workload profile – ❌ Fails at runtime

Configuration Summary

Subnet and vNet integrations are properly set up to allow access to the NFS mount.

The NFS volume mount is configured with the following recommended options:

vers=4,minorversion=1,sec=sys,nconnect=4

Observed Behavior

  • For test-mount, everything works as expected: I can connect to the container via Azure Portal Console, list the NFS directory, read/write files, etc.
    User's image
  • For test-mount-on-gpu, the container fails to start, and I receive the following error message:
{
    "TimeStamp": "2025-06-26 09:57:38.6804462 \u002B0000 UTC",
    "Type": "Warning",
    "ContainerAppName": "test-mount-on-gpu",
    "RevisionName": "test-mount-on-gpu--0000011",
    "ReplicaName": "test-mount-on-gpu--0000011-5795754cc7-6sm6k",
    "Msg": "Container \u0027test-mount-on-gpu\u0027 was terminated with exit code \u0027\u0027 and reason \u0027VolumeMountFailure\u0027. Shell command exited with non-zero status code. StatusCode = 32 | StdOut =  | StdErr = mount: /podr/volume/cda42832b5354e8bbe2f51742357a757: bad option; for several filesystems (e.g. nfs, cifs) you might need a /sbin/mount.\u003Ctype\u003E helper program.\n",
    "Reason": "ContainerTerminated",
    "EventSource": "ContainerAppController",
    "Count": 1
}

My Understanding / Hypothesis

It seems that the GPU-based Consumption workload environment may be missing some required utilities or kernel modules for mounting NFS volumes. This would explain why the same setup works in a standard container app but fails on the GPU profile. I have tried to install nfs-common and nfs-kernel-server in the docker image, but it doesn't work either.

Request for Support

Is NFS volume mounting supported for GPU Consumption workload profiles?

If so, are there additional steps or configurations required to enable it (e.g., installing extra packages, using a specific image base, enabling a preview feature)?

If not supported, is there an ETA or alternative workaround for using persistent storage in GPU-based Azure Container Apps?

Any guidance or official clarification from the Azure team would be greatly appreciated, as GPU workloads often depend on fast and persistent storage like NFS.

Azure Container Apps
Azure Container Apps

An Azure service that provides a general-purpose, serverless container platform.

0 comments No comments

1 answer

Sort by: Most helpful
  1. Ranashekar Guda 2,905 Reputation points Moderator
    2025-06-27T18:42:00.7133333+00:00

    Hello @Daniel,
    Based on the error and your testing, it looks like the issue is specific to the Consumption-GPU-NC24-A100 workload profile. While your setup works fine on the standard Consumption profile, the GPU version fails due to a VolumeMountFailure, which points to a problem with mounting the NFS Azure File share.

    This happens because NFS volume mounting is currently not supported on GPU-based Consumption workload profiles in Azure Container Apps. These environments are more restricted and may lack the required system utilities or kernel support for NFS mounts. Installing NFS packages inside the container doesn’t help here, since the issue lies in the host environment controlled by Azure.

    As a workaround, we recommend using Azure Blob Storage with caching to store your model weights. You can download them at startup to local disk (like /tmp) to reduce startup time. Alternatively, if you need full NFS support with GPU workloads, consider using Azure Kubernetes Service (AKS), which allows custom storage configurations, or Azure Machine Learning for more managed GPU environments with persistent storage options.

    Kindly refer below links:
    Use storage mounts in Azure Container Apps
    https://azure.microsoft.com/en-au/updates?id=public-preview-nfs-azure-files-volume-mount-support-in-azure-container-apps&utm_source=chatgpt.com

    Hope this helps. Do let us know if you any further queries.

    Was this answer helpful?

    1 person found this answer helpful.

Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.