Azure Machine Learning
An Azure machine learning service for building and deploying models.
3,189 questions
Azure AutoML - deep learning job fails with "unknown or invalid runtime name: nvidia" error
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(220.8, 59%, 31%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVP%3C/text%3E%3C/svg%3E)
Volodymyr Petryshyn
10
Reputation points
Hello,
I'm experimenting with Azure AutoML and trying to setup job with enabled deep learning,
I've created a compute instance using
Standard_NC6s_v3 (6 cores, 112 GB RAM, 736 GB disk) which has NVIDIA Tesla,
and tried to run AutoML on it, but my jobs are failing with error:
Failed to execute command group with error API queried with a bad parameter: {"message":"unknown or invalid runtime name: nvidia"}
I've also tried to run AutoML with deep learning on Serverless GPU and my AutoML job was stuck on "Setting up the run" status for 19 hours, so I canceled it because I saw in the costs report that Azure will bill it.
Please share your thoughts how to make it work
Thank you
{count} vote
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(278.4, 1%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EYM%3C/text%3E%3C/svg%3E)
YutongTie-MSFT 53,931 Reputation points2024-12-07T05:21:07.8166667+00:00 @Volodymyr Petryshyn Thanks for reaching out to us, I have seen the same issue before, could you please check if you have installed nvidia container toolkit? If not, please installed it -
sudo apt-get install -y nvidia-container-toolkit
I hope this helps.
Regards,
Yutong
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(262.40000000000003, 56.00000000000001%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVP%3C/text%3E%3C/svg%3E)
Volodymyr Petryshyn 0 Reputation points2024-12-07T11:17:58.7033333+00:00 Thank you for your response, I've installed it, the terminal output
sudo apt-get install -y nvidia-container-toolkit Reading package lists... Done Building dependency tree Reading state information... Done The following additional packages will be installed: libnvidia-container-tools libnvidia-container1 nvidia-container-toolkit-base The following NEW packages will be installed: libnvidia-container-tools libnvidia-container1 nvidia-container-toolkit nvidia-container-toolkit-base 0 upgraded, 4 newly installed, 0 to remove and 10 not upgraded. Need to get 5787 kB of archives. After this operation, 27.6 MB of additional disk space will be used. Get:1 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64 libnvidia-container1 1.16.2-1 [926 kB] Get:2 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64 libnvidia-container-tools 1.16.2-1 [21.4 kB] Get:3 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64 nvidia-container-toolkit-base 1.16.2-1 [3654 kB] Get:4 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64 nvidia-container-toolkit 1.16.2-1 [1185 kB] Fetched 5787 kB in 0s (30.1 MB/s) Selecting previously unselected package libnvidia-container1:amd64. (Reading database ... 173435 files and directories currently installed.) Preparing to unpack .../libnvidia-container1_1.16.2-1_amd64.deb ... Unpacking libnvidia-container1:amd64 (1.16.2-1) ... Selecting previously unselected package libnvidia-container-tools. Preparing to unpack .../libnvidia-container-tools_1.16.2-1_amd64.deb ... Unpacking libnvidia-container-tools (1.16.2-1) ... Selecting previously unselected package nvidia-container-toolkit-base. Preparing to unpack .../nvidia-container-toolkit-base_1.16.2-1_amd64.deb ... Unpacking nvidia-container-toolkit-base (1.16.2-1) ... Selecting previously unselected package nvidia-container-toolkit. Preparing to unpack .../nvidia-container-toolkit_1.16.2-1_amd64.deb ... Unpacking nvidia-container-toolkit (1.16.2-1) ... Setting up nvidia-container-toolkit-base (1.16.2-1) ... Setting up libnvidia-container1:amd64 (1.16.2-1) ... Setting up libnvidia-container-tools (1.16.2-1) ... Setting up nvidia-container-toolkit (1.16.2-1) ... Processing triggers for libc-bin (2.31-0ubuntu9.16) ...but job fails with the same error, I also tried to restart the instance and create a new AutoML job but the same result
Failed to execute command group with error API queried with a bad parameter: {"message":"unknown or invalid runtime name: nvidia"}
Sign in to comment
Sign in to answer