How to Upgrade the NVIDIA Driver Version for an AKS Cluster GPU Node

Question

How to Upgrade the NVIDIA Driver Version for an AKS Cluster GPU Node

Elvis Baugh 20

I need to upgrade the NVIDIA driver version on an AKS cluster GPU node from version 510 to version 535 to run my ML app compiled with CUDA version 12.2. Downgrading to the matching CUDA version breaks some of my TensorFlow packages. I followed the examples from this NVIDIA page but was unsuccessful. How can I successfully upgrade the NVIDIA driver version on an AKS cluster GPU node?

vipullag-MSFT 26,487 Reputation points Moderator

2023-12-19T04:56:50.77+00:00

Hello Elvis Baugh

Any update on the issue?

Just checking in to see if you got a chance to see previous response.

If the suggested response helped you resolve your issue, please 'Accept as answer', so that it can help others in the community looking for help on similar topics.
Elvis Baugh 20 Reputation points

2023-12-19T17:47:26.7266667+00:00

Hi Vipullag,

Sorry for the delay in response. The link you have provided me here was the same steps I had followed originally that took me to that page I have added in my previous comment. I chose option three (3) which was using the NVIDIA GPU Operator, and where it says, "When you follow this approach, you can install the Operator without any special considerations or arguments. Refer to Install NVIDIA GPU Operator." Thats how I ended up on that page and tried following those steps.

So are you saying instead I should try Installing the Operator for Preinstalled Driver and Toolkit instead?

Also does it matter what node size I am using in order to have the default nvidia driver version to be 535 and cuda 12.*? I am currently using the Standard_NV6ads_A10_v5. Since I require a node size for the purpose of ML, are there specific node size I need in order to run appplication for AI and deep-learning?
vipullag-MSFT 26,487 Reputation points Moderator

2024-01-03T01:40:30.61+00:00

Hello Elvis Baugh
Apologies in delayed response on this, I checked with internal team for on your query.
Based on your ask, looks like you are using the AKS GPU Image which is currently in preview.

Please refer to this link (https://learn.microsoft.com/en-us/azure/aks/gpu-cluster#options-for-using-nvidia-gpus) for the options in AKS as of today.
Using the NVIDIA device plugin via a Daemonset is what the majority of our customers are using today.

However, NVIDIA Operator is an encouraged approach. You can find details on the NVIDIA site, https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/microsoft-aks.html, for running it in AKS.
For more details, please check this repo https://github.com/NVIDIA/gpu-operator

Hope this helps.
vipullag-MSFT 26,487 Reputation points Moderator

2024-01-04T00:57:58.81+00:00

Hello Elvis Baugh

Any update on the issue?

Just checking in to see if you got a chance to see previous response.
Elvis Baugh 20 Reputation points

2024-01-05T07:04:50.4833333+00:00

Hi Vipullag,

Thank you for your response. I have tried the suggested approach and I am now able to successfully configure my AKS GPU image with the desired nvidia driver and cuda version.

Thank you!

Accepted answer

0 additional answers

Your answer

vipullag-MSFT 26,487 Reputation points Moderator

2023-12-19T04:56:50.77+00:00

Hello Elvis Baugh

Any update on the issue?

Just checking in to see if you got a chance to see previous response.

If the suggested response helped you resolve your issue, please 'Accept as answer', so that it can help others in the community looking for help on similar topics.
Elvis Baugh 20 Reputation points

2023-12-19T17:47:26.7266667+00:00

Hi Vipullag,

Sorry for the delay in response. The link you have provided me here was the same steps I had followed originally that took me to that page I have added in my previous comment. I chose option three (3) which was using the NVIDIA GPU Operator, and where it says, "When you follow this approach, you can install the Operator without any special considerations or arguments. Refer to Install NVIDIA GPU Operator." Thats how I ended up on that page and tried following those steps.

So are you saying instead I should try Installing the Operator for Preinstalled Driver and Toolkit instead?

Also does it matter what node size I am using in order to have the default nvidia driver version to be 535 and cuda 12.*? I am currently using the Standard_NV6ads_A10_v5. Since I require a node size for the purpose of ML, are there specific node size I need in order to run appplication for AI and deep-learning?
vipullag-MSFT 26,487 Reputation points Moderator

2024-01-03T01:40:30.61+00:00

Hello Elvis Baugh
Apologies in delayed response on this, I checked with internal team for on your query.
Based on your ask, looks like you are using the AKS GPU Image which is currently in preview.

Please refer to this link (https://learn.microsoft.com/en-us/azure/aks/gpu-cluster#options-for-using-nvidia-gpus) for the options in AKS as of today.
Using the NVIDIA device plugin via a Daemonset is what the majority of our customers are using today.

However, NVIDIA Operator is an encouraged approach. You can find details on the NVIDIA site, https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/microsoft-aks.html, for running it in AKS.
For more details, please check this repo https://github.com/NVIDIA/gpu-operator

Hope this helps.
vipullag-MSFT 26,487 Reputation points Moderator

2024-01-04T00:57:58.81+00:00

Hello Elvis Baugh

Any update on the issue?

Just checking in to see if you got a chance to see previous response.
Elvis Baugh 20 Reputation points

2024-01-05T07:04:50.4833333+00:00

Hi Vipullag,

Thank you for your response. I have tried the suggested approach and I am now able to successfully configure my AKS GPU image with the desired nvidia driver and cuda version.

Thank you!

Answer 1

Hello Elvis Baugh

Welcome to Microsoft Q&A Platform, thanks for posting your query here.

The nodes in AKS contains specific versioned software for its operation and interaction with Azure. Upgrading the version of component -outside of the supported node re-image [1]- may result in negative impact or unexpected consequences. It is not recommended.

This is a publicly accessible change log [2] which contains information about the latest in new releases. For example Release 2023-06-04 [3].

Based on your ask, looks like you are doing unsupported modifications in the cluster. The documentation [4] you referred is about an unsupported method.

Can you please check this document "Use GPUs for compute-intensive workloads on Azure Kubernetes Service (AKS)" [5] which explains how to install NVIDIA support in AKS.

Hope this helps.

Share via

How to Upgrade the NVIDIA Driver Version for an AKS Cluster GPU Node

0 additional answers

Your answer