Do the Nv-v4 series VMs on Azure support nested virtualization yet. I don't mind having to install Windows 11 (I read somewhere that the virtualization feature on AMD hardware would only be rolled out in Windows 11), as long as Microsoft supplies the GPU drivers for the same. For a long time, there was a driver issue with Nv-v4 series VMs; I hope that doesn't repeat with Windows 11. I require Docker AND GPU; Docker requires Hyper V; Hyper V requires hardware-enabled nested virtualization. For almost a year I had been using a d-v3 VM (Standard D4s v3, to be specific) that supported virtualization pretty well. I was hoping they'd bring that support to the N class VMs too someday, but it has taken quite a while now. When might we expect it to happen? Or do they already? Microsoft Learn on the subject seem to imply that the N class doesn't support it yet.

Hello @The A Man , Welcome to Microsoft Q&A Platform. Thank you for reaching out & hope you are doing well. I understand that your requirement is to have Docker AND GPU in your Azure VM and would like to know if Nv-v4 series VMs on Azure support nested virtualization yet. I checked with the Azure VM Product group team and they have provided the below update: NVv4 or any other N series VMs do not support nested virtualization. Nested ParaV support is on the roadmap for future NV SKUs but not for NVv4. On the Nested virtualization side we have introduced GPU partitioning with NVv4. The smaller GPU VM sizes in NVv4 provide better benefit compared to nested virtualization for GPU sharing because GPU-P is done at the hardware level. Please refer : https://azure.microsoft.com/en-in/blog/power-your-azure-gpu-workstations-with-flexible-gpu-partitioning/ Kindly let us know if the above helps or you need further assistance on this issue. ---------------------------------------------------------------------------------------------------------------- Please " Accept the answer " if the information helped you. This will help us and others in the community as well.

Nested Virtualization With GPU on Azure

Accepted answer

GitaraniSharma-MSFT 49,701 Reputation points Microsoft Employee

2022-03-10T09:32:49.367+00:00

Hello @The A Man ,

Welcome to Microsoft Q&A Platform. Thank you for reaching out & hope you are doing well.

I understand that your requirement is to have Docker AND GPU in your Azure VM and would like to know if Nv-v4 series VMs on Azure support nested virtualization yet.

I checked with the Azure VM Product group team and they have provided the below update:
NVv4 or any other N series VMs do not support nested virtualization. Nested ParaV support is on the roadmap for future NV SKUs but not for NVv4.

On the Nested virtualization side we have introduced GPU partitioning with NVv4. The smaller GPU VM sizes in NVv4 provide better benefit compared to nested virtualization for GPU sharing because GPU-P is done at the hardware level.
Please refer : https://azure.microsoft.com/en-in/blog/power-your-azure-gpu-workstations-with-flexible-gpu-partitioning/

Kindly let us know if the above helps or you need further assistance on this issue.

----------------------------------------------------------------------------------------------------------------

Please "Accept the answer" if the information helped you. This will help us and others in the community as well.
Please sign in to rate this answer.

1 person found this answer helpful.
The A Man 71 Reputation points

2022-03-10T11:03:14.193+00:00

By the way, just the other day I was able to run Nv-v4 VM with Windows 11 image and install the Microsoft-provided Windows 10 AMD drivers and have the GPU recognized and function from the OS. Your official docs are a bit outdated; the docs said Nv class VMs do not support Windows 11 yet when actually they do. [Although, technically speaking, the hardware should not have been capable of running Windows 11 (or at least officially supported), which is kinda funny, you know; Microsoft's very own Windows 11 doesn't support their own Azure hardware; LOL. Obviously, one can run a dedicated Windows 11 machine from Azure, but who would do that, knowing how costly that could get.]

GitaraniSharma-MSFT 49,701 Reputation points Microsoft Employee

2022-03-10T11:46:21.24+00:00

Hello @The A Man ,

Thank you for your feedback.

AFAIK, Windows 11 is now generally available on Azure Virtual Desktop. And configuring graphics processing unit (GPU) acceleration for Azure Virtual Desktop is supported on Windows 10 version 1511 or newer OS using Azure's NV-series, NVv3-series, NVv4-series or NCasT4_v3-series VM sizes.
Refer : https://azure.microsoft.com/en-us/updates/windows-11-is-now-generally-available-on-azure-virtual-desktop/
https://learn.microsoft.com/en-us/azure/virtual-desktop/configure-vm-gpu

Let me discuss your scenario with the Product group team and get back to you with an update. If the official docs are outdated, I will provide the feedback to the backend team for appropriate doc updates.

Regards,
Gita

The A Man 71 Reputation points

2022-03-10T15:17:59.767+00:00

Thanks for your timely replies, really.

I did look into Azure Virtual Desktops a while ago; the problem is, I'm not a team, I'm just a single developer trying to streamline my remote desktop workflow, and Azure Virtual Desktops (or Windows Virtual Desktops, as they used to call it) seems like an overkill when the intended user is just one person as opposed to a team. I've been doing remote development on Azure for more than a year now, using two VMs (D4s-v3 for nested virtualization and Nv4as-v4 for Unreal Engine). As far as I know, you still can't do nested virtualization on Azure Virtual Desktops, or can you? Not to mention system-hibernation isn't supported on AVD (Azure Virtual Desktops) either. So there's really very little incentive to move to AVD as of now. If Azure Virtual Desktop offers support for hibernation, I'd move instantly. But I guess even hibernation support, if it ever comes, will land on Virtual Machines first and AVD later.

Please correct me if I'm wrong.

---------------------------------------

This is the docs article that implicitly says the driver doesn't work on Windows 11 when it actually does: https://learn.microsoft.com/en-us/azure/virtual-machines/windows/n-series-amd-driver-setup
Sure, those Azure machines do not meet the Secure Boot and TPM requirements (and in some cases, even the processor) requirements, but you can always install Windows from the image directly.

GitaraniSharma-MSFT 49,701 Reputation points Microsoft Employee

2022-03-11T18:43:10.303+00:00

Hello @The A Man ,

Apologies for the delay in my response.

Please find the update from the Azure Product Group team regarding Windows 11 support on Nv-v4 VM:

Windows 11 support is going through e2e (end to end) validations and we will officially announce the support when we are ready. The doc pages specify the officially supported version, but it does not mean that other versions don’t work at all.

Now coming back to nested virtualization support on Azure Virtual Desktops. It is not officially supported which means it may work with the nested virtualization supported VMs & OS but if it breaks, official support won't be provided.
AVD is ultimately a VM that has been added into a hostpool. So if all the pre-reqs are met, you can enable nested virtualization on AVD.

The Pre-requisites for nested virtualization: https://learn.microsoft.com/en-us/virtualization/hyper-v-on-windows/user-guide/nested-virtualization#prerequisites
AVD supports Windows Server 2016/Windows 10 or greater OS.
Refer : https://learn.microsoft.com/en-us/azure/virtual-desktop/overview#supported-virtual-machine-os-images
And you can find the Hyper-threaded and capable of running nested virtualization VMs in this list : https://learn.microsoft.com/en-us/azure/virtual-machines/acu

To answer your last question, system-hibernation feature is currently not available for Azure VMs. At present, we do not have an ETA when this feature will be available.
You can upvote the below feedback requesting this feature.
https://feedback.azure.com/d365community/idea/a9516e53-be25-ec11-b6e6-000d3a4f0f1c

The A Man 71 Reputation points

2022-03-11T20:11:54.077+00:00

Wow, you're the best! Thanks for the detailed reply; that answers all my queries.

Upvoted the feature feedback for hibernation. Seeing from that page it looks like it's been five years since that feature was requested. Just the other day I dug deeper into why hibernation is hard for a cloud vendor to deliver; basically, hibernation on our personal computers works because the saved memory image is guaranteed to be loaded on the same hardware it was saved from. That's not the case on the cloud. If a cloud vendor can guarantee that all machine hardware in their data centres is exactly identical (even the serial numbers of the hardware devices) then maybe, just maybe, hibernation might work without issues; anyway, the real devs know way better than me; who am I to comment. Let's just hope the feature does get delivered one day...

Thanks for your answers. I posted the same questions on the Microsoft community website, on Reddit and here, all over a week ago, and you're the only one who even responded. Thank you so so much. Your effort is much appreciated.

The A Man

GitaraniSharma-MSFT 49,701 Reputation points Microsoft Employee

2022-03-14T07:34:52.593+00:00

Happy to help! And appreciate your feedback.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Nested Virtualization With GPU on Azure

0 additional answers

Your answer