Enable nested virtualization in Azure Lab Services
Article
Nested virtualization enables you to create a lab in Azure Lab Services that contains a multi-VM environment. To avoid that lab users need to enable nested virtualization on their lab VM and install the nested VMs inside it, you can prepare a lab template. When you publish the lab, each lab user has a lab VM that already contains the nested virtual machines.
Virtualization applications other than Hyper-V are not supported for nested virtualization. This includes any software that requires hardware virtualization extensions.
Important
Select Large (nested virtualization) or Medium (nested virtualization) for the virtual machine size when creating the lab. Nested virtualization will not work otherwise.
Prerequisites
An Azure account with an active subscription. If you don't have an Azure subscription, create a free account before you begin.
To enable nested virtualization on the template VM, you first connect to the VM by using a remote desktop (RDP) client. You can then apply the configuration changes by either running a PowerShell script or using Windows tools.
You can use a PowerShell script to set up nested virtualization on a template VM in Azure Lab Services. The following steps guide you through how to use the Lab Services Hyper-V scripts. The steps are intended for Windows Server 2016, Windows Server 2019, or Windows 10.
The script may require the machine to be restarted. Follow instructions from the script and re-run the script until Script completed is seen in the output.
Don't forget to reset the execution policy. Run the following command:
Set-ExecutionPolicy default -force
You can set up nested virtualization on a template VM in Azure Lab Services using Windows roles and tools directly. There are a few things needed on the template VM enable nested virtualization. The following steps describe how to manually set up a Lab Services machine template with Hyper-V. Steps are intended for Windows Server 2016 or Windows Server 2019.
The following steps describe the actions to enable Hyper-V on Windows Server using Server Manager. After enabling Hyper-V, Hyper-V manager is available to add, modify, and delete client VMs.
In Server Manager, on the Dashboard page, select Add Roles and Features.
On the Before you begin page, select Next.
On the Select installation type page, keep the default selection of Role-based or feature-based installation and then select Next.
On the Select destination server page, select Select a server from the server pool. The current server is already selected. Select Next.
On the Select server roles page, select Hyper-V.
The Add Roles and Features Wizard pop-up appears. Select Include management tools (if applicable). Select the Add Features button.
On the Select server roles page, select Next.
On the Select features page, select Next.
On the Hyper-V page, select Next.
On the Create Virtual Switches page, accept the defaults, and select Next.
On the Virtual Machine Migration page, accept the defaults, and select Next.
On the Default Stores page, accept the defaults, and select Next.
On the Confirm installation selections page, select Restart the destination server automatically if required.
When the Add Roles and Features Wizard pop-up appears, select Yes.
Select Install.
Wait for the Installation progress page to indicate that the Hyper-V role is complete. The machine may restart in the middle of the installation.
Select Close.
2. Enable the DHCP role
When you create a client VM, it needs an IP address in the Network Address Translation (NAT) network. Create the NAT network in a later step.
To assign the IP addresses automatically, configure the lab VM template as a DHCP server:
In Server Manager, on the Dashboard page, select Add Roles and Features.
On the Before you begin page, select Next.
On the Select installation type page, select Role-based or feature-based installation and then select Next.
On the Select destination server page, select the current server from the server pool and then select Next.
On the Select server roles page, select DHCP Server.
The Add Roles and Features Wizard pop-up appears. Select Include management tools (if applicable). Select Add Features.
Note
You may see a validation error stating that no static IP addresses were found. This warning can be ignored for our scenario.
On the Select server roles page, select Next.
On the Select features page, select Next.
On the DHCP Server page, select Next.
On the Confirm installation selections page, select Install.
Wait for the Installation progress page to indicate that the DHCP role is complete.
Select Close.
3. Enable the Routing and Remote Access role
Next, enable the Routing service to enable routing network traffic between the VMs on the template VM.
In Server Manager, on the Dashboard page, select Add Roles and Features.
On the Before you begin page, select Next.
On the Select installation type page, select Role-based or feature-based installation and then select Next.
On the Select destination server page, select the current server from the server pool and then select Next.
On the Select server roles page, select Remote Access. Select OK.
On the Select features page, select Next.
On the Remote Access page, select Next.
On the Role Services page, select Routing.
The Add Roles and Features Wizard pop-up appears. Select Include management tools (if applicable). Select Add Features.
Select Next.
On the Web Server Role (IIS) page, select Next.
On the Select role services page, select Next.
On the Confirm installation selections page, select Install.
Wait for the Installation progress page to indicate that the Remote Access role is complete.
Select Close.
4. Create virtual NAT network
Now that you've installed all the necessary roles, you can create the NAT network. The creation process involves creating a switch and the NAT network, itself.
A NAT network assigns a public IP address to a group of VMs on a private network to allow connectivity to the internet. In this case, the group of private VMs consists of the nested VMs. The NAT network allows the nested VMs to communicate with one another.
A switch is a network device that handles receiving and routing of traffic in a network.
Create a new virtual switch
To create a virtual switch in Hyper-V:
Open Hyper-V Manager from Windows Administrative Tools.
Select the current server in the left-hand navigation menu.
Select Virtual Switch Manager… from the Actions menu on the right-hand side of the Hyper-V Manager.
On the Virtual Switch Manager pop-up, select Internal for the type of switch to create. Select Create Virtual Switch.
For the newly created virtual switch, set the name to something memorable. For this example, we use 'LabServicesSwitch'. Select OK.
A new network adapter is created. The name is similar to 'vEthernet (LabServicesSwitch)'. To verify open the Control Panel, select Network and Internet, select View network status and tasks. On the left, select Change adapter settings.
Create a NAT network
To create a NAT network on the lab template VM:
Open the Routing and Remote Access tool from Windows Administrative Tools.
Select the local server in the left navigation page.
Choose Action -> Configure and Enable Routing and Remote Access.
When Routing and Remote Access Server Setup Wizard appears, select Next.
On the Configuration page, select Network address translation (NAT) configuration. Select Next.
Warning
Do not choose the 'Virtual private network (VPN) access and NAT' option.
On NAT Internet Connection page, choose 'Ethernet'. Don't choose the 'vEthernet (LabServicesSwitch)' connection we created in Hyper-V Manager. Select Next.
Select Finish on the last page of the wizard.
When the Start the service dialog appears, select Start Service.
Wait until service is started.
5. Update network adapter settings
Next, associate the IP address of the network adapter with the default gateway IP of the NAT network you created earlier. In this example, assign an IP address of 192.168.0.1, with a subnet mask of 255.255.255.0. Use the virtual switch that you created earlier.
Open the Control Panel, select Network and Internet, select View network status and tasks.
On the left, select Change adapter settings.
In the Network Connections window, double-click on 'vEthernet (LabServicesSwitch)' to show the vEthernet (LabServicesSwitch) Status details dialog.
Select the Properties button.
Select Internet Protocol Version 4 (TCP/IPv4) item and select the Properties button.
In the Internet Protocol Version 4 (TCP/IPv4) Properties dialog:
Select Use the following IP address.
For the IP address, enter 192.168.0.1.
For the subnet mask, enter 255.255.255.0.
Leave the default gateway and DNs servers blank.
Note
The range for the NAT network will be, in CIDR notation, 192.168.0.0/24. This range provides usable IP addresses from 192.168.0.1 to 192.168.0.254. By convention, gateways have the first IP address in a subnet range.
Select OK.
6. Create DHCP Scope
Next, you can add a DHCP scope. In this case, our NAT network is 192.168.0.0/24 in CIDR notation. This range provides usable IP addresses from 192.168.0.1 to 192.168.0.254. The scope you create must be in that range of usable addresses, excluding the IP address you assigned in the previous step.
Open Administrative Tools and open the DHCP administrative tool.
In the DHCP tool, expand the node for the current server and select IPv4.
From the Action menu, choose New Scope….
When the New Scope Wizard appears, select Next on the Welcome page.
On the Scope Name page, enter 'LabServicesDhcpScope' or something else memorable for the name. Select Next.
On the IP Address Range page, enter the following values.
192.168.0.100 for the Start IP address
192.168.0.200 for the End IP address
24 for the Length
255.255.255.0 for the Subnet mask
Select Next.
On the Add Exclusions and Delay page, select Next.
On the Lease Duration page, select Next.
On the Configure DHCP Options page, select Yes, I want to configure these options now. Select Next.
On the Router (Default Gateway)
Add 192.168.0.1, if not done already. Select Next.
On the Domain Name and DNS Servers page, add 168.63.129.16 as a DNS server IP address, if not done already. 168.63.129.16 is the IP address for an Azure static DNS server. Select Next.
On the WINS Servers page, select Next.
One the Activate Scope page, select Yes, I want to activate this scope now. Select Next.
On the Completing the New Scope Wizard page, select Finish.
You've now configured your template VM to use nested virtualization and create VMs inside it.
Troubleshooting
The Linux VM is only showing a black screen
Perform the following steps to verify your nested VM configuration:
Hyper-V doesn't start with error The virtual machine is using processor-specific xsave features not supported
This error can happen when a lab user leaves the Hyper-V VM in the saved state. You can right-select the VM in Hyper-V Manager and select Delete saved state.
Caution
Deleting the saved state means that any unsaved work is lost, but anything saved to disk remains intact.
This error can happen when the Hyper-V VM is turned off and the VHDX file is corrupted. If the lab user has created a backup of the VDHX file, or saved a snapshot, they can restore the VM from that point.
Increase the number vCPUs and memory that is assigned to the Hyper-V VM in Hyper-V Manager. The total number of vCPUs can't exceed the number of cores of the host VM (lab VM). If you're using variable memory, the default option, increase the minimum amount of memory assigned to the VM. The maximum amount of assigned memory (if using variable memory) can exceed the amount of memory of the host VM. This allows greater flexibility when having to complete intensive operations on just one of the Hyper-V VMs.
If you're using the Medium (Nested Virtualization) VM size for the lab, consider using the Large (Nested Virtualization) VM size instead to have more compute resources for each lab VM.
Internet connectivity isn't working for nested VMs
Confirm that you followed the previous steps for enabling nested virtualization. Consider using the PowerShell script option.
If you're running a system administration class, consider not using the host VM (lab VM) as the DHCP server.
Changing the settings of the lab VM can cause issues with other lab VMs. Create an internal or private NAT network and have one of the VMs act as the DHCP, DNS, or domain controller. Using private over internal does mean that Hyper-V VMs don't have internet access.
Check the network adapter settings for the Hyper-V VM:
Set the IP address of the DNS server and DHCP server to 168.63.129.16.
The ping command from a Hyper-V VM to the host VM doesn't work. To test internet connectivity, launch a web browser and verify that the web page loads correctly.