Prepare your nested virtualization environment for AKS Edge Essentials

Article
11/28/2023

This article describes how to set up a nested virtualization environment to deploy an Azure Kubernetes Service (AKS) Edge Essentials cluster.

Note

Deploying AKS Edge Essentials on top of a nested virtualization environment on VMware ESXi is supported per VMware KB2009916. Other nested virutalization deployments are not supported for production scenarios and are limited to developer purposes. This guide assumes you're using the Hyper-V hypervisor. We do not support using a non-Microsoft hypervisor, such as KVM.

Prerequisites

See the system requirements.
OS requirements: install Windows 10/11 IoT Enterprise/Enterprise/Pro on your machine and activate Windows. We recommend using the latest client version 22H2 (OS build 19045) or Server 2022 (OS build 20348). You can download a version of Windows 10 here or Windows 11 here.

Deployment on Windows VM on VMware ESXi

VMware ESXi 7.0 and 8.0 versions can host AKS Edge Essentials on top of a Windows virtual machine. See VMware KB2009916 for more information about VMware ESXi nested virtualization support.

To set up AKS Edge Essentials on a VMware ESXi Windows virtual machine, use the following steps:

Create a Windows virtual machine on the VMware ESXi host. For more information about VMware VM deployment, see VMware - Deploying Virtual Machines.

Note

If you're creating a Windows 11 virtual machine, ensure that it meets the minimum requirements by Microsoft to run Windows 11. For more information about Windows 11 VM VMware support, see Installing Windows 11 as a guest OS on VMware.

Turn off the virtual machine created in previous step.
Select the Windows virtual machine and then Edit settings.
Search for Hardware virtualization and turn on Expose hardware assisted virtualization to the guest OS.
Select Save and start the virtual machine.
Install the Hyper-V hypervisor. If you're using Windows client, make sure you Install Hyper-V on Windows 10. If you're using Windows Server, make sure you install the Hyper-V role.

Deployment on Azure virtual machines

If you're running AKS Edge Essentials on top of an Azure VM, ensure that you use an Azure Compute Unit (ACU) that supports nested virtualization. For more information, see Azure Compute Unit (ACU). Also, Azure VMs don't support using an external virtual switch, so AKS Edge Essentials deployments on top of the VM host OS are limited to single-machine clusters.

Deployment environment

This section describes the nested architecture, highlighting the main components and configuration needed. The levels of virtualization described later in this article are:

L0 Host OS: Windows host operating system. This OS can be running bare-metal or as a virtual machine, but in this article, the L0 Host OS is the one that creates the nested Windows L1 VMs.
L1 Virtual Machine: Windows VM running on top of L0 Windows host OS. This VM has the AKS Edge Essentials installation.
L2 Virtual Machine: AKS Edge Essentials nested VM (Linux or Windows) running on top of the L1 Windows virtual machine.

You can set up a nested environment using an internal or external virtual switch. However, this article assumes you're using an internal virtual switch. IP addresses of the L0 Windows machine and L1/L2 virtual machines can change depending on the networking scenario. This article assumes that you're using the 172.20.1.0/24 IP address family.

Also, deploying AKS Edge Essentials Windows nodes is optional, and impacts your assigned memory requirements. This article describes a Linux-only deployment, but you can add your Windows nodes by adding the appropriate configuration to the deployment JSON files.

Tip

If you're using external virtual switches for the deployment, make sure you use the correct network adapters and IP addresses allocations.

The previous diagram shows the different virtual machines and components of this nested architecture.

Device/VM	OS	Level	Parent	Memory	IP Address
Windows host OS	Windows	L0	-	32/64 GB	172.20.1.1
Windows-VM-1	Windows	L1	Windows host OS	8/16 GB	172.20.1.2
Windows-VM-2	Windows	L1	Windows host OS	8/16 GB	172.20.1.3
AKS-Edge-Linux-VM-1	CBL-Mariner	L2	Windows-VM-1	4 GB	172.20.1.4
AKS-Edge-Windows-VM-1 (optional)	Windows	L2	Windows-VM-1	4 GB	172.20.1.5
AKS-Edge-Linux-VM-2	CBL-Mariner	L2	Windows-VM-2	4 GB	172.20.1.6
AKS-Edge-Windows-VM-2 (optional)	Windows	L2	Windows-VM-2	4 GB	172.20.1.7

Set up nested virtual machines

The following guide is an example of IP address allocation. You can use your own allocation based on your network environment and requirements. Naming conventions for virtual machines and virtual hardware assignments are also suggested, but you can use your own configuration.

Open an elevated PowerShell session.

Create an internal virtual switch:

New-VMSwitch -Name "AKS-Int" -SwitchType Internal

If you created the switch correctly, you should see something similar to the following output:

Name    SwitchType NetAdapterInterfaceDescription
----    ---------- ------------------------------
AKS-Int Internal

Assign an IP address to the AKS-Int virtual switch. This example uses the 172.20.1.0/24 network. The Windows host OS uses this virtual switch IP address to communicate with the other virtual machines and AKS Edge Essentials nodes:
```
$ifIndex = (Get-NetAdapter -Name "vEthernet (AKS-Int)").ifIndex
New-NetIPAddress –IPAddress "172.20.1.1" -PrefixLength "24" -InterfaceIndex $ifIndex
```
Create a NAT table for connecting the internal virtual switch and the internal network connected devices with the external/Internet network:
```
 New-NetNat -Name "AKS-EE-Int-Network" -InternalIPInterfaceAddressPrefix "172.20.1.0/24"
```
Using Hyper-V Manager, create the first Windows virtual machine and name it Windows-VM-1. For more information about virtual machine creation, see Windows Server virtualization. During the configuration of the VM, make sure you correctly set up the following parameters:
- Processors: Number of virtual processors: 4
- Memory: RAM: 8192 MB
- Network adapter: Virtual switch: AKS-Int
When Windows setup and configuration are finished, turn off your Windows-VM-1 virtual machine.
Enable nested virtualization for Windows-VM-1. For more information about nested virtualization, see Run Hyper-V in a Virtual Machine with Nested Virtualization:
```
Set-VMProcessor -VMName <windows-vm-name> -ExposeVirtualizationExtensions $true
```

Enable MAC spoofing for Windows-VM-1:

Get-VMNetworkAdapter -VMName <windows-vm-name> | Set-VMNetworkAdapter -MacAddressSpoofing On

Turn on the Windows-VM-1 virtual machine and connect using the Hyper-V Connect option.
Inside the Windows-VM-1 virtual machine, open an elevated PowerShell session.

Check the adapters connected to the virtual machine. Look for the name of the connected Ethernet adapter <name>. For example, Ethernet 2:

ipconfig

You should see something similar to the following output:

PS C:\Windows\system32> ipconfig

Windows IP Configuration

Ethernet adapter Ethernet 2:

   Connection-specific DNS Suffix  . : mshome.net
   Link-local IPv6 Address . . . . . : fe80::f4b3:63b3:20d0:2e60%12
   Autoconfiguration IPv4 Address. . : 169.254.233.233
   Subnet Mask . . . . . . . . . . . : 255.255.0.0
   Default Gateway . . . . . . . . . :

Ethernet adapter vEthernet (Default Switch):

   Connection-specific DNS Suffix  . :
   Link-local IPv6 Address . . . . . : fe80::545e:7d5:812b:c17b%18
   IPv4 Address. . . . . . . . . . . : 172.21.144.1
   Subnet Mask . . . . . . . . . . . : 255.255.240.0
   Default Gateway . . . . . . . . . :

Get the ifIndex of the Ethernet adapter with the name from previous step:
```
$ifIndex = (Get-NetAdapter -Name "<name>").ifIndex
```
Configure static IP 172.20.1.2 and gateway IP 172.20.1.1 and DNS Server 172.20.1.1:

Tip

If you are using an Azure VM, use the Windows host OS (L0) DNS server. Use the ipconfig /all command to get the DNS server address. Check that you're able to get internet access using your web browser. If you have no access, check the if the DNS server is correctly configured:
```
New-NetIPAddress –IPAddress "172.20.1.2" -DefaultGateway "172.20.1.1" -PrefixLength $ifIndex
Set-DNSClientServerAddress –InterfaceIndex $ifIndex –ServerAddresses "172.20.1.1"
```
Set up the Windows-VM-1 virtual machine, following the Prepare your machines for AKS Edge Essentials procedure.
Install all AKS Edge Essentials prerequisites for the Windows-VM-1 virtual machine. For more information about prerequisites, see AKS Edge Essentials requirements and support matrix:
```
Install-AksEdgeHostFeatures
```
Create the AKS Edge Essentials full-scale deployment configuration file. Once the JSON is created, add the required parameters. Make sure to change the network adapter for external virtual switch creation, DNS server, assigned memory, and correct static IP addresses. For more information, see Full Kubernetes deployments in AKS Edge Essentials:
```
New-AksEdgeConfig -NodeType Linux -DeploymentType ScalableCluster
```
Deploy the AKS Edge Essentials full-scale deployment using the configuration JSON file created in previous step:
```
New-AksEdgeDeployment -JsonConfigFilePath .\AksEdgeDeployConfigTemplate.json
```

Enable pinging ICMP traffic on the AKS-Edge-VM-1 virtual machine:

Invoke-AksEdgeNodeCommand -NodeType "Linux" -command "sudo iptables -A INPUT -p ICMP -j ACCEPT"

Ping AKS-Edge-VM-1 from the Windows-VM-1 virtual machine:
```
ping 172.20.1.4
```
Ping AKS-Edge-VM-1 from the Windows host OS:
```
ping 172.20.1.4
```
Warning

If the ping requests to the AKS-Edge-VM-1 VM fail, review the IP configuration and troubleshoot the network connections.

Inside Windows-VM-1, get the scaleConfig JSON file to be used in Windows-VM-2:

 New-AksEdgeScaleConfig -NodeType Linux -ScaleType AddMachine -LinuxNodeIp <VM2-Linux-Node-IP>

Create the Windows-VM-2 virtual machine. Repeat steps 4 to 14 with the new Windows-VM-2 parameters:
- Virtual machine name: Windows-VM-2
- IP Address: 172.20.1.3
- Processors: Number of virtual processors: 4
- Memory: 8192 MB
- Network adapter (Virtual switch): AKS-Int
- IP Gateway address: 172.20.1.1
- DNS Server address: 172.20.1.1
Test the ping from Windows-VM-2 to AKS-Edge-VM-1:
```
ping 172.20.1.4
```
Warning

If the ping requests to the AKS-Edge-VM-1 fail, review the IP configuration and troubleshoot the network connections.
Set up the Windows-VM-2 virtual machine, following the Prepare your machines for AKS Edge Essentials procedure.
Install all AKS Edge Essentials prerequisites for the Windows-VM-2 virtual machine. For more information about prerequisites, see AKS Edge Essentials requirements and support matrix:
```
Install-AksEdgeHostFeatures
```
Connect to the Windows-VM-1 virtual machine, and using an elevated PowerShell session, get the AKS Edge cluster join configuration:
```
New-AksEdgeScaleConfig -NodeType Linux -ScaleType AddMachine -LinuxNodeIp <AKS-Edge-Linux-VM-2>
```
Copy the AKS Edge Essentials scale configuration file from Windows-VM-1 and modify it with the appropriate parameters. For more information, see Scaling out on multiple machines.
Deploy AKS-Edge-VM-2 on Window-VM-2 using the scale configuration JSON file from the previous step:
```
New-AksEdgeDeployment -JsonConfigFilePath <scale-config-json>
```

Validate that the node was added to the cluster. On any node in the cluster, run the following cmdlet:

kubectl get nodes

If everything was correctly set up, you should see both Linux nodes running in your cluster

PS C:\> kubectl get nodes
NAME                     STATUS   ROLES                       AGE     VERSION
windows-vm1-ledge        Ready    control-plane,etcd,master   3m45s   v1.24.3+k3s-
windows-vm2-ledge        Ready    control-plane,etcd,master  10m25s   v1.24.3+k3s-

Next steps

Add workloads by deploying applications
Troubleshoot your deployment