Quickstart: Using Azure CLI to deploy an AKS hybrid cluster on a Windows Server node in an Azure VM
In this quickstart, you'll learn how to use Azure CLI to test creating AKS hybrid clusters on-premises. You can use this quickstart to run a quick proof of concept on an Azure VM, or if you do not have a Windows Server or Azure Stack HCI cluster.
If you do have a Windows Server or Azure Stack HCI cluster, follow this detailed guide to get started with deploying AKS hybrid clusters from Azure.
Before you begin
Before you begin, make sure you meet the following requirements:
- Have access to an Azure subscription.
- Make sure you're an owner of the Azure subscription.
Step 1: Register your Azure subscription for features and providers
Register the following Azure providers on your Azure subscription. Make sure you login to Azure first. You only need to do this operation once per Azure subscription.
az provider register --namespace Microsoft.Kubernetes --wait
az provider register --namespace Microsoft.ExtendedLocation --wait
az provider register --namespace Microsoft.ResourceConnector --wait
az provider register --namespace Microsoft.HybridContainerService --wait
az provider register --namespace Microsoft.HybridConnectivity --wait
Step 2: Create an Azure VM and deploy Windows Server on the Azure VM
To keep things simple, we'll show you how to deploy your VM via an Azure Resource Manager template. The Deploy to Azure button, when clicked, takes you directly to the Azure portal, and upon sign-in, provides you with a form to complete. If you want to open this in a new tab, hold CTRL when you click the button.
After clicking the Deploy to Azure button, enter the details, which should look something similar to those shown below, and click Review and Create.
Note
For customers with Software Assurance, Azure Hybrid Benefit for Windows Server allows you to use your on-premises Windows Server licenses and run Windows virtual machines on Azure at a reduced cost. By selecting Yes for the "Already have a Windows Server License", you confirm you have an eligible Windows Server license with Software Assurance or Windows Server subscription to apply this Azure Hybrid Benefit and have reviewed the Azure hybrid benefit compliance.
The custom template will be validated, and if all of your entries are correct, you can select Create. In 30 minutes, your VM will be created.
Access your Azure VM
With your Azure Virtual Machine (AKSHCIHost001) successfully deployed and configured, you're ready to connect to the VM using Remote Desktop.
If you're not already signed into the Azure portal, sign in with the same credentials you previously used. Once signed in, enter "azshci" in the search box on the dashboard, and in the search results select your AKSHCIHost001 virtual machine.
In the Overview blade for your VM, at the top of the blade, select Connect, and from the drop-down options select RDP. On the newly opened Connect blade, ensure that Public IP is selected. Also ensure that the RDP port matches what you provided at deployment time. By default, this should be 3389. Then select Download RDP File and choose a suitable folder to store the .rdp file.
Once downloaded, locate the .rdp file on your local machine, and double-click to open it. Click Connect and when prompted, enter the credentials you supplied when creating the VM earlier. Accept any certificate prompts, and within a few minutes you should be successfully logged into the Azure VM.
Step 3: Install Azure CLI on the Azure VM
Once you've logged into the Azure VM, run the following command to install Az CLI.
$ProgressPreference = 'SilentlyContinue'; Invoke-WebRequest -Uri https://aka.ms/installazurecliwindows -OutFile .\AzureCLI.msi; Start-Process msiexec.exe -Wait -ArgumentList '/I AzureCLI.msi /quiet'; rm .\AzureCLI.msi
Exit
Step 4: Install Az CLI extensions on the Azure VM
$env:PATH += ";C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\wbin;"
az extension add -n k8s-extension --upgrade
az extension add -n customlocation --upgrade
az extension add -n arcappliance --upgrade --version 0.2.27
az extension add -n hybridaks --upgrade
Step 5: Install pre-requisite PowerShell repositories
Close all open PowerShell windows on your Azure VM and open a fresh new PowerShell admin window and run the following command:
Set-PSRepository -Name "PSGallery" -InstallationPolicy Trusted
Install-PackageProvider -Name NuGet -Force
Install-Module -Name PowershellGet -Force
Exit
Open a new PowerShell admin window and run the following command:
Install-Module -Name AksHci -Repository PSGallery -AcceptLicense -Force
Exit
Open a new PowerShell admin window and run the following command:
Install-Module -Name ArcHci -Repository PSGallery -AcceptLicense -Force
Exit
Open a new PowerShell admin window and run the following command:
Initialize-AksHciNode
Exit
Open a new PowerShell admin window and run the following command:
New-Item -Path "V:\" -Name "AKS-HCI" -ItemType "directory" -Force
New-Item -Path "V:\AKS-HCI\" -Name "Images" -ItemType "directory" -Force
New-Item -Path "V:\AKS-HCI\" -Name "WorkingDir" -ItemType "directory" -Force
New-Item -Path "V:\AKS-HCI\" -Name "Config" -ItemType "directory" -Force
Exit
Step 6: Install the AKS on Windows Server management cluster
Open a new PowerShell admin window and run the following command:
$vnet=New-AksHciNetworkSetting -Name "mgmt-vnet" -vSwitchName "InternalNAT" -gateway "192.168.0.1" -dnsservers "192.168.0.1" -ipaddressprefix "192.168.0.0/16" -k8snodeippoolstart "192.168.0.4" -k8snodeippoolend "192.168.0.10" -vipPoolStart "192.168.0.150" -vipPoolEnd "192.168.0.160"
Set-AksHciConfig -vnet $vnet -imageDir "V:\AKS-HCI\Images" -workingDir "V:\AKS-HCI\WorkingDir" -cloudConfigLocation "V:\AKS-HCI\Config" -version "1.0.13.10907" -cloudServiceIP "192.168.0.4"
Next, set the Azure subscription and resource group variables and then run Set-AksHciRegistration
:
$sub = <Azure subscription>
$rgName = <Azure resource group>
#Use device authentication to login to Azure. Follow the steps you see on the screen
Set-AksHciRegistration -SubscriptionId $sub -ResourceGroupName $rgName -UseDeviceAuthentication
Run the following command to install AKS on Windows Server host:
Install-AksHci
Validate AKS on Windows Server version
Make sure your AKS on Windows Server version is the following. Expected Output:
1.0.13.10907
Do not proceed if you have any errors! If you face an issue installing AKS on Windows Server, review the troubleshooting section. If the troubleshooting section does not help you, file a GitHub issue. Attach logs using Get-AksHciLogs
so that we can help you faster.
Step 7: Generate pre-requisite YAML files needed to deploy Azure Arc Resource Bridge
$subscriptionId = <Azure subscription ID>
$resourceGroup = <Azure resource group>
$location=<Azure location. Can be "eastus" or "westeurope">
$workingDir = "V:\AKS-HCI\WorkDir"
$arcAppName="arc-resource-bridge"
$configFilePath= $workingDir + "\hci-appliance.yaml"
$arcExtnName = "aks-hybrid-ext"
$customLocationName="azurevm-customlocation"
Generate the Azure Arc Resource Bridge YAML files:
New-ArcHciAksConfigFiles -subscriptionID $subscriptionId -location $location -resourceGroup $resourceGroup -resourceName $arcAppName -workDirectory $workingDir -vnetName "appliance-vnet" -vSwitchName "InternalNAT" -gateway "192.168.0.1" -dnsservers "192.168.0.1" -ipaddressprefix "192.168.0.0/16" -k8snodeippoolstart "192.168.0.11" -k8snodeippoolend "192.168.0.11" -controlPlaneIP "192.168.0.161"
Sample output:
HCI login file successfully generated in 'V:\AKS-HCI\WorkingDir\kvatoken.tok'
Generating ARC HCI configuration files...
Config file successfully generated in 'V:\AKS-HCI\WorkingDir'
Step 8: Deploy Azure Arc Resource Bridge
Once you've generated the YAML files, run the following command to validate the generated YAML files. Remember to log in to Azure before running these commands.
az account set -s $subscriptionid
az arcappliance validate hci --config-file $configFilePath
After you've validated the YAML files, run the following command to download the VHD image for deploying Arc Resource Bridge.
az arcappliance prepare hci --config-file $configFilePath
After you've downloaded the VHD image for deploying Arc Resource Bridge, you can now run the following command to deploy Arc Resource Bridge.
az arcappliance deploy hci --config-file $configFilePath --outfile $workingDir\config
Once you've deployed Arc Resource Bridge, run the following command to connect it to Azure.
az arcappliance create hci --config-file $configFilePath --kubeconfig $workingDir\config
Connecting Arc Resource Bridge to Azure may take up to 10 minutes to finish. To check the status of your deployment, run the following command. The Arc Resource Bridge must be in Running
state.
az arcappliance show --resource-group $resourceGroup --name $arcAppName --query "status" -o tsv
Expected output:
Running
Step 9: Install the AKS hybrid extension on the Arc Resource Bridge
To install the AKS hybrid extension on the Arc Resource Bridge, run the following command:
az k8s-extension create -g $resourceGroup -c $arcAppName --cluster-type appliances --name $arcExtnName --extension-type Microsoft.HybridAKSOperator --config Microsoft.CustomLocation.ServiceAccount="default"
Once you've created the AKS hybrid extension on top of the Arc Resource Bridge, run the following command to check if the cluster extension provisioning state says Succeeded. It might say something else at first, but you can try again after 10 minutes.
az k8s-extension show --resource-group $resourceGroup --cluster-name $arcAppName --cluster-type appliances --name $arcExtnName --query "provisioningState" -o tsv
Expected output:
Succeeded
Step 10: Create a Custom Location on top of the Azure Arc Resource Bridge
Run the following commands to create a Custom Location on top of the Arc Resource Bridge. You will choose this Custom Location when creating the AKS hybrid cluster through Az CLI or through the Azure portal.
$ArcApplianceResourceId=az arcappliance show --resource-group $resourceGroup --name $arcAppName --query id -o tsv
$ClusterExtensionResourceId=az k8s-extension show --resource-group $resourceGroup --cluster-name $arcAppName --cluster-type appliances --name $arcExtnName --query id -o tsv
To create a Custom Location, run the following command:
az customlocation create --name $customLocationName --namespace "default" --host-resource-id $ArcApplianceResourceId --cluster-extension-ids $ClusterExtensionResourceId --resource-group $resourceGroup
Once you create the Custom Location on top of the Arc Resource Bridge, run the following command to check if the Custom Location provisioning state says Succeeded. It might say something else at first, but you can try again after 10 minutes.
az customlocation show --name $customLocationName --resource-group $resourceGroup --query "provisioningState" -o tsv
Expected output:
Succeeded
Step 11: Create a local network for AKS hybrid clusters and connect it to Azure
Run the following command to create the local network for AKS hybrid clusters:
New-KvaVirtualNetwork -name hybridaks-vnet -vSwitchName "InternalNAT" -gateway "192.168.0.1" -dnsservers "192.168.0.1" -ipaddressprefix "192.168.0.0/16" -k8snodeippoolstart "192.168.0.15" -k8snodeippoolend "192.168.0.25" -vipPoolStart "192.168.0.162" -vipPoolEnd "192.168.0.170" -kubeconfig $workingDir\config
Once you've created the local network, connect it to Azure by running the following command:
$clid = az customlocation show --name $customLocationName --resource-group $resourceGroup --query "id" -o tsv
az hybridaks vnet create -n "azvnet" -g $resourceGroup --custom-location $clId --moc-vnet-name "hybridaks-vnet"
$vnetId = az hybridaks vnet show -n "azvnet" -g $resourceGroup --query id -o tsv
Step 12: Download the Kubernetes VHD image to your Azure VM
Run the following command to download the Kubernetes VHD image for your AKS hybrid clusters to your Azure VM:
Add-KvaGalleryImage -kubernetesVersion 1.21.9
Step 13: Create an Azure AD group and add Azure AD members to it
In order to connect to the AKS hybrid cluster from anywhere, you need to create an Azure AD group and add members to it. All the members in the Azure AD group will have cluster administrator access to the AKS hybrid cluster. Make sure to add yourself to the Azure AD group. If you do not add yourself, you will not be able to access the AKS hybrid cluster using kubectl
.
To learn more about how to create an Azure AD group, visit how to manage and create Azure AD groups. You will need the objectID
of the Azure AD group to create AKS hybrid clusters. You can skip this step if you already have an Azure AD group.
Step 14: Create an AKS hybrid cluster using Azure CLI
Run the following command to create an AKS hybrid cluster using Azure CLI.
az hybridaks create --name <Name of your AKS hybrid cluster> --resource-group $resourceGroup --custom-location $clid --vnet-ids $vnetId --kubernetes-version "v1.21.9" --aad-admin-group-object-ids <Azure AD group object ID> --generate-ssh-keys
Add a Linux nodepool to the AKS hybrid cluster
az hybridaks nodepool add -n <nodepool name> --resource-group $resourceGroup --cluster-name <aks hybrid cluster name>
Step 15: Connect to your AKS hybrid cluster using kubectl
and Azure AD
In order to connect to your AKS hybrid cluster using kubectl
, run the following command.
az hybridaks proxy --resource-group $resourceGroup --name <aks-hybrid cluster name> --file .\target-config
Let this command run for as long as you want to access your cluster. If this command times out, close the PowerShell window, open a fresh one and run the command again.
List the pods of the provisioned cluster using the kubeconfig
In a different command line window, run the following command pointing to the kubeconfig you downloaded in the previous step.
kubectl get pods -A --kubeconfig .\target-config
Expected Output:
NAME READY STATUS RESTARTS AGE
akshci-telemetry-initial-h6xr9 0/1 Completed 0 48s
calico-kube-controllers-5477b8df74-8sxbq 1/1 Running 0 9h
calico-node-6hhwb 1/1 Running 0 8h
calico-node-tmqmh 1/1 Running 0 9h
calico-patch-99pc8 1/1 Running 0 9h
calico-patch-z4pn2 1/1 Running 0 8h
calicoctl 1/1 Running 0 9h
certificate-controller-manager-7c7cb9746c-ppdt7 1/1 Running 0 9h
coredns-7489c675f5-84xcz 1/1 Running 0 9h
coredns-7489c675f5-qbfjn 1/1 Running 0 9h
csi-akshcicsi-controller-5f6bff6dc8-j4dzx 5/5 Running 0 9h
csi-akshcicsi-node-jjzxg 3/3 Running 0 8h
csi-akshcicsi-node-pj2sp 3/3 Running 0 9h
etcd-moc-l1hj0nllpvs 1/1 Running 0 9h
kube-apiserver-moc-l1hj0nllpvs 1/1 Running 0 9h
kube-controller-manager-moc-l1hj0nllpvs 1/1 Running 0 9h
kube-proxy-8jhdl 1/1 Running 0 9h
kube-proxy-lsk6f 1/1 Running 0 8h
kube-scheduler-moc-l1hj0nllpvs 1/1 Running 0 9h
moc-cloud-controller-manager-5fbb78df68-6d7b5 1/1 Running 0 9h
Next steps
Once you've finished quickly trying out this preview in an Azure VM, you can take a look at the following documents related to the preview: