Bring your own Container Network Interface (CNI) plugin with Azure Kubernetes Service (AKS)

Article
02/27/2023
3 minutes to read

Kubernetes does not provide a network interface system by default; this functionality is provided by network plugins. Azure Kubernetes Service provides several supported CNI plugins. Documentation for supported plugins can be found from the networking concepts page.

While the supported plugins meet most networking needs in Kubernetes, advanced users of AKS may desire to utilize the same CNI plugin used in on-premises Kubernetes environments or to make use of specific advanced functionality available in other CNI plugins.

This article shows how to deploy an AKS cluster with no CNI plugin pre-installed, which allows for installation of any third-party CNI plugin that works in Azure.

Support

BYOCNI has support implications - Microsoft support will not be able to assist with CNI-related issues in clusters deployed with BYOCNI. For example, CNI-related issues would cover most east/west (pod to pod) traffic, along with kubectl proxy and similar commands. If CNI-related support is desired, a supported AKS network plugin can be used or support could be procured for the BYOCNI plugin from a third-party vendor.

Support will still be provided for non-CNI-related issues.

Prerequisites

For ARM/Bicep, use at least template version 2022-01-02-preview or 2022-06-01
For Azure CLI, use at least version 2.39.0
The virtual network for the AKS cluster must allow outbound internet connectivity.
AKS clusters may not use 169.254.0.0/16, 172.30.0.0/16, 172.31.0.0/16, or 192.0.2.0/24 for the Kubernetes service address range, pod address range, or cluster virtual network address range.
The cluster identity used by the AKS cluster must have at least Network Contributor permissions on the subnet within your virtual network. If you wish to define a custom role instead of using the built-in Network Contributor role, the following permissions are required:
- Microsoft.Network/virtualNetworks/subnets/join/action
- Microsoft.Network/virtualNetworks/subnets/read
The subnet assigned to the AKS node pool cannot be a delegated subnet.
AKS doesn't apply Network Security Groups (NSGs) to its subnet and will not modify any of the NSGs associated with that subnet. If you provide your own subnet and add NSGs associated with that subnet, you must ensure the security rules in the NSGs allow traffic within the node CIDR range. For more details, see Network security groups.

Cluster creation steps

Deploy a cluster

Deploying a BYOCNI cluster requires passing the --network-plugin parameter with the parameter value of none.

First, create a resource group to create the cluster in:
```
az group create -l <Region> -n <ResourceGroupName>
```

Then create the cluster itself:

az aks create -l <Region> -g <ResourceGroupName> -n <ClusterName> --network-plugin none

When using an Azure Resource Manager template to deploy, pass none to the networkPlugin parameter to the networkProfile object. See the [Azure Resource Manager template documentation][deploy-arm-template] for help with deploying this template, if needed.

{
  "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
    "clusterName": {
      "type": "string",
      "defaultValue": "aksbyocni"
    },
    "location": {
      "type": "string",
      "defaultValue": "[resourceGroup().location]"
    },
    "kubernetesVersion": {
      "type": "string",
      "defaultValue": "1.22"
    },
    "nodeCount": {
      "type": "int",
      "defaultValue": 3
    },
    "nodeSize": {
      "type": "string",
      "defaultValue": "Standard_B2ms"
    }
  },
  "resources": [
    {
      "type": "Microsoft.ContainerService/managedClusters",
      "apiVersion": "2022-06-01",
      "name": "[parameters('clusterName')]",
      "location": "[parameters('location')]",
      "identity": {
        "type": "SystemAssigned"
      },
      "properties": {
        "agentPoolProfiles": [
          {
            "name": "nodepool1",
            "count": "[parameters('nodeCount')]",
            "mode": "System",
            "vmSize": "[parameters('nodeSize')]"
          }
        ],
        "dnsPrefix": "[parameters('clusterName')]",
        "kubernetesVersion": "[parameters('kubernetesVersion')]",
        "networkProfile": {
          "networkPlugin": "none"
        }
      }
    }
  ]
}

When using a Bicep template to deploy, pass none to the networkPlugin parameter to the networkProfile object. See the Bicep template documentation for help with deploying this template, if needed.

param clusterName string = 'aksbyocni'
param location string = resourceGroup().location
param kubernetesVersion string = '1.22'
param nodeCount int = 3
param nodeSize string = 'Standard_B2ms'

resource aksCluster 'Microsoft.ContainerService/managedClusters@2022-06-01' = {
  name: clusterName
  location: location
  identity: {
    type: 'SystemAssigned'
  }
  properties: {
    agentPoolProfiles: [
      {
        name: 'nodepool1'
        count: nodeCount
        mode: 'System'
        vmSize: nodeSize
      }
    ]
    dnsPrefix: clusterName
    kubernetesVersion: kubernetesVersion
    networkProfile: {
      networkPlugin: 'none'
    }
  }
}

Deploy a CNI plugin

When AKS provisioning completes, the cluster will be online, but all of the nodes will be in a NotReady state:

$ kubectl get nodes
NAME                                STATUS     ROLES   AGE    VERSION
aks-nodepool1-23902496-vmss000000   NotReady   agent   6m9s   v1.21.9

$ kubectl get node -o custom-columns='NAME:.metadata.name,STATUS:.status.conditions[?(@.type=="Ready")].message'
NAME                                STATUS
aks-nodepool1-23902496-vmss000000   container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized

At this point, the cluster is ready for installation of a CNI plugin.

Next steps

Learn more about networking in AKS in the following articles: