AKS Cluster CNI and ClusterIP

Question

Dear AKS Users,

I have started to use Azure AKS and had a question. So, I have a created a private Azure AKS Cluster with Azure CNI which is connected to my VLAN 10.0.0.0/16 and Subnet 10.0.3.0/24. I understand that my pods will get an IP from the 10.0.3.0/24 range which is understood. I am not sure on the 'Kubernetes Service Address Range' and 'Docker Bridge Address'.

1. What should I specify there ?
2. Lets say if i specify 10.0.4.0/24 to the Kubernetes Service Address Range, will it mean that all the services that i create it will be from that range ?
3. If so, will I then be able to route/connect directly from any machine connected within the same VNET (example a VM that in same vnet but different subnet) to a service of type cluster IP on its cluster-IP within the k8s cluster ? I would not need an ingress for that or i will still need an ingress ?
4. If i have to use ingress for (3) above, what is the difference between using a CNI vs Kubenet. I understand Kubenet services also need Ingress as they can't be directly connected to. Our use-case is that we have a Site-To-Site VPN between Azure and On-Premise and we want kubernetes services to connect to on-site services both-ways. I understand I have to use the CNI for that. If i use kubenet, I can only have one-way (pod initiated) connection to the on-site services but not vice-versa. Thanks

Answer 1

@geek876 , Thank you for your question(s).

I am not sure on the 'Kubernetes Service Address Range' and 'Docker Bridge Address'.

1. What should I specify there ?

Kubernetes service address range: This parameter is the set of virtual IPs that Kubernetes assigns to internal services in your cluster. You can use any private address range that satisfies the following requirements:

• Must not be within the virtual network IP address range of your cluster
• Must not overlap with any other virtual networks with which the cluster virtual network peers
• Must not overlap with any on-premises IPs
• Must not be within the ranges 169.254.0.0/16, 172.30.0.0/16, 172.31.0.0/16, or 192.0.2.0/24

Although it's technically possible to specify a service address range within the same virtual network as your cluster, doing so is not recommended. Unpredictable behavior can result if overlapping IP ranges are used. For more information, see the FAQ section of this article. For more information on Kubernetes services, see Services in the Kubernetes documentation.

Service address CIDR must be smaller than /12. You can reuse this range across different AKS clusters.

For example:

az aks show -g $RG -n akstest --query networkProfile.serviceCidr -o tsv  
10.0.0.0/16  
az aks show -g $RG -n akstest --query networkProfile.podCidr -o tsv  
10.244.0.0/16  

The Kubernetes Service Address Range for this cluster is 10.0.0.0/16 and the Pod CIDR is 10.244.0.0/16

Thus, from

az aks get-credentials -g $RG -n akstest  
Merged "akstest" as current context in $HOME/.kube/config  
  
kubectl get all -o wide  
NAME        READY   STATUS    RESTARTS   AGE   IP            NODE                                NOMINATED NODE   READINESS GATES  
pod/nginx   1/1     Running   0          20s   10.244.2.20   aks-agentpool-xxxxxxxx-vmss000001   <none>           <none>  
NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE    SELECTOR  
service/kubernetes   ClusterIP   10.0.0.1     <none>        443/TCP   359d   <none>  

we can see that the Cluster IP of Service resources (here service/kubernetes, CLuster IP: 10.0.0.1) in Kubernetes are allocated from the Kubernetes Service Address Range (here 10.0.0.0/16)

Pods (here pod/nginx, IP: 10.244.2.20) get their IP addresses from the Pod CIDR (here 10.244.0.0/16). With Azure CNI network plugin, the Pod CIDR is the address range of the Azure Virtual Network subnet with which the AKS cluster is deployed.

Docker Bridge address: The Docker bridge network address represents the default docker0 bridge network address present in all Docker installations. While docker0 bridge is not used by AKS clusters or the pods themselves, you must set this address to continue to support scenarios such as docker build within the AKS cluster. It is required to select a CIDR for the Docker bridge network address because otherwise Docker will pick a subnet automatically which could conflict with other CIDRs. You must pick an address space that does not collide with the rest of the CIDRs on your networks, including the cluster's service CIDR and pod CIDR. Default is 172.17.0.1/16. You can reuse this range across different AKS clusters.

For example:

az aks show -g $RG -n akstest --query networkProfile.dockerBridgeCidr -o tsv  
172.17.0.1/16  

This cluster has a Docker Bridge Address Range of 172.17.0.1/16

On SSH into a node on the AKS Cluster

root@aks-agentpool-xxxxxxxx-vmss000000:/# docker network ls  
NETWORK ID          NAME                DRIVER              SCOPE  
b9d678e05482        bridge              bridge              local  
3a646177172f        host                host                local  
df1b33cceea6        none                null                local  
root@aks-agentpool-xxxxxxxx-vmss000000:/# docker inspect b9d678e05482  
[  
    {  
        "Name": "bridge",  
        "Id": "b9d678e05482xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",  
          ...  
        "Driver": "bridge",  
        "EnableIPv6": false,  
        "IPAM": {  
            "Driver": "default",  
            "Options": null,  
            "Config": [  
                {  
                    "Subnet": "172.17.0.0/16",  
                    "Gateway": "172.17.0.1"  
                }  
            ]  
        },  
        ...  
    }  
]  

For more information please check this document

Lets say if I specify 10.0.4.0/24 to the Kubernetes Service Address Range, will it mean that all the services that I create it will be from that range ?

The ClusterIP of the services in the cluster will be allocated from the Kubernetes Service Address Range.

If so, will I then be able to route/connect directly from any machine connected within the same VNET (example a VM that in same vnet but different subnet) to a service of type cluster IP on its cluster-IP within the k8s cluster ? I would not need an ingress for that or I will still need an ingress ?

With Azure Container Networking Interface (CNI), every pod gets an IP address from the subnet and can be accessed directly from any device within the network or a connected network. Reference

ClusterIP of Services, on the other will be allocated from the Kubernetes Service Address Range, discussed above. Although, it's technically possible to specify a service address range within the same virtual network as your cluster, doing so is not recommended. Unpredictable behavior can result if overlapping IP ranges are used. For more information, see the FAQ section of this article.

Instead, we recommend you to use an internal load balancer with Azure Kubernetes Service (AKS). An internal load balancer makes a Kubernetes service accessible only to applications running in the same virtual network (and connected networks) as the Kubernetes cluster.

Ingress exposes HTTP and HTTPS routes from outside the cluster to services within the cluster. Traffic routing is controlled by rules defined on the Ingress resource.
Here is a simple example where an Ingress sends all its traffic to one Service:

For more information please check this document.

To summarize, unless your use-case involves specific in-cluster routing rules for the Services on AKS, Ingress is not necessary. For Kubernetes Services to connect both ways with your on-premise network which is connected to the AKS cluster, services of type Azure internal load balancer should be sufficient.

If I have to use ingress for (3) above, what is the difference between using a CNI vs Kubenet. I understand Kubenet services also need Ingress as they can't be directly connected to. Our use-case is that we have a Site-To-Site VPN between Azure and On-Premise and we want kubernetes services to connect to on-site services both-ways. I understand I have to use the CNI for that. If i use kubenet, I can only have one-way (pod initiated) connection to the on-site services but not vice-versa.

Both kubenet and Azure CNI provide network connectivity for your AKS clusters. However, there are advantages and disadvantages to each. At a high level, the following considerations apply:

kubenet

• Conserves IP address space.
• Uses Kubernetes internal or external load balancer to reach pods from outside of the cluster.
• You manually manage and maintain user-defined routes (UDRs).
• Maximum of 400 nodes per cluster.

Azure CNI

• Pods get full virtual network connectivity and can be directly reached via their private IP address from connected networks.
• Requires more IP address space.

The following behavior differences exist between kubenet and Azure CNI:

Regarding DNS, with both kubenet and Azure CNI plugins DNS are offered by CoreDNS, a deployment running in AKS with its own autoscaler. For more information on CoreDNS on Kubernetes, see Customizing DNS Service. CoreDNS by default is configured to forward unknown domains to the DNS functionality of the Azure Virtual Network where the AKS cluster is deployed. Hence, Azure DNS and Private Zones will work for pods running in AKS.

Summing up, if you need AKS pod connectivity to and from your on-premise network then Azure CNI is the only option for network plugin. However, if you want outbound traffic from the AKS pod(s) to your on-premise network directly and inbound traffic to the AKS pod(s) via Kubernetes Service resources (internal load balancer) then you can use either kubenet or Azure CNI as network plugin.

----------

Hope this helps.

Please "Accept as Answer" if it helped, so that it can help others in the community looking for help on similar topics.

Answer 2

Thanks for the detailed answer @SRIJIT-BOSE-MSFT

So from my perspective, as we have a Site-To-Site VPN between Azure and On-Site, I would go with a CNI network. I have to use CNI as I want k8s to use the VNET that is connected to the On-Site via Site-To-Site VPN. Using kubenet won't give me that connectivity I suppose as it would provision a new VNET.

As far as connecting to PODs is concerned, i know with CNI, i can directly connect to PODs, but i don't think i need that. I will be connecting to and from the 'services' only (both ways between onsite <-> Azure AKS) but that can be provided by CNI and KubeNet both. My main reason for using CNI would be for Site-To-Site connectivity. Please confirm this.

Secondly, thanks for clarifying on the 'Kubernetes Service IP and Docker Bridge IP'. I will use a range (that I don't use elsewhere) for my service IPs and probably use default for Docker Bridge IP

Thanks.