Team, We need to create a new cluster regarding which I have few questions: How many node pools are considered as part of better management of production cluster If multiple node pools are a good approach, then how to organize the user and system nodes across the multiple node pools and how many nodes should we keep only for system node pools We are going to use internal load balancer and delete the default load balancer which AKS creates. Does using this leads to issues in the connectivity across node pools. Can each node pool has nodes of different size. For eg. node pool 1 has 2 nodes of sku B2s and Node pool 2 has all nodes of size D2S Should we use Azure CNI or Kubenet for networking in such case. Once our architecture diagram is ready, in Microsoft, is it possible to connect with any architect who can review our architecture and give us the sign off on the design. Should we reach out to our CSAM for this. Would be grateful if someone could suggest us on these queries. Thank you very much. Kind Regards, Tanul

Hello Tanul Let me try to answer your questions below: 1 - As per https://learn.microsoft.com/en-us/azure/aks/use-system-pools?tabs=azure-cli , it is recommended to have at least system node pool dedicated for system components and a user node pool for application workload. So the answer is at least two node pools. 2 - Labels and/or taints/tolerations can be used to separate system and application workload. The same link from the above should be read for more details. The number of nodes depends on your workload, and you should assess this based on how intensive your application(s) are. As per that link, "If you run a single system node pool for your AKS cluster in a production environment, we recommend you use at least three nodes for the node pool." 3 - You should not delete the default load balancer (called "kubernetes") that the AKS cluster creates. That is not supported and will bring your AKS cluster into an unsupported scenario. If you don't want to have the Load Balancer, you should use UDR outbound type. More details: https://learn.microsoft.com/en-us/azure/aks/egress-udr 4 - Yes, each node pool can have different SKU sizes or the same size. There is no restriction here. 5 - The details provided are not enough to provide an opinion of Azure CNI or Kubenet should be used. I would recommend you reading the below to understand what is best for you: https://learn.microsoft.com/en-us/azure/aks/concepts-network#kubenet-basic-networking https://learn.microsoft.com/en-us/azure/aks/configure-kubenet https://learn.microsoft.com/en-us/azure/aks/configure-azure-cni 6 - Yes, you should contact the CSAM to assign a Cloud Solution Architect, if available as per your contract. Please "Accept the answer" and "Upvote" it if it was helpful. Thank you!

Hello @Tanul let me answer your questions as below: 1 - How many node pools are considered as part of better management of the production cluster of at least two node-pools: a - system-node pool with at least one node https://learn.microsoft.com/en-us/azure/aks/use-system-pools?tabs=azure-cli and to make sure the system pods are scheduled on it you should an annotations https://learn.microsoft.com/en-us/azure/aks/use-system-pools?tabs=azure-cli#add-a-dedicated-system-node-pool-to-an-existing-aks-cluster b - Depending on your workload you can add user node-pools as much as you want to depend on your workload. 2 - If multiple node pools are a good approach, then how to organize the user and system nodes across the multiple node pools and how many nodes should we keep only for system node pools? here as mentioned in the first question you need at least one system node-pool regarding the user node-pool it depends on your application's workload resource such as CPU, Memory you need to calculate it, and depending on the workload you can add more user node-pools. 3 - We are going to use an internal load balancer and delete the default load balancer which AKS creates. Does using this leads to issues in the connectivity across node pools? it's not supported to make a manual deletion for aks resources once it has been created, you can create an AKS cluster using UDR here, it will not create a public load balancer, and you can restrict the egress traffic. https://learn.microsoft.com/en-us/azure/aks/limit-egress-traffic#deploy-aks-with-outbound-type-of-udr-to-the-existing-network or you can go with private AKS: https://learn.microsoft.com/en-us/azure/aks/private-clusters 4 - Can each node pool has nodes of different size. For eg. node pool 1 has 2 nodes of sku B2s and Node pool 2 has all nodes of size D2S yes you can add a flag once you create AKS cluster or added a new node-pool to select the VM SKU using this flag: --node-vm-size Standard_D2pds_v5 https://learn.microsoft.com/en-us/azure/aks/use-multiple-node-pools#add-a-node-pool 5 - Should we use Azure CNI or Kubenet for networking in such case. you should be familiar with Kubenet and CNI to select what is better for your environment: 1 - Kubenet is uses once you have many pods, and nodes and you want to reduce the number of the IPs on the subnet. here you will have more latency as the number of hops will be increase. https://learn.microsoft.com/en-us/azure/aks/configure-kubenet 2 - using azure it will use higher number of IPs on the subnet, as the pods will take the same range of IPS on the subnet level, much better performance, as here the latency will be reduced. https://learn.microsoft.com/en-us/azure/aks/configure-azure-cni 6 - Once our architecture diagram is ready, in Microsoft, is it possible to connect with any architect who can review our architecture and give us the sign off on the design. Should we reach out to our CSAM for this? it depends on your support plan or higher CSAM will help to find CSA to help you in your architecture. Please "Accept the answer" and "Upvote" it if it was helpful. Thank you!

Strategies regarding number of node pools and their management?

Tanul 1,251

Team, We need to create a new cluster regarding which I have few questions:

How many node pools are considered as part of better management of production cluster
If multiple node pools are a good approach, then how to organize the user and system nodes across the multiple node pools and how many nodes should we keep only for system node pools
We are going to use internal load balancer and delete the default load balancer which AKS creates. Does using this leads to issues in the connectivity across node pools.
Can each node pool has nodes of different size. For eg. node pool 1 has 2 nodes of sku B2s and Node pool 2 has all nodes of size D2S
Should we use Azure CNI or Kubenet for networking in such case.
Once our architecture diagram is ready, in Microsoft, is it possible to connect with any architect who can review our architecture and give us the sign off on the design. Should we reach out to our CSAM for this.

Would be grateful if someone could suggest us on these queries. Thank you very much. Kind Regards, Tanul

Accepted answer

Andrei Barbu 2,576 Reputation points Microsoft Employee

2023-04-13T08:19:03.42+00:00

Hello Tanul Let me try to answer your questions below:
1 - As per https://learn.microsoft.com/en-us/azure/aks/use-system-pools?tabs=azure-cli, it is recommended to have at least system node pool dedicated for system components and a user node pool for application workload. So the answer is at least two node pools.

2 - Labels and/or taints/tolerations can be used to separate system and application workload. The same link from the above should be read for more details. The number of nodes depends on your workload, and you should assess this based on how intensive your application(s) are. As per that link, "If you run a single system node pool for your AKS cluster in a production environment, we recommend you use at least three nodes for the node pool."

3 - You should not delete the default load balancer (called "kubernetes") that the AKS cluster creates. That is not supported and will bring your AKS cluster into an unsupported scenario. If you don't want to have the Load Balancer, you should use UDR outbound type. More details: https://learn.microsoft.com/en-us/azure/aks/egress-udr

4 - Yes, each node pool can have different SKU sizes or the same size. There is no restriction here.

5 - The details provided are not enough to provide an opinion of Azure CNI or Kubenet should be used. I would recommend you reading the below to understand what is best for you:

https://learn.microsoft.com/en-us/azure/aks/concepts-network#kubenet-basic-networking

https://learn.microsoft.com/en-us/azure/aks/configure-kubenet

https://learn.microsoft.com/en-us/azure/aks/configure-azure-cni

6 - Yes, you should contact the CSAM to assign a Cloud Solution Architect, if available as per your contract.

Please "Accept the answer" and "Upvote" it if it was helpful.

Thank you!
Please sign in to rate this answer.
Tanul 1,251 Reputation points

2023-04-14T09:09:35.3633333+00:00

@Andrei Barbu Thank you for sharing the details.

But currently, we already have a cluster which was created 4 years ago by some other engineer. It is very old with 8 availability sets type nodes in it due to which we are planning to create a new one. It does not have any public load balancer and I can't see any public IP as well.

Our AKS only has internal load balancer inside a subnet whose VNET is further connected to the organization network via another VNET having a common Azure public load balancer and express routes connected to company servers.

I am unable to figure out that what type of configuration does my existing AKS has as we don't have any terraform code for our existing cluster. Is there any way to confirm that whether we have UDR based or private AKS here? Would be grateful if you can share any commands etc.

I'm just guessing that I dont have private AKS because I can directly connect to the cluster via my company laptop and hit kubectl commands. Means I don't have to open any jump server for the connection.

Andrei Barbu 2,576 Reputation points Microsoft Employee

2023-04-14T09:23:57.99+00:00

Hello Tanul If you are using Basic Load Balancer on the existing AKS cluster, then public IP address doesn't get created by default. It gets created only if you create a Kubernetes Service type Public Load Balancer. Using Basic LB is a way to avoid having a public IP, but it is not recommended to use it with an AKS cluster due to multiple limitations like: https://learn.microsoft.com/en-us/azure/load-balancer/skus#skus

You can check the configuration of your AKS cluster with az aks show -g <rgname> -n <aksname> and look/grep for the needed information, or easier, check the Networking tab in Azure Portal. For example here, I use as outbound type Standard LB and my cluster is NOT private.

If you found this helpful, please accept it as an answer. Thank you!

Tanul 1,251 Reputation points

2023-04-14T09:41:39.4866667+00:00

@Andrei Barbu You are genius..

Correct this is exactly we are using. Basic LB and not private cluster.

But what should I do now.. In this kind of my organization networking structure. Could you please suggest the best possible solution. I am not allowed to make public ip's etc. Which is correct I think so from organization perspective.

How to have exactly same type of behavior, which I currently have, with standard type internal load balancers. Would be grateful if you can suggest one specific approach. Would be really thankful

Tanul 1,251 Reputation points

2023-04-14T09:44:02.9+00:00

I wanted to use standard because with basic multiple node pool is not possible

Andrei Barbu 2,576 Reputation points Microsoft Employee

2023-04-14T10:16:30.8066667+00:00

Hello Tanul! The AKS nodes need to have outbound connectivity (to the Internet) to be able to get provisioned and pull images for system components, like mcr.microsoft.com. Unfortunately there is no way to have Standard Load Balancer without any public IP. The public IP that gets created automatically created when using Standard LB is used to allow nodes to get to internet. The only solution I see is to use UDR type and have a firewall with a separate IP address (not as part of the AKS cluster). Maybe there are other options that I didn't explore yet and for that you may want to have a Cloud Solution Architect from Microsoft as you asked in your initial message, by contacting the CSAM.

If you found this helpful, please accept it as an answer. Thank you!

Tanul 1,251 Reputation points

2023-04-14T10:29:57.9133333+00:00

@Andrei Barbu, Thanks but is it mandatory to have a firewall. We have express routes. Isn't that enough to have connectivity. Let say if I have UDR with private standard load balancer. Is there any possibility to connect to AKS without having any separate jump server in Azure/on-prem. I was going through [this article] but, I 'm unable to fully understand how he achieved this and what is this privatelink DNS. Do we need to have this in Azure.

Tanul 1,251 Reputation points

2023-04-14T11:27:07.37+00:00

@Andrei Barbu, One last question if let say we create the standard load balancer and public ip.. Like in a normal way then how to disable its access with public internet. Does UDR help in this.. Why I am asking because with private AKS deployment using Azure devops or github action will be a tedious task.

Andrei Barbu 2,576 Reputation points Microsoft Employee

2023-04-14T11:58:28.6566667+00:00

Tanul, you mentioned about an article but it was not linked in the comment. However, I am not familiar with the scenario of ExpressRoute with AKS. All I can say is that as long as you have outbound type Standard LoadBalancer, is that the public IP will be created and it is not supported to delete it. You cannot choose to have public or private (internal) LB when using outbound type Standard LoadBalancer. You will have at least one public IP for the nodes to go to internet. Regarding how to disable the access of the IP to public Internet, one way I am thinking of would be using a NSG, but some endpoints from Internet would still need to be allowed, like https://learn.microsoft.com/en-us/azure/aks/limit-egress-traffic#required-outbound-network-rules-and-fqdns-for-aks-clusters

Tanul 1,251 Reputation points

2023-04-14T12:15:31.36+00:00

@Andrei Barbu, Referring to this article.

http://cloudhacks.blog/2020/11/creating-a-fully-internal-aks-cluster-no-public-ip/

The only reason behind deleting the public ip is that applications we will host as pods should not be accessible to internet. Otherwise, my org doesn't have any problem. My org don't allow to stop the incoming traffic at ingress controller level. They want it to be stopped before reaching to AKS. Is it possible to this at load balancer level or at any other level before traffic reaches to ingress controller? If it is then its fine.. This is the only reason behind deleting the public ip.

Tanul 1,251 Reputation points

2023-04-14T12:21:26.82+00:00

Please correct me if I'm wrong. As far as I know that public ip is also needed with standard lb to enable connectivity between Azure devops pipelines and AKS. I just want to keep it secure and disable the connectivity of application(and, if possible kubectl access as well) over internet. Can we do these things by any chance.

Andrei Barbu 2,576 Reputation points Microsoft Employee

2023-04-14T13:12:47.4366667+00:00

I am not familiar with the architecture from that link. Regarding kubectl, that can be kept private by using private AKS cluster, but you only the IP of the API server will be private, if you use Standard LB outbound type, you will still have a public IP associated with the kubernetes Load Balancer. Regarding Azure DevOps, you should be aware of this limitation for the private AKS cluster: There's no support for Azure DevOps Microsoft-hosted Agents with private clusters. Consider using Self-hosted Agents.

Tanul 1,251 Reputation points

2023-04-14T14:20:53.95+00:00

@Andrei Barbu, Hmm ok. So one idea. If I create a normal AKS with standard sku.. And, then create the internal load balancer as well behind which ingress controller etc. is running.. Does that work. Will it be the correct architecture in my case? Can we use this architecture in production. See technically even with basic load balancer also, one ip is attached to the api server url. The only difference is that it is not showing as a separate resource in my All resource list bceause it is a random public ip provided by microsoft. But, once I create a new cluster with standard LB sku a separate resource with public ip will be there which will add some cost. Am I correct?

Andrei Barbu 2,576 Reputation points Microsoft Employee

2023-04-14T14:28:17.9933333+00:00

If you create an AKS cluster with standard SKU and then an internal LB to expose the ingress controller then yes, that is correct and common use case. Regarding the basic LB scenario, to be honest, I am not sure, so don't take it as a correct response, it is just my assumption: The public IP doesn't exist as your resource and to go in the internet, it uses the default outbound. Regarding the cost, yes, it exists, I would suggest you simulate that with https://azure.microsoft.com/en-us/pricing/calculator/ by adding the resources (like Public IP - Standard and Load Balancer - Standard)

Tanul 1,251 Reputation points

2023-04-14T14:45:28.78+00:00

@Andrei Barbu, Thank you so much for sharing this document. I also think the same. Its highly likely that under the hood default public ip assign with basic one. Why I am sure because through my mobile and personal laptop I am able to ping the api server address. So, the only difference I can see now is cost which is 7.3$ per month for 2 clusters.

One more thing, if I ask my network team to add Site-2-Site VPN at Vnet level then I think my whole problem is solved. Then even after having public IP everything is only accessible via company laptop only if organization VPN is enabled. Am I correct?

If yes, then to access any application, hosted inside kubernetes, do we need to enable VPN in the laptop as well or webapps/microservice will work without VPN also?

Tanul 1,251 Reputation points

2023-04-14T17:01:48.0966667+00:00

@Andrei Barbu, Hey any suggestion on this Site-2-Site VPN query please. Thank you :)

Andrei Barbu 2,576 Reputation points Microsoft Employee

2023-04-14T19:56:36.5366667+00:00

Unfortunately I've never used that so I can't have an opinion. I saw you asked a new question for that and I am sure someone from the community will provide useful insights.

If for the initial question on this post you've got useful details from my side, I would appreciate if you can Accept it as an answer. Thank you!

Tanul 1,251 Reputation points

2023-04-15T03:44:21.35+00:00

Thanks
Sign in to comment

1 additional answer

Ammar-Abdelqader01 926 Reputation points Microsoft Employee

2023-04-13T08:38:04.4166667+00:00

Hello @Tanul let me answer your questions as below:

1 - How many node pools are considered as part of better management of the production cluster of at least two node-pools:

a - system-node pool with at least one node https://learn.microsoft.com/en-us/azure/aks/use-system-pools?tabs=azure-cli

and to make sure the system pods are scheduled on it you should an annotations https://learn.microsoft.com/en-us/azure/aks/use-system-pools?tabs=azure-cli#add-a-dedicated-system-node-pool-to-an-existing-aks-cluster

b - Depending on your workload you can add user node-pools as much as you want to depend on your workload. 2 - If multiple node pools are a good approach, then how to organize the user and system nodes across the multiple node pools and how many nodes should we keep only for system node pools?

here as mentioned in the first question you need at least one system node-pool regarding the user node-pool it depends on your application's workload resource such as CPU, Memory you need to calculate it, and depending on the workload you can add more user node-pools. 3 - We are going to use an internal load balancer and delete the default load balancer which AKS creates. Does using this leads to issues in the connectivity across node pools? it's not supported to make a manual deletion for aks resources once it has been created, you can create an AKS cluster using UDR here, it will not create a public load balancer, and you can restrict the egress traffic.
https://learn.microsoft.com/en-us/azure/aks/limit-egress-traffic#deploy-aks-with-outbound-type-of-udr-to-the-existing-network
or you can go with private AKS:
https://learn.microsoft.com/en-us/azure/aks/private-clusters
4 - Can each node pool has nodes of different size. For eg. node pool 1 has 2 nodes of sku B2s and Node pool 2 has all nodes of size D2S
yes you can add a flag once you create AKS cluster or added a new node-pool to select the VM SKU using this flag:
--node-vm-size Standard_D2pds_v5

https://learn.microsoft.com/en-us/azure/aks/use-multiple-node-pools#add-a-node-pool
5 - Should we use Azure CNI or Kubenet for networking in such case.
you should be familiar with Kubenet and CNI to select what is better for your environment:

1 - Kubenet is uses once you have many pods, and nodes and you want to reduce the number of the IPs on the subnet.
here you will have more latency as the number of hops will be increase. https://learn.microsoft.com/en-us/azure/aks/configure-kubenet
2 - using azure it will use higher number of IPs on the subnet, as the pods will take the same range of IPS on the subnet level, much better performance, as here the latency will be reduced. https://learn.microsoft.com/en-us/azure/aks/configure-azure-cni
6 - Once our architecture diagram is ready, in Microsoft, is it possible to connect with any architect who can review our architecture and give us the sign off on the design. Should we reach out to our CSAM for this?
it depends on your support plan or higher CSAM will help to find CSA to help you in your architecture.
Please "Accept the answer" and "Upvote" it if it was helpful. Thank you!
Please sign in to rate this answer.
Tanul 1,251 Reputation points

2023-04-14T09:02:36.2+00:00

@AmmarAbdlQader Thank you for sharing the details.

But currently, we already have a cluster which was created 4 years ago by some other engineer. It is very old with 8 availability sets type nodes in it due to which we are planning to create a new one. It does not have any public load balancer and I can't see any public IP as well.

Our AKS only has internal load balancer inside a subnet whose VNET is further connected to the organization network via another VNET having a common Azure public load balancer and express routes connected to company servers.

I am unable to figure out at all that what type of configuration does my existing AKS has as we don't have any terraform code for our existing cluster. Is there any way to confirm that whether we have UDR based or private AKS here? Would be grateful if you can share any commands etc.

I'm just guessing that I dont have private AKS because I can directly connect to the cluster via my company laptop and hit kubectl commands. Means I don't have to open any jump server for the connection.

Tanul 1,251 Reputation points

2023-04-14T10:08:12.2766667+00:00

We are not allowed to have any type of public ip..

Tanul 1,251 Reputation points

2023-04-14T11:44:43.5833333+00:00

Does creating udr type leads to issues in CI/CD deployments directly via Azure devops and Github actions.

Ammar-Abdelqader01 926 Reputation points Microsoft Employee

2023-04-16T06:09:07.0533333+00:00

Hello @Tanul

once the UDR is configured correctly should not have an issue with CI/CD.
please check this documentation using UDR also:https://learn.microsoft.com/en-us/azure/aks/egress-udr

I would appreciate it if you can accept it as an answer. Thank you!

Tanul 1,251 Reputation points

2023-04-17T08:23:28.0933333+00:00

@AmmarAbdlQader, Thank you for sharing. But this link has no steps written or something like that.. Do we have any script or any documents explaining the steps need to create the cluster with this UDR thing..
Sign in to comment