The CrashLoopBackOff of the pods in the AKS cluster cannot be resolved.

Question

I deployed a Kubernetes module following the installation guidance on the AKS cluster that was built referring to the following URL.

https://learn.microsoft.com/ja-jp/azure/firewall/protect-azure-kubernetes-service#clean-up-resources
But one of the pods in the AKS cluster is displaying the following error and the CrashLoopBackOff cannot be resolved.

I0406 06:36:45.627781 1 server.go:99] "Used resources" resources="certificatesigningrequests,configmaps,cronjobs,daemonsets,deployments,endpoints,horizontalpodautoscalers,ingresses,jobs,leases,limitranges,mutatingwebhookconfigurations,namespaces,networkpolicies,nodes,persistentvolumeclaims,persistentvolumes,poddisruptionbudgets,pods,replicasets,replicationcontrollers,resourcequotas,secrets,services,statefulsets,storageclasses,validatingwebhookconfigurations,volumeattachments" I0406 06:36:45.627845 1 types.go:136] "Using all namespaces" I0406 06:36:45.627856 1 server.go:121] "Metric allow-denylisting" allowDenyStatus="Excluding the following lists that were on denylist: " W0406 06:36:45.627874 1 client_config.go:617] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work. I0406 06:36:45.628266 1 server.go:249] "Tested communication with server" E0406 06:36:55.669217 1 main.go:67] "Failed to run kube-state-metrics" err="failed to create client: error while trying to communicate with apiserver: Get "https://10.2.0.1:443/version": EOF"

Here is the network configuration for the cluster:

Address Space: 10.20.0.0/16
Network type (plugin): Azure CNI
Pod CIDR: -
Service CIDR: 10.2.0.0/16
DNS service IP: 10.2.0.10
Docker bridge CIDR: 172.18.0.1/16
Network policy: none
Load balancer: Standard
HTTP application routing: disabled
Private cluster: disabled
Authorized IP ranges: disabled
Application Gateway Ingress Controller: disabled

Do we need additional configurations for Azure Firewall's classic rules?

Answer 1

Hi @Anonymous Thanks for reaching Microsoft Q&A. Please add DNAT rule to Azure firewall. https://learn.microsoft.com/en-us/azure/firewall/protect-azure-kubernetes-service#add-a-dnat-rule-to-azure-firewall Hope this helps. Please "Accept as Answer" if it helped, so that it can help others in the community looking for help on similar topics.

Answer 2

Thank you for your advice.
However, the method you taught me did not solve the problem.
Also, my explanation was insufficient.

I would like to solve the problem of not being able to connect to the container with the IP address of 10.2.0.1 (cluster IP) in the same cluster.

I had assumed that the pods within the cluster could communicate with each other because the following settings had already been executed, but in reality, it is not possible.
Of course, the problem can be solved by setting wildcards for the source IP, target IP, and destination port in the network rules of the firewall. However, instead of that, I would like to restrict it by setting appropriate IP addresses for the source IP and target IP. Could you please provide me with instructions on how to do so?

Answer 3

We have resolved this issue.

The cause was that we were unable to communicate with the IP address of the API server that was assigned when we created the AKS cluster (Figure.1,2).
Therefore, we added the IP address of the API server to the network rules of the firewall to allow communication (Figure.3). Figure.1）

Figure.2)

Figure.3)

I overlooked the fact that the aforementioned configuration was properly mentioned in the following URL (Figure.4). https://learn.microsoft.com/ja-jp/azure/aks/limit-egress-traffic#required-outbound-network-rules-and-fqdns-for-aks-clusters
Figure.4)

Answer 4

@Anonymous I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this!

Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others ", I'll repost your solution in case you'd like to "Accept " the answer. Accepted answers show up at the top, resulting in improved discoverability for others.

Issue: The CrashLoopBackOff of the pods in the AKS cluster cannot be resolved. Getting the error : "failed to create client while trying to communicate with apiserver".

Cause: Customer was unable to communicate with the IP address of the API server that was assigned when they created the AKS cluster.

Solution: Adding the IP address of the API server to the network rules of the firewall to allow communication, resolved the issue.

The API server public IP configuration is mentioned in the following documentation: Required outbound network rules and FQDNs for AKS clusters

If your issue remains unresolved or have further questions, please let us know in the comments how we can assist. We are here to help you and strive to make your experience better and greatly value your feedback.