Customize CoreDNS with Azure Kubernetes Service

Bài viết
10/30/2024

Azure Kubernetes Service (AKS) uses the CoreDNS project for cluster DNS management and resolution with all 1.12.x and higher clusters. For more information about CoreDNS customization and Kubernetes, see the official upstream documentation.

AKS is a managed service, so you can't modify the main configuration for CoreDNS (a CoreFile). Instead, you use a Kubernetes ConfigMap to override the default settings. To see the default AKS CoreDNS ConfigMaps, use the kubectl get configmaps --namespace=kube-system coredns -o yaml command.

This article shows you how to use ConfigMaps for basic CoreDNS customization options of in AKS. This approach differs from configuring CoreDNS in other contexts, such as CoreFile.

Note

Previously, kube-dns was used for cluster DNS management and resolution, but it's now deprecated. kube-dns offered different customization options via a Kubernetes config map. CoreDNS is not backwards compatible with kube-dns. Any customizations you previously used must be updated for CoreDNS.

Before you begin

This article assumes that you have an existing AKS cluster. If you need an AKS cluster, you can create one using Azure CLI, Azure PowerShell, or the Azure portal.
Verify the version of CoreDNS you're running. The configuration values may change between versions.
When you create configurations like the examples below, your names in the data section must end in .server or .override. This naming convention is defined in the default AKS CoreDNS ConfigMap, which you can view using the kubectl get configmaps --namespace=kube-system coredns -o yaml command.

Plugin support

All built-in CoreDNS plugins are supported. No add-on/third party plugins are supported.

Rewrite DNS

You can customize CoreDNS with AKS to perform on-the-fly DNS name rewrites.

Create a file named corednsms.yaml and paste the following example configuration. Make sure to replace <domain to be rewritten> with your own fully qualified domain name.

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns-custom
  namespace: kube-system
data:
  test.server: |
    <domain to be rewritten>.com:53 {
    log
    errors
    rewrite stop {
      name regex (.*)\.<domain to be rewritten>\.com {1}.default.svc.cluster.local
      answer name (.*)\.default\.svc\.cluster\.local {1}.<domain to be rewritten>.com
    }
    forward . /etc/resolv.conf # you can redirect this to a specific DNS server such as 10.0.0.10, but that server must be able to resolve the rewritten domain name
    }

Important

If you redirect to a DNS server, such as the CoreDNS service IP, that DNS server must be able to resolve the rewritten domain name.

Create the ConfigMap using the kubectl apply configmap command and specify the name of your YAML manifest.
```
kubectl apply -f corednsms.yaml
```
Verify the customizations have been applied using the kubectl get configmaps and specify your coredns-custom ConfigMap.
```
kubectl get configmaps --namespace=kube-system coredns-custom -o yaml
```
To reload the ConfigMap and enable Kubernetes Scheduler to restart CoreDNS without downtime, perform a rolling restart using kubectl rollout restart.
```
kubectl -n kube-system rollout restart deployment coredns
```

Custom forward server

If you need to specify a forward server for your network traffic, you can create a ConfigMap to customize DNS.

Create a file named corednsms.yaml and paste the following example configuration. Make sure to replace the forward name and the address with the values for your own environment.

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns-custom
  namespace: kube-system
data:
  test.server: | # you may select any name here, but it must end with the .server file extension
    <domain to be rewritten>.com:53 {
        forward foo.com 1.1.1.1
    }

Create the ConfigMap using the kubectl apply configmap command and specify the name of your YAML manifest.
```
kubectl apply -f corednsms.yaml
```
To reload the ConfigMap and enable Kubernetes Scheduler to restart CoreDNS without downtime, perform a rolling restart using kubectl rollout restart.
```
kubectl -n kube-system rollout restart deployment coredns
```

Use custom domains

You may want to configure custom domains that can only be resolved internally. For example, you may want to resolve the custom domain puglife.local, which isn't a valid top-level domain. Without a custom domain ConfigMap, the AKS cluster can't resolve the address.

Create a new file named corednsms.yaml and paste the following example configuration. Make sure to update the custom domain and IP address with the values for your own environment.

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns-custom
  namespace: kube-system
data:
  puglife.server: | # you may select any name here, but it must end with the .server file extension
    puglife.local:53 {
        errors
        cache 30
        forward . 192.11.0.1  # this is my test/dev DNS server
    }

Create the ConfigMap using the kubectl apply configmap command and specify the name of your YAML manifest.
```
kubectl apply -f corednsms.yaml
```
To reload the ConfigMap and enable Kubernetes Scheduler to restart CoreDNS without downtime, perform a rolling restart using kubectl rollout restart.
```
kubectl -n kube-system rollout restart deployment coredns 
```

Stub domains

CoreDNS can also be used to configure stub domains.

Create a file named corednsms.yaml and paste the following example configuration. Make sure to update the custom domains and IP addresses with the values for your own environment.

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns-custom
  namespace: kube-system
data:
  test.server: | # you may select any name here, but it must end with the .server file extension
    abc.com:53 {
     errors
     cache 30
     forward . 1.2.3.4
    }
    my.cluster.local:53 {
        errors
        cache 30
        forward . 2.3.4.5
    }

Create the ConfigMap using the kubectl apply configmap command and specify the name of your YAML manifest.
```
kubectl apply -f corednsms.yaml
```
To reload the ConfigMap and enable Kubernetes Scheduler to restart CoreDNS without downtime, perform a rolling restart using kubectl rollout restart.
```
kubectl -n kube-system rollout restart deployment coredns
```

Hosts plugin

All built-in plugins are supported, so the CoreDNS hosts plugin is available to customize /etc/hosts as well.

Create a file named corednsms.yaml and paste the following example configuration. Make sure to update the IP addresses and hostnames with the values for your own environment.

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns-custom # this is the name of the configmap you can overwrite with your changes
  namespace: kube-system
data:
    test.override: | # you may select any name here, but it must end with the .override file extension
          hosts { 
              10.0.0.1 example1.org
              10.0.0.2 example2.org
              10.0.0.3 example3.org
              fallthrough
          }

Create the ConfigMap using the kubectl apply configmap command and specify the name of your YAML manifest.
```
kubectl apply -f corednsms.yaml
```
To reload the ConfigMap and enable Kubernetes Scheduler to restart CoreDNS without downtime, perform a rolling restart using kubectl rollout restart.
```
kubectl -n kube-system rollout restart deployment coredns
```

Invalid search domain completions for internal.cloudapp.net and reddog.microsoft.com

Azure DNS configures a default search domain of <vnetId>.<region>.internal.cloudapp.net in virtual networks using Azure DNS and a non-functional stub reddog.microsoft.com in virtual networks using custom DNS servers (see the name resolution for resources documentation for more details). Kubernetes configures pod DNS settings with ndots: 5 to properly support cluster service hostname resolution. These two configurations combine to result in invalid search domain completion queries that never succeed being sent to upstream name servers while the system processes through the domain search list. These invalid queries cause name resolution delays and can place extra load on upstream DNS servers.

As of the v20241025 AKS release, AKS configures CoreDNS to respond with NXDOMAIN in the following two cases in order to prevent these invalid search domain completion queries from being forwarded to upstream DNS:

Any query for the root domain or a subdomain of reddog.microsoft.com.
Any query for a subdomain of internal.cloudapp.net that has seven or more labels in the domain name.
- This configuration allows virtual machine resolution by hostname to still succeed. For example, CoreDNS sends aks12345.myvnetid.myregion.internal.cloudapp.net (6 labels) to Azure DNS, but rejects mcr.microsoft.com.myvnetid.myregion.internal.cloudapp.net (8 labels)

This block is implemented in the default server block in the Corefile for the cluster. If needed, this rejection configuration can be disabled by creating custom server blocks for the appropriate domain with a forward plugin enabled:

Create a file named corednsms.yaml and paste the following example configuration. Make sure to update the IP addresses and hostnames with the values for your own environment.

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns-custom # this is the name of the configmap you can overwrite with your changes
  namespace: kube-system
data:
    override-block.server:
       internal.cloudapp.net:53 {
           errors
           cache 30
           forward . /etc/resolv.conf
       }
       reddog.microsoft.com:53 {
           errors
           cache 30
           forward . /etc/resolv.conf
       }

Create the ConfigMap using the kubectl apply configmap command and specify the name of your YAML manifest.
```
kubectl apply -f corednsms.yaml
```
To reload the ConfigMap and enable Kubernetes Scheduler to restart CoreDNS without downtime, perform a rolling restart using kubectl rollout restart.
```
kubectl -n kube-system rollout restart deployment coredns
```

Troubleshooting

For general CoreDNS troubleshooting steps, such as checking the endpoints or resolution, see Debugging DNS resolution.

Configure CoreDNS pod scaling

Sudden spikes in DNS traffic within AKS clusters are a common occurrence due to the elasticity that AKS provides for workloads. These spikes can lead to an increase in memory consumption by CoreDNS pods. In some cases, this increased memory consumption could cause Out of memory issues. To preempt this issue, AKS clusters auto scale CoreDNS pods to reduce memory usage per pod. The default settings for this auto scaling logic are stored in the coredns-autoscaler ConfigMap. However, you may observe that the default auto scaling of CoreDNS pods is not always aggressive enough to prevent Out of memory issues for your CoreDNS pods. In this case, you can directly modify the coredns-autoscaler ConfigMap. Please note that simply increasing the number of CoreDNS pods without addressing the root cause of the Out of memory issue may only provide a temporary fix. If there is not enough memory available across the nodes where the CoreDNS pods are running, increasing the number of CoreDNS pods will not help. You may need to investigate further and implement appropriate solutions such as optimizing resource usage, adjusting resource requests and limits, or adding more memory to the nodes.

CoreDNS uses horizontal cluster proportional autoscaler for pod auto scaling. The coredns-autoscaler ConfigMap can be edited to configure the scaling logic for the number of CoreDNS pods. The coredns-autoscaler ConfigMap currently supports two different ConfigMap key values: linear and ladder which correspond to two supported control modes. The linear controller yields a number of replicas in [min,max] range equivalent to max( ceil( cores * 1/coresPerReplica ) , ceil( nodes * 1/nodesPerReplica ) ). The ladder controller calculates the number of replicas by consulting two different step functions, one for core scaling and another for node scaling, yielding the max of the two replica values. For more information on the control modes and ConfigMap format, please consult the upstream documentation.

Important

A minimum of 2 CoreDNS pod replicas per cluster is recommended. Configuring a minimum of 1 CoreDNS pod replica may result in failures during operations which require node draining, such as cluster upgrade operations.

To retrieve the coredns-autoscaler ConfigMap, you can run the kubectl get configmap coredns-autoscaler -n kube-system -o yaml command which will return the following:

apiVersion: v1
data:
  ladder: '{"coresToReplicas":[[1,2],[512,3],[1024,4],[2048,5]],"nodesToReplicas":[[1,2],[8,3],[16,4],[32,5]]}'
kind: ConfigMap
metadata:
  name: coredns-autoscaler
  namespace: kube-system
  resourceVersion: "..."
  creationTimestamp: "..."

Enable DNS query logging

Add the following configuration to your coredns-custom ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns-custom
  namespace: kube-system
data:
  log.override: | # you may select any name here, but it must end with the .override file extension
        log

Apply the configuration changes and force CoreDNS to reload the ConfigMap using the following commands:

# Apply configuration changes
kubectl apply -f corednsms.yaml

# Force CoreDNS to reload the ConfigMap
kubectl -n kube-system rollout restart deployment coredns

View the CoreDNS debug logging using the kubectl logs command.
```
kubectl logs --namespace kube-system -l k8s-app=kube-dns
```

Next steps

This article showed some example scenarios for CoreDNS customization. For information on the CoreDNS project, see the CoreDNS upstream project page.

To learn more about core network concepts, see Network concepts for applications in AKS.

Chia sẻ qua