Редактиране

Споделяне чрез

Capture a TCP dump from a Linux node in an AKS cluster

  • Статия

Networking issues may occur when you're using a Microsoft Azure Kubernetes Service (AKS) cluster. To help investigate these issues, this article explains how to capture a TCP dump from a Linux node in an AKS cluster, and then download the capture to your local machine.

Prerequisites

Note

You can automate TCP capture through a Helm chart, which can run in the background as a DaemonSet. For more information, see this custom GitHub tool for capturing TCP dumps, or use the steps in the following sections.

Step 1: Find the nodes to troubleshoot

How do you determine which node to pull the TCP dump from? You first get the list of nodes in the AKS cluster using the Kubernetes command-line client, kubectl. Follow the instructions to connect to the cluster and run the kubectl get nodes --output wide command using the Azure portal or Azure CLI. A node list that's similar to the following output appears:

$ kubectl get nodes --output wide
NAME                                STATUS   ROLES   AGE   VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION     CONTAINER-RUNTIME
aks-agentpool-34796016-vmss000000   Ready    agent   45h   v1.20.9   10.240.1.81    <none>        Ubuntu 18.04.6 LTS               5.4.0-1062-azure   containerd://1.4.9+azure
aks-agentpool-34796016-vmss000002   Ready    agent   45h   v1.20.9   10.240.2.47    <none>        Ubuntu 18.04.6 LTS               5.4.0-1062-azure   containerd://1.4.9+azure

Step 2: Connect to a Linux node

The next step is to establish a connection to the AKS cluster node that you want to capture the network trace from. For more information, see Create an interactive shell connection to a Linux node.

Step 3: Make sure tcpdump is installed

After you've established a connection to the AKS Linux node, verify the tcpdump tool has been previously installed on a node by running tcpdump --version. If tcpdump hasn't been installed, the following error text appears:

# tcpdump --version
bash: tcpdump: command not found

Then install tcpdump on your pod by running the Advanced Package Tool's package handling utility, apt-get:

apt-get update && apt-get install tcpdump

If tcpdump is installed, something similar to the following text appears:

# tcpdump --version
tcpdump version 4.9.3
libpcap version 1.8.1
OpenSSL 1.1.1  11 Sep 2018

Step 4: Create a packet capture

To capture the dump, run the tcpdump command as follows:

# tcpdump --snapshot-length=0 -vvv -w /capture.cap
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
Got 6

Note

Running tcpdump without using filtering parameters can significantly increase the size of the Packet Capture (PCAP) file, especially for long runs. Therefore, we recommend that you add filters, such as source, destination, and port. For example:

  • tcpdump dst 192.168.1.100
  • tcpdump dst host.mydomain.com
  • tcpdump port http or port ftp or port smtp or port imap or port pop3 or port telnet

While the trace is running, replicate your issue many times. This action make sure that the issue is captured within the TCP dump. Note the time stamp while you replicate the issue. To stop the packet capture when you're done, press Ctrl+C:

# tcpdump -s 0 -vvv -w /capture.cap
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
^C526 packets captured
526 packets received by filter
0 packets dropped by kernel

Step 5: Transfer the capture locally

After you complete the packet capture, identify the helper pod so you can copy the dump locally. Open a second console, and then get a list of pods by running kubectl get pods, as shown below.

$ kubectl get pods
NAME                                                    READY   STATUS    RESTARTS   AGE
azure-vote-back-6c4dd64bdf-m4nk7                        1/1     Running   0          3m29s
azure-vote-front-85b4df594d-jhpzw                       1/1     Running   0          3m29s
node-debugger-aks-nodepool1-38878740-vmss000000-jfsq2   1/1     Running   0          60s

The helper pod has a prefix of node-debugger-aks, as shown in the third row. Replace the pod name, and then run the following kubectl command. These commands retrieve the packet capture for your Linux node.

kubectl cp node-debugger-aks-nodepool1-38878740-vmss000000-jfsq2:/capture.cap capture.cap

Note

If the chroot /host command was used when entering the debug pod, add /host before /capture.cap for the source file.

Third-party information disclaimer

The third-party products that this article discusses are manufactured by companies that are independent of Microsoft. Microsoft makes no warranty, implied or otherwise, about the performance or reliability of these products.

Contact us for help

If you have questions or need help, create a support request, or ask Azure community support. You can also submit product feedback to Azure feedback community.