Can we get real client IP in the request to a service running on AKS?

Question

I have an AKS cluster with nginx internal ingress and linkerd (as sidecar)

I also have an Asp.Net Core service deployed there that reports telemetry to an App Insights bucket which does NOT mask client IPs.

However, the reported client IP is of an AKS node rather than the actual client.

I ran 3 requests (using curl on the client machine) and got the following client_IP addresses: These are all IPs of AKS nodes:

mark@L-R910LPKW:~$ k get node -o wide | grep -E '10\.16\.232\.(5|7|8)'
aks-appnodepool1-28692080-vmss000000   Ready    agent   34d   v1.28.3   10.16.232.8   <none>        Ubuntu 22.04.3 LTS   5.15.0-1052-azure   containerd://1.7.5-1
aks-linux-21546199-vmss000001          Ready    agent   34d   v1.28.3   10.16.232.5   <none>        Ubuntu 22.04.3 LTS   5.15.0-1052-azure   containerd://1.7.5-1
aks-toolsnp1-41813426-vmss000000       Ready    agent   34d   v1.28.3   10.16.232.7   <none>        Ubuntu 22.04.3 LTS   5.15.0-1052-azure   containerd://1.7.5-1
mark@L-R910LPKW:~$

The returned IPs are all over the place. 10.16.232.5 corresponds to a system node, 10.16.232.7 - to a node dedicated to running the various tools, including the nginx ingress:

mark@L-R910LPKW:~$ k get pod -A -o wide | grep -E 'internal-ingress-ingress-nginx-controller.+aks-toolsnp1-41813426-vmss000000'
system-d-internal-ingress          internal-ingress-ingress-nginx-controller-558f8f48d7-fkm99        1/1     Running     0              29d     10.254.3.46    aks-toolsnp1-41813426-vmss000000       <none>           <none>
system-d-internal-ingress          internal-ingress-ingress-nginx-controller-558f8f48d7-scdf9        1/1     Running     0              29d     10.254.3.47    aks-toolsnp1-41813426-vmss000000       <none>           <none>
system-d-internal-ingress          internal-ingress-ingress-nginx-controller-558f8f48d7-vd45m        1/1     Running     0              29d     10.254.3.48    aks-toolsnp1-41813426-vmss000000       <none>           <none>
system-d-internal-ingress          internal-ingress-ingress-nginx-controller-558f8f48d7-xm4wt        1/1     Running     0              29d     10.254.3.49    aks-toolsnp1-41813426-vmss000000       <none>           <none>
mark@L-R910LPKW:~$

Finally, 10.16.232.8 maps to the node where the service itself is running along with the linkerd sidecar image:

mark@L-R910LPKW:~$ k get pod -n aida-dev -o wide | grep -E 'aida-api.+aks-appnodepool1-28692080-vmss000000'
aida-api-77586856d-rmmw2                                          2/2     Running   0              16h    10.254.2.88    aks-appnodepool1-28692080-vmss000000   <none>           <none>
mark@L-R910LPKW:~$

This is all very confusing. Why the result is so inconsistent? I can understand if the client IP is that of the application node, because linkerd is a proxy and can replace the incoming client IP. I can understand if the client IP is that of the tools node, because nginx is also a proxy and thus can also replace the client IP. But I do not understand how come we have 3 different possibilities? And in general - is it possible to get the real client IP?

Answer 1

HelloKharitonov, Mark (P01F70F) Welcome to microsoft Q&A,Thankyou for posting your query here. You can set service.spec.externalTrafficPolicy to local in the service definition of your Asp.Net Core service

apiVersion: v1
kind: Service
metadata:
  name: aspnetcore-service
spec:
  type: LoadBalancer
  externalTrafficPolicy: Local
  ports:
  - port: 80
  selector:
    app: aspnetcore-app

This should maintain the client's IP address on the connection to the pod. Hope this helps you, Please accept the answer if it is helpful else revert back with errors if any to investigate in detail.