How do I allow pods to access the IMDS when using IPVS as kube-proxy scheduler?

Question

How do I allow pods to access the IMDS when using IPVS as kube-proxy scheduler?

Orestis Gklavas 0

We are using an AKS cluster, whose services still use AAD Pod Identity (yes, I know it is deprecated, and yes, it will be migrated to Workload Identity soon).

Due to some performance issues, I decided to try setting kube-proxy to use the IPVS scheduler instead of the default iptables. Indeed, the performance was much higher. However, as I became painfully aware when deploying a nodepool, the NMI pods were unable to contact IMDS to get the required tokens, so my pods would hang at the init state.

I attempted to toy around with kube-proxy IPVS paramters such as excludeCIDRs, with 169.254.169.254/32 as the targt CIDR, but these don't seem to get added to the daemonset's command line params (no configmap is present in the kube-system namespace).

In addition, the "configure kube-proxy" page ( https://learn.microsoft.com/en-us/azure/aks/configure-kube-proxy) points to the AKS Cluster Schema which is supposed to contain the full kube-proxy configuration structure, but I can't seem to be able to find this structure.

Any ideas are welcome. Thanks for taking the time to read through this.

Orestis Gklavas 0 Reputation points

2025-03-11T12:48:03.14+00:00

Hi @Pramidha Yathipathi ,

Well, it seems solution 2 is a bit tricky, since - at least when using the debug pod that Microsoft suggests for connecting to the node - I have no permission to modify iptables. Perhaps I'll give it a shot with direct ssh, however I am not sure about the persistence.

As for the hostNetwork: true in the NMI pods, it was already set.

In any case, your first solution pointed me to something: you have excludedCIDRs as a top level setting, while I was attempting to do the same thing by adding it inside the ipvsConfig block. In either case, this setting does not appear when I describe or get the kube-proxy daemonSet, so I can't be sure if the setting is somehow applied, however it seems to be able to communicate now. If you have any insights on how I can verify if exludedCIDRs has been applied, I'd be grateful.

UPDATE: Not sure why it seemed to work, however when I applied the config to the main cluster the pods still cannot access IMDS. So I guess excludedCIDRs is unsupported.
Pramidha Yathipathi 20 Reputation points Microsoft External Staff

2025-03-12T10:50:17.0533333+00:00
Hi Orestis Gklavas,

Seems like excludedCIDRs isn’t working as expected in AKS. Since it’s not showing up in the kube-proxy DaemonSet,try checking the logs instead:

kubectl logs -n kube-system -l k8s-app=kube-proxy

If excludedCIDRs was applied, you should see some mention of it in there.

– If excludedCIDRs really isn’t working, one option could be setting hostNetwork: true on the application pods that need IMDS access (not just the NMI pods).

– If you get SSH access and can modify iptables.

If the comment was helpful, please don't forget to click "Upvote".

If you have any further queries, please let us know we are glad to help you.

Thank You.
Pramidha Yathipathi 20 Reputation points Microsoft External Staff

2025-03-13T08:15:03.9866667+00:00

Hi Orestis Gklavas,

Just checking in to see if you had a chance to review the comment on your question. Feel free to reach out if you have any further queries.

If you found the information useful, please click "Upvote" on the post to let us know.

Thank You.
Orestis Gklavas 0 Reputation points

2025-03-13T08:23:28.8133333+00:00

Hi @Pramidha Yathipathi . For some strange reason it seems that the cluster where I run the tests right now seems to work (the pods are not failing). Deploying IPVS on the main cluster, however, causes the pods to fail. The only difference I've seen so far, is that I deployed a new nodepool on the main cluster and moved some services there. That's when the failure occured.

I'm currently trying to replicate this condition, by creating a new nodepool on my test cluster. I'll post any interesting update.
Pramidha Yathipathi 20 Reputation points Microsoft External Staff

2025-03-13T09:03:08.17+00:00

Hi Orestis Gklavas,

Please reach out to me, if you have any other queries. Please do tell us after doing workaround on replicating the condition, by creating a new nodepool on test cluster.

Thank you.
Pramidha Yathipathi 20 Reputation points Microsoft External Staff

2025-03-17T09:13:45.9366667+00:00

Hi Orestis Gklavas,

I just wanted to check the status of this case. Please feel free to tag me in the comments for any queries.

Thank you.

1 answer

Your answer

Orestis Gklavas 0 Reputation points

2025-03-11T12:48:03.14+00:00

Hi @Pramidha Yathipathi ,

Well, it seems solution 2 is a bit tricky, since - at least when using the debug pod that Microsoft suggests for connecting to the node - I have no permission to modify iptables. Perhaps I'll give it a shot with direct ssh, however I am not sure about the persistence.

As for the hostNetwork: true in the NMI pods, it was already set.

In any case, your first solution pointed me to something: you have excludedCIDRs as a top level setting, while I was attempting to do the same thing by adding it inside the ipvsConfig block. In either case, this setting does not appear when I describe or get the kube-proxy daemonSet, so I can't be sure if the setting is somehow applied, however it seems to be able to communicate now. If you have any insights on how I can verify if exludedCIDRs has been applied, I'd be grateful.

UPDATE: Not sure why it seemed to work, however when I applied the config to the main cluster the pods still cannot access IMDS. So I guess excludedCIDRs is unsupported.
Pramidha Yathipathi 20 Reputation points Microsoft External Staff

2025-03-12T10:50:17.0533333+00:00

Hi Orestis Gklavas,

Seems like excludedCIDRs isn’t working as expected in AKS. Since it’s not showing up in the kube-proxy DaemonSet,try checking the logs instead:

kubectl logs -n kube-system -l k8s-app=kube-proxy

If excludedCIDRs was applied, you should see some mention of it in there.

– If excludedCIDRs really isn’t working, one option could be setting hostNetwork: true on the application pods that need IMDS access (not just the NMI pods).

– If you get SSH access and can modify iptables.

If the comment was helpful, please don't forget to click "Upvote".

If you have any further queries, please let us know we are glad to help you.

Thank You.
Pramidha Yathipathi 20 Reputation points Microsoft External Staff

2025-03-13T08:15:03.9866667+00:00

Hi Orestis Gklavas,

Just checking in to see if you had a chance to review the comment on your question. Feel free to reach out if you have any further queries.

If you found the information useful, please click "Upvote" on the post to let us know.

Thank You.
Orestis Gklavas 0 Reputation points

2025-03-13T08:23:28.8133333+00:00

Hi @Pramidha Yathipathi . For some strange reason it seems that the cluster where I run the tests right now seems to work (the pods are not failing). Deploying IPVS on the main cluster, however, causes the pods to fail. The only difference I've seen so far, is that I deployed a new nodepool on the main cluster and moved some services there. That's when the failure occured.

I'm currently trying to replicate this condition, by creating a new nodepool on my test cluster. I'll post any interesting update.
Pramidha Yathipathi 20 Reputation points Microsoft External Staff

2025-03-13T09:03:08.17+00:00

Hi Orestis Gklavas,

Please reach out to me, if you have any other queries. Please do tell us after doing workaround on replicating the condition, by creating a new nodepool on test cluster.

Thank you.
Pramidha Yathipathi 20 Reputation points Microsoft External Staff

2025-03-17T09:13:45.9366667+00:00

Hi Orestis Gklavas,

I just wanted to check the status of this case. Please feel free to tag me in the comments for any queries.

Thank you.

Answer 1

Hi Orestis Gklavas,

Switching to IPVS improves performance, but it also changes how traffic is routed, which can cause issues with IMDS access. Since AAD Pod Identity relies on NMI pods reaching IMDS (169.254.169.254), you’ll need to make sure this traffic isn’t affected by IPVS rules.

1.Exclude IMDS from kube-proxy rules

Since AKS doesn’t use a ConfigMap for kube-proxy, you’ll need to set excludeCIDRs during cluster updates. You can try:

az aks update \
  --resource-group <your-resource-group> \
  --name <your-cluster-name> \
  --kube-proxy-config '{"mode":"ipvs","excludeCIDRs":["169.254.169.254/32"]}'

However, this might not always work as expected in AKS.

Manually adjust iptables on nodes If the above doesn’t work, you can manually add an iptables rule to ensure traffic to IMDS is handled properly:

iptables -t nat -A PREROUTING -d 169.254.169.254/32 -j RETURN

This should allow traffic to bypass IPVS.

3.Run NMI pods with hostNetwork: true

Modifying the NMI deployment to use hostNetwork: true can help, as it allows the pods to bypass kube-proxy and communicate directly with IMDS.

Please refer the below documents:

https://learn.microsoft.com/en-us/azure/virtual-machines/instance-metadata-service?tabs=windows

https://learn.microsoft.com/en-us/azure/aks/imds-restriction

If the comment was helpful, please don't forget to click "Upvote".

If you have any further queries, please let us know we are glad to help you.

Thank You.

Share via

How do I allow pods to access the IMDS when using IPVS as kube-proxy scheduler?

1 answer

Your answer