IP issues with Azure Kubernetes Service Load Balancer over UDP

Question

My company is using an Azure Kubernetes Service. We have an internal load balancer in place which should handle UDP communication through a VPN connection:

apiVersion: v1  
kind: Service  
metadata:  
  name: middleware-5683  
  labels:  
    app: middleware  
    env: test  
  annotations:  
    service.beta.kubernetes.io/azure-load-balancer-internal: "true"  
spec:  
  type: LoadBalancer  
  loadBalancerIP: 10.1.0.240  
  sessionAffinity: None  
  externalTrafficPolicy: Local  
  ports:  
    - name: middleware-5683  
      port: 5683  
      targetPort: 5683  
      protocol: UDP  
  selector:  
    app: middleware  

Regarding this documentation, we currently have option 4 in place, which is labeled "second worst" (why?).

We have a bunch of devices sending messages to the load balancer. These devices receive responses originated from the load balancer IP (10.1.0.240), if they send messages in short intervals. This is the behaviour we expect.

When they pause sending, for about a minute or longer, they suddenly start to receive responses from private backend servers (IPs unequal to 10.1.0.240):

[00:02:12.631,286] <dbg> at_cmd.socket_thread_fn: Listening on socket  
[00:02:12.805,114] <dbg> udp_receive: checking for reply from [*10.1.0.240*]  
[...]  
[00:05:09.279,113] <dbg> at_cmd.socket_thread_fn: Listening on socket  
[00:05:09.452,972] <dbg> udp_receive: checking for reply from [*10.1.0.97*]  

Why can this happen? We stumbled across a feature which is called "Floating IP", but the documentation could't help us in this case.

Answer 1

For your option 4 and why it is the "second worst":

SNAT Exhaustion. If your application does not have a high load, this is not a concern for you. If it does, you should be able to configure outbound rules for your Azure Load Balancer to resolve this. The doc you linked applies to outbound internet connections, so it is not fully applicable in your scenario.

The documentation goes into detail about Public load balancer outbound communication, but does not discuss what happens for internal load balancer outbound communication.

If you need to clearly define the addresses that the responses are sent from, your best bet is through an outbound rule.