Load balancer Intermittent connection

Hari Guvvala 20

I’m using a Standard Internal Load Balancer with a private frontend IP in Azure. However, the VMs in the backend pool lose internet access once a load balancing rule is defined. I’ve configured a NAT gateway with a standard SKU, and both the load balancer and backend health probe statuses are 100%. When I troubleshoot the load balancer and VMs, no defects are found. Can you suggest how to resolve this issue?

KapilAnanth-MSFT 47,196 Reputation points Microsoft Employee

2024-08-09T10:16:48.47+00:00
@Hari Guvvala , from your verbatim - I see that the issue is intermittent.

Correct me if I am wrong.

I suspect that you are facing SNAT port exhaustion

As next steps,

I suggest you Scale SNAT for NAT gateway by attaching additional IP Addresses and see if the performance improves.

Each NAT gateway public IP address provides 64,512 SNAT ports to make outbound connections.

Cheers,

Kapil
Hari Guvvala 20 Reputation points

2024-08-09T11:01:54.1033333+00:00
Hi @KapilAnanth-MSFT

Please find my inline comments below.

I suggest you Scale SNAT for NAT gateway by attaching additional IP Addresses and see if the performance improves. => I have added NAT gateway public IP prefixes and able to ping those IP's in VMs , but still facing intermittent connection issue.

1 answer

KapilAnanth-MSFT 47,196 Reputation points Microsoft Employee

2024-08-08T13:21:17.7366667+00:00
@Hari Guvvala ,

Welcome to the Microsoft Q&A Platform. Thank you for reaching out & I hope you are doing well.

In Standard ILB,

Backend VMs do not have internet connectivity - this is expected.

However, with NAT gateway or Instance level Public IP - you should be able to have internet connection.

As next steps,

Can you confirm if the Public IP you see while accessing internet is same as NAT Gateway's IP ?

It is possible that you are using default outbound access in the VMs

Can you confirm if the NAT Gateway is attached to the Subnet of the backend VMs and not the subnet of the front end Private IP of the ILB?

Can you confirm there are no route tables attached to the Subnet of the backend VMs?

Make sure the configuration of NAT Gateway is correct as described in Integrate a NAT gateway with an ILB

Cheers,

Kapil
Please sign in to rate this answer.
Hari Guvvala 20 Reputation points

2024-08-09T04:38:11.9733333+00:00

Hi @KapilAnanth-MSFT ,

Thanks for your indeed support.

Please find my below inline comments.

Can you confirm if the Public IP you see while accessing internet is same as NAT Gateway's IP ?

It is possible that you are using default outbound access in the VMs => Yes, I can be able to see the NAT Gateway IP in VM internet access.

Can you confirm if the NAT Gateway is attached to the Subnet of the backend VMs and not the subnet of the front-end Private IP of the ILB? => Yes, NAT Gateway is attached to the subnet of the backend VMs. Initially both VM subnet and front-end subnet mapped with NAT Gateway. After changing frontend IP without NAT gateway, I can see much better improvement, but sometimes I can see the connectivity lost. Are we missing any configuration, can you please suggest.

Can you confirm there are no route tables attached to the Subnet of the backend VMs? => No, route tables attached to the subnet.

Make sure the configuration of NAT Gateway is correct as described in Integrate a NAT gateway with an ILB => Yes, I have done proper NAT gateway configuration.

KapilAnanth-MSFT 47,196 Reputation points Microsoft Employee

2024-08-09T11:21:52.5066667+00:00

@Hari Guvvala

May I ask how exactly are you confirming there are intermitted connectivity issues?

Also, are you testing this with a single destination endPoint or general Internet?

Like www.bing.com and www.google.com or just a specific Public IP

Can you check the Dropped Packets metrics of the NAT Gateway during the time of the issue?

Hari Guvvala 20 Reputation points

2024-08-09T11:45:46.0366667+00:00

Hi @KapilAnanth-MSFT

Please find my below inline comments.

May I ask how exactly are you confirming there are intermitted connectivity issues? =>When I attempt to use telnet with the LB front-end IP and health probe port (example: telnet 10.0.X.X 6001) on the VM, it sometimes connects and sometimes doesn’t. It tries to connect but eventually times out.

Also, are you testing this with a single destination endPoint or general Internet?

Like www.bing.com and www.google.com or just a specific Public IP => Yes, I am testing it single destination, i.e frontendIP.

Can you check the Dropped Packets metrics of the NAT Gateway during the time of the issue? => Verified it Packets Dropped count is 0 (zero) and

Datapath Availability: 100%,

Total SNAT Connection Count: 2k

Attempted Connections: 1.3k

Failed Connections: 0(zero)

Regards,

Hari

KapilAnanth-MSFT 47,196 Reputation points Microsoft Employee

2024-08-12T05:42:27.9266667+00:00

@Hari Guvvala ,

I am afraid I am confused by your statements.

You mentioned "the VMs in the backend pool lose internet access once a load balancing rule is defined"

You also mentioned "I am testing it single destination, i.e frontendIP.", "When I attempt to use telnet with the LB front-end IP and health probe port (example: telnet 10.0.X.X 6001) on the VM"

If you are using a "Standard Internal Load Balancer with a private frontend IP",

Doesn't that make the destination a private IP (Non-Internet)

Moreover, may I ask why are you even doing this?

This is not supported by design.

See : FAQ | Can I access the frontend of my internal load balancer from the participating backend pool VM?

This mentions Azure Load Balancer doesn't support this scenario.

For a more detailed explanation, see Access of the internal load balancer frontend from the participating load balancer backend pool VM

If an internal load balancer is configured inside a virtual network, and one of the participant backend VMs is trying to access the internal load balancer frontend, failures can occur when the flow is mapped to the originating VM. This scenario isn't supported.

Please let us know if we can be of any further assistance here.

Thanks,

Kapil

Please Accept an answer if correct.

Original posters help the community find answers faster by identifying the correct answer.

Hari Guvvala 20 Reputation points

2024-08-14T07:14:13.2733333+00:00

Hi @KapilAnanth-MSFT ,I have tested on the Standard Internal Load Balancer in two scenarios:

one with NAT gateway integration and the other without it. In both cases, we encountered intermittent connectivity issues and every 10 seconds connections getting dropped. Additionally, I attempted to use the Load Balancer’s frontend IP with another subnet, which was not mapped with the NAT gateway. This also resulted in intermittent connectivity issues with the private IP.

The health probe status for both the Internal Load Balancer (ILB) and Virtual Machines (VMs) is at 100%. During troubleshooting of both the ILB and backend VMs, no defects were found.

I was gone throw this link to sort out the connectivity issue, using NAT Gateway scenario but no luck. https://www.insentragroup.com/gb/insights/geek-speak/cloud-and-modern-data-center/azure-load-balancer-breaks-the-internet/

Note: As per your requested questions commented below.

Moreover, may I ask why are you even doing this?: Deployed code changes into VM , using ILB Fronted-IP in APIM backend URL to get the response.

Could you please advise me on the most effective method to resolve this issue.

Regards,

Hari

KapilAnanth-MSFT 47,196 Reputation points Microsoft Employee

2024-08-15T06:12:57.2+00:00

@Hari Guvvala ,

Please review my previous comment.

#1

Your ask is not supported.

From FAQ, Can I access the frontend of my internal load balancer from the participating backend pool VM?

No, Azure Load Balancer doesn't support this scenario.

Internal load balancers don't translate outbound originated connections to the front end of an internal load balancer because both are in private IP address space. Public load balancers provide outbound connections from private IP addresses inside the virtual network to public IP addresses. For internal load balancers, this approach avoids potential SNAT port exhaustion inside a unique internal IP address space, where translation isn't required.

A side effect is that if an outbound flow from a VM in the backend pool attempts a flow to front end of the internal load balancer in its pool and is mapped back to itself, the two legs of the flow don't match. Because they don't match, the flow fails. The flow succeeds if the flow didn't map back to the same VM in the backend pool that created the flow to the front end.

The bold part from above explains why the issue is intermittent.

#2

Also, I am afraid your understanding of how NAT Gateway works is incorrect.

NAT Gateway can only provide outbound connectivity to Public IP workloads.

It cannot provide connectivity to Private IP Address (which is your case)

So NAT Gateway is not the solution here

This means,

NAT Gateway can provide your VMs with internet access

say www.bing.com or www.google.com

But NAT Gateway cannot help you connect to any private IP for that matter (not only Load Balancer Frontend)

#3

If you require that the backend pool connect to it's Load Balancer's front end,

You must use Application Gateway.

See the resolution section in Access of the internal load balancer frontend from the participating load balancer backend pool VM

This means, you have to revamp your architecture

I hope this makes things clear.

Thanks,

Kapil

Please Accept an answer if correct.

Original posters help the community find answers faster by identifying the correct answer.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Load balancer Intermittent connection

1 answer

Your answer