VPN Gateway max number of NICs connected in (peered) networks

Question

VPN Gateway max number of NICs connected in (peered) networks

Bas Pruijn 956

Our VPN Gateway has approx 500 private endpoints connected, shared over several VNETs. Now the VPN gateway is (according to Microsoft) at 100% CPU load. We already upgraded from VpnGw1 to VpnGw2, but the error still occurs.

Can you please respond how many NICs or Private endpoints you have connected to your VPN Gateway. Please also mention the SKU.

At the moment nobody can tell me what SKU we are required to have to be able to allow for our workload. We use VNET peering with the 'remote virtual network's gateway' option, since this is required for the VPN to work.

GitaraniSharma-MSFT 50,096 Reputation points Microsoft Employee Moderator

2023-06-21T10:10:20.4233333+00:00

Hello @Bas Pruijn ,

Welcome to Microsoft Q&A Platform. Thank you for reaching out & hope you are doing well.

I'm not sure if I understand your question correctly.

There is no such limit of max number of NICs connected to VPN gateway.

Only the below limits are applicable:

Number of private endpoints per virtual network - 1000

Refer: https://learn.microsoft.com/en-us/azure/azure-resource-manager/management/azure-subscription-service-limits#private-link-limits

Virtual networks per region per subscription - 1,000

Virtual network peerings per virtual network - 500

Refer: https://learn.microsoft.com/en-us/azure/azure-resource-manager/management/azure-subscription-service-limits#azure-resource-manager-virtual-networking-limits

VNet Address Prefixes per VPN gateway - 600

Refer: https://learn.microsoft.com/en-us/azure/azure-resource-manager/management/azure-subscription-service-limits#virtual-network-gateway-limits

The VPN connections and throughput depends on the SKU of your VPN gateway used, which is listed in the below doc:

https://learn.microsoft.com/en-us/azure/vpn-gateway/vpn-gateway-about-vpngateways#benchmark

Could you please explain your issue in a bit more detail for further discussion?

Regards,

Gita
Bas Pruijn 956 Reputation points

2023-06-21T12:14:38.6166667+00:00

Hi @GitaraniSharma-MSFT ,

Thank you for reaching out. We have a VPN gateway set up in our HUB network. This hub is peered to 7 spoke networks. Each of these networks is a /16 network, containing a number of subnets. In each of the spoke networks we have azure functions, SQL databases, Keyvaults and storage accounts connected via private endpoints. in total we have over 500 private endpoints. For our Azure Functions we also use VNET integration.

Our developers and monitoring team use the VPN gateway to create a P2S connection to access the resources while developing, debugging or testing and monitoring of the environment. We do not use S2S connections.

Until recently we were using VpnGW1 without any issues. Recently we added an application gateway to one of the subnets and all P2S users are complaining about dropping connections. Resource health of our VPN Gateway shows that our gateway is frequently going into standby mode.

We contacted Microsoft Support. They noticed our Gateway was experiencing an unusual high CPU load. According to them we needed to upgrade to a higher SKU. So we did migrate to VpnGW2. After a couple of days we again start experiencing frequent dropping connections. Again due to the gateway going to standby mode. Their solution: upgrade again.

Since we are experiencing this within a startup company, having a VPN connection costing approx $1.000 per month for only max 6 concurrent users is close to unacceptable. Furthermore, since we are planning to extend the application landscape with a couple of hundred additional private endpoints.

The support team is claiming the high CPU load is already showing in their statistics from before May 14th (when we started experiencing issues). The high load seems to be caused by lsaas process. And This is due to the number of NMAgent calls that GW are getting. This is not something we can either monitor nor change.

So my question is: why is the VPN gateway not working with 'only' 500 private endpoint connections?

The cloud engineer linked to my support case is claiming our setup is using 'way to many' private endpoints. However, nobody could tell me how to change the set-up using less private endpoints and still remain secure. Furthermore, nobody could tell me how many endpoints can be handled by a VPN gateway of a certain SKU.
Joe Carlyle 661 Reputation points MVP

2023-06-21T13:53:19.5433333+00:00

Private endpoints cannot be aware of the VPNGW. That's a ridiculous statement by the engineer.

Have you tried simply resetting the gateway? I don't see how the upgrade is relevant to your concurrent connections, it appears more like a fault to me. https://learn.microsoft.com/en-us/azure/vpn-gateway/reset-gateway
Bas Pruijn 956 Reputation points

2023-06-22T08:06:51.6466667+00:00

Hi @Joe Carlyle . Thank you for responding to my question. I do agree that it really sounds strange the private endpoint is constantly registering itself at the VPN Gateway. I would expect the gateway to handle this. Since my switch can do similar mapping of IP addresses with a very, very slow CPU and very limited memory, I do net expect the VPN gateway to go haywire on approx 500 addresses.

of course we did a reset of the gateway. This did not fix it. After that, we did a second reset (as advised in the documentation) and that still did not fix it. However, a reset takes current set-up and configuration from all the internal VpnGW processes and uses them on a different VM. So, if one of the processes on the GW is misconfigured, the reset-replica would be too.
Joe Carlyle 661 Reputation points MVP

2023-06-22T08:24:47.8966667+00:00

If this was me, I would rebuild a hub vnet and gateway in parallel then cut my P2S connections over, however, that may be very difficult for your team.

I would continue to push support for an escalation and more detailed analysis of the issue.
GitaraniSharma-MSFT 50,096 Reputation points Microsoft Employee Moderator

2023-06-26T10:34:45.7666667+00:00
Hello @Bas Pruijn , We wish to take a look on your existing support request to find more details regarding your issue. Request you to please send an email with subject line "ATTN gishar | VPN Gateway max number of NICs connected in (peered) networks" to AzCommunity[at]Microsoft[dot]com with the following details, I will follow-up with you.

Reference this Q&A thread

Your Azure Subscription ID

Your support request number.

Note: Do not share any PII data as a public comment.

We will post a summarized answer once the issue is resolved.

Regards,

Gita
Bas Pruijn 956 Reputation points

2023-06-26T15:08:35.63+00:00

@Joe Carlyle I would love to do so. However, recreating the VPN gateway would lead to some issues. Due to a (quite strict) network-IP-plan I need to use the same network IP ranges if I recreate the gateway. This implies I need to remove the gateway first and then rebuild.

Problem with this is that during the recreation of the gateway the Dev and Monitoring teams cannot access the network. This is a difficult planning in a 24/7 organisation.

1 answer

Your answer

GitaraniSharma-MSFT 50,096 Reputation points Microsoft Employee Moderator

2023-06-21T10:10:20.4233333+00:00

Hello @Bas Pruijn ,

Welcome to Microsoft Q&A Platform. Thank you for reaching out & hope you are doing well.

I'm not sure if I understand your question correctly.

There is no such limit of max number of NICs connected to VPN gateway.

Only the below limits are applicable:

Number of private endpoints per virtual network - 1000

Refer: https://learn.microsoft.com/en-us/azure/azure-resource-manager/management/azure-subscription-service-limits#private-link-limits

Virtual networks per region per subscription - 1,000

Virtual network peerings per virtual network - 500

Refer: https://learn.microsoft.com/en-us/azure/azure-resource-manager/management/azure-subscription-service-limits#azure-resource-manager-virtual-networking-limits

VNet Address Prefixes per VPN gateway - 600

Refer: https://learn.microsoft.com/en-us/azure/azure-resource-manager/management/azure-subscription-service-limits#virtual-network-gateway-limits

The VPN connections and throughput depends on the SKU of your VPN gateway used, which is listed in the below doc:

https://learn.microsoft.com/en-us/azure/vpn-gateway/vpn-gateway-about-vpngateways#benchmark

Could you please explain your issue in a bit more detail for further discussion?

Regards,

Gita
Joe Carlyle 661 Reputation points MVP

2023-06-21T13:53:19.5433333+00:00

Private endpoints cannot be aware of the VPNGW. That's a ridiculous statement by the engineer.

Have you tried simply resetting the gateway? I don't see how the upgrade is relevant to your concurrent connections, it appears more like a fault to me. https://learn.microsoft.com/en-us/azure/vpn-gateway/reset-gateway
Bas Pruijn 956 Reputation points

2023-06-22T08:06:51.6466667+00:00

Hi @Joe Carlyle . Thank you for responding to my question. I do agree that it really sounds strange the private endpoint is constantly registering itself at the VPN Gateway. I would expect the gateway to handle this. Since my switch can do similar mapping of IP addresses with a very, very slow CPU and very limited memory, I do net expect the VPN gateway to go haywire on approx 500 addresses.

of course we did a reset of the gateway. This did not fix it. After that, we did a second reset (as advised in the documentation) and that still did not fix it. However, a reset takes current set-up and configuration from all the internal VpnGW processes and uses them on a different VM. So, if one of the processes on the GW is misconfigured, the reset-replica would be too.
Joe Carlyle 661 Reputation points MVP

2023-06-22T08:24:47.8966667+00:00

If this was me, I would rebuild a hub vnet and gateway in parallel then cut my P2S connections over, however, that may be very difficult for your team.

I would continue to push support for an escalation and more detailed analysis of the issue.
GitaraniSharma-MSFT 50,096 Reputation points Microsoft Employee Moderator

2023-06-26T10:34:45.7666667+00:00

Hello @Bas Pruijn , We wish to take a look on your existing support request to find more details regarding your issue. Request you to please send an email with subject line "ATTN gishar | VPN Gateway max number of NICs connected in (peered) networks" to AzCommunity[at]Microsoft[dot]com with the following details, I will follow-up with you.

Reference this Q&A thread

Your Azure Subscription ID

Your support request number.

Note: Do not share any PII data as a public comment.

We will post a summarized answer once the issue is resolved.

Regards,

Gita
Bas Pruijn 956 Reputation points

2023-06-26T15:08:35.63+00:00

@Joe Carlyle I would love to do so. However, recreating the VPN gateway would lead to some issues. Due to a (quite strict) network-IP-plan I need to use the same network IP ranges if I recreate the gateway. This implies I need to remove the gateway first and then rebuild.

Problem with this is that during the recreation of the gateway the Dev and Monitoring teams cannot access the network. This is a difficult planning in a 24/7 organisation.

Answer 1

Bas Pruijn 956

Hi All.

The work-around we implemented is the following:

remove all the 'use remote gateway' items from the network peering
create an Azure Firewall in the hub network
configure the firewall to ALWAYS use SNAT
create a route table (UDR) on the gateway subnet, that is forwarding all the internal network addresses to the Firewal

Now all the VPN traffic is forwarded to the firewall. The firewall is now proxying the traffic to the peered network. The answer is sent back to the VPN client by the firewall. We were able to downscale the VPN Gateway back to SKU 1.

Even though we do not know the root cause for this issue, we can work using this work around.

GitaraniSharma-MSFT 50,096 Reputation points Microsoft Employee Moderator

2023-07-17T13:31:51.89+00:00

@Bas Pruijn , thank you for sharing the workaround. Glad to hear that the issue is mitigated now.

As the support & backend team previously reported, the issue was due to the number of NMAgent calls that the VPN gateway is getting. The Gateway VNET has 7 Vnet peerings which might have a lot of VMs/services causing high number of NMAgent calls. If the gateway is being used as remote gateway from multiple VNets, all the VMs/services in those VNets will fetch the routes using NMAgent.

Now that you’ve removed all the 'use remote gateway' options from the Vnet peering and created an Azure Firewall in the hub network to route all VPN traffic through the firewall using UDR, the high number of NMagent calls on the VPN gateway is reduced, which was the root cause of this issue. Even though we couldn’t isolate which services were causing this high number of NMAgent calls (as there were 500 private endpoints connected to the VPN gateway shared over several Vnets), we know that the load on the VPN gateway was due to these calls and by removing this load, we were able to reduce the high CPU utilization on the VPN gateway.

This workaround might help others facing similar issues.

Regards,

Gita

Share via

VPN Gateway max number of NICs connected in (peered) networks

1 answer

Your answer