Azure route server causes loss of connectivity from on-prem to azure

Steven Johnston 40

On prem network connectivity into azure is by means of Cisco SDWAN terminating in virtual wan hub which routes into azure vnets via Palo Alto NVA's. Deploying an Azure route server in the NVA vnet causes loss of connectivity from on prem to azure. Loss of connectivity occurs precisely when the route server finishes deploying, don't even get the chance to add BGP peerings

Documentation states this..

"When you create or delete a Route Server in a virtual network that contains a virtual network gateway (ExpressRoute or VPN), expect downtime until the operation is complete. If you have an ExpressRoute circuit connected to the virtual network where you're creating or deleting the Route Server, the downtime doesn't affect the ExpressRoute circuit or its connections to other virtual networks."

Would a virtual wan hub connection fall under this caveat? Should i wait longer after deployment for connectivity to be restored?

thanks

KapilAnanth-MSFT 35,336 Reputation points Microsoft Employee

2023-08-16T13:10:48.6866667+00:00

@Steven Johnston

Welcome to the Microsoft Q&A Platform. Thank you for reaching out & I hope you are doing well.

I understand that you are using a custom NVA deployed into a Spoke VNET

i.e, https://learn.microsoft.com/en-us/azure/virtual-wan/scenario-route-through-nvas-custom

Please let me know if my understanding is incorrect.

Can you please provide more details on how the routing it set up from vHub to the NVA?

Thanks,

Kapil
Steven Johnston 40 Reputation points

2023-08-16T13:38:08.5766667+00:00

That's correct NVA is deployed in a spoke vNet

I am using static routing in the vHub towards the NVA, next hop is spoke vNet

Diagram explains the topology, i am deploying the route server to new subnet in vnet-hub
KapilAnanth-MSFT 35,336 Reputation points Microsoft Employee

2023-08-16T14:19:22.3933333+00:00
@Steven Johnston

Thanks for sharing a detailed information of your architecture.

I see traffic to VNET1 and VNET2 are going via the NVA (Palo Alto)

However, traffic to the VNET-HUB is going directly.

When you say OnPrem connectivity is disrupted,

I can understand that connectivity to VNET1 and VNET2 are going down

But can you confirm if you are able to connect to any resource in the VNET-HUB?

To isolate this issue,

I would suggest we test the connectivity to VNET-HUB

You can spin a dummy VM in this VNET and check connectivity from OnPrem.

Also, during the disruption, can you please effective routing table of any VM in VNET-HUB and (VNET-1 or VNET-2)

Refer: https://learn.microsoft.com/en-us/azure/network-watcher/diagnose-vm-network-routing-problem#view-details-of-a-route

If you have a test or dev environment, can you please do the above and let me know?

Cheers,

Kapil.
Steven Johnston 40 Reputation points

2023-08-16T14:39:52.4033333+00:00

I'll spin up a VM in vnet Hub for testing. I'm not sure if traffic from Vnet1 and Vnet2 was affected or not but what i did lose for sure was i connectivity to mgt and priv ip of both the firewalls

The vHub routing table was unaffected so i can only presume the devices in vnet-hub somehow picked up a new (empty?) routing table from the route server and perhaps black holing traffic
Steven Johnston 40 Reputation points

2023-08-16T15:26:52.6633333+00:00

It looks like all the routes from vHub are lost when the route server becomes active. routes for vnet peerings are still there

before i deploy the route server the VM has a full routing table, with the on-prem routes populated and a next hop of the vHub BGP devices.

after deploying route server all the on-prem routes are gone.
KapilAnanth-MSFT 35,336 Reputation points Microsoft Employee

2023-08-24T11:59:33.96+00:00

@Steven Johnston

Reaching out to check if there are any questions on this.

Please let us know if we can be of any further assistance here.

Thanks,

Kapil

Please Accept an answer if correct. Original posters help the community find answers faster by identifying the correct answer.

Accepted answer

KapilAnanth-MSFT 35,336 Reputation points Microsoft Employee

2023-08-18T09:22:53.44+00:00
@Steven Johnston

Welcome to the Microsoft Q&A Platform. Thank you for reaching out & I hope you are doing well.

I understand that you would like to deploy a Route Server in a VNET connected to a vWAN vHub.

I checked this internally and our Product Group confirmed this is not a supported scenario.

Deploying a Route Server into the Spoke VNET will stop the VNET from learning the routes from the Hub.

I have created a request to update the FAQ section to highlight this here :

Can a spoke VNet have a Azure Route Server

If you could let us know the requirement for deploying a ARS into the SpokeVNET,

We could try and suggest some alternatives

In your case, I believe you were planning to eliminate UDRs and use BGP to route traffic from VNets VNET-1 and VNET-2

In this case, based on your architecture, I am afraid we should rely on the UDRs only

NOTE

Azure does not recommend multi-hub scenarios.

https://learn.microsoft.com/en-us/azure/architecture/networking/hub-spoke-vwan-architecture

The general recommendation is to use Azure Firewall or NVA Partners in the vHUB directly to secure traffic

Should there be any follow-up questions or concerns, please let us know and we shall try to address them.

Thanks,

Kapil

Please Accept an answer if correct. Original posters help the community find answers faster by identifying the correct answer.
Please sign in to rate this answer.
Steven Johnston 40 Reputation points

2023-08-18T15:22:17.4133333+00:00

Thank you for checking.

My intention was to use the RS to peer with the two NVA firewalls, and set as-path prepend on one firewall for specific set of prefixes, in order to ensure traffic for those prefixes exits the NVA with shortest as-path.

The reason for this is that there are S2S VPN terminating on one of the NVA, we use static routes in the UDR to direct traffic towards the NVA hosting the VPN prefixes at the moment.

In future we will build a backup S2S VPN on the other NVA with the same prefixes, so we need a mechanism like the Route Server that can populate the spoke UDR dynamically in event that the primary NVA fails.

I was thinking perhaps i can BGP peer the NVA's with the vWan Hub instead of using Route Server , but i am not sure how to configure the routing in the spoke vNets. Peer with the vWan instead of the vNet-Hub perhaps?

KapilAnanth-MSFT 35,336 Reputation points Microsoft Employee

2023-08-19T12:18:02.3733333+00:00

@Steven Johnston

Thanks for letting us know.

Your scenario is documented here : BGP peering with a virtual hub

However, I am afraid that a user-defined route (UDR) to point to NVA IP would be required in the spoke VNETs "vnet1" and "vnet2".

Cheers,

Kapil

Please Accept an answer if correct. Original posters help the community find answers faster by identifying the correct answer.
Sign in to comment

Azure route server causes loss of connectivity from on-prem to azure

0 additional answers