vnet peering, NVA, and BGP

AdamTyler-3590 70 Reputation points
2024-04-17T22:26:44.1966667+00:00

Hello everyone. I've been trying to achieve a robust network design for infrastructure on the Azure platform. I've established vnets in both West US and South Central US. The vnets are peered and the following two options are enabled.

wus-vnet
Allow wus-vnet to access scus-vnet
Allow wus-vnet to receive forwarded traffic from scus-vnet

scus-vnet
Allow scus-vnet to access wus-vnet
Allow scus-vnet to receive forwarded traffic from wus-vnet

I've deployed an NVA (FortiGate) perimeter router to each vnet. The goal being that either NVA can serve as a perimeter gateway for both vnets in the event one or the other NVA failed or was down for maintenance. An alternative if you will to an HA firewall/router setup which IMO isn't a great option on a VM platform. This design also lends itself well to leveraging Azure Site Recovery between regions with the perimeter already staged and ready.

The real trick was getting all of the routing to update within our on-prem environment (which speaks OSPF) and Azure network (which is only compatible with BGP) to update automatically and correctly under different failover scenarios. There is a smattering of route-maps that exist on these NVAs handling route suppression and redistribution of enterprise routes. As well as the propagation of the zero route from Azure out to the internet.

All seems to be working, but I'm struggling to find anyone else that has been down this road. It seems you can run into someone who knows networking, but not Azure or the reverse being strong in Azure, but lacking in networking experience.

What I find troubling is the FAQ related to the Azure route server I am leveraging to accomplish this design.

https://learn.microsoft.com/en-us/azure/route-server/route-server-faq

Specifically, the question "Can I peer two Azure Route Servers in two peered virtual networks and enable the NVAs connected to the Route Servers to talk to each other"

This isn't exactly what I am doing, but close. In my case, each of the NVAs has a peering with the Route Server in each vnet. I'm using AS path prepending to make the remote NVA look less desirable. So the remote NVA isn't used unless the local NVA is down.

The other detail that caught my eye on the FAQ page was the question "Does Azure Route Server support virtual network peering?" Microsoft's answer to this question talks about enabling the vnet peering feature "Use the remote virtual network's gateway or Route Server". Initially I was hopeful this basically would peer two route servers together, but it seems this can't be enabled if you have a Route Server deployed in both vnets. You get an error with language like, "Failed to save virtual network peering vnet already has a gateway configured". This peering feature appears to be geared for a vnet that has no gateway or Route Server in it, which is not compatible with my design.

A high-level overview of the design below.. So here I am with a working solution that seems to meet our needs. We have a point of entry into our Azure network from both West US and South Central US and all workloads reconverge automatically if either NVA is taken offline. Obviously, latency goes up a bit if WUS flows through SCUS to the outside world, but still acceptable. Have I committed a network or Azure crime here? I'm meeting with a 3rd party to review this and provide input soon. Would be great to get this "certified" in some way before we start moving critical workloads onto the platform. I haven't seen a whitepaper from Microsoft on config examples or suggestions.
User's image

Azure Virtual Network
Azure Virtual Network
An Azure networking service that is used to provision private networks and optionally to connect to on-premises datacenters.
2,147 questions
0 comments

2 answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. Silvia Wibowo 3,011 Reputation points Microsoft Employee
    2024-04-18T06:45:39.71+00:00

    Hi @AdamTyler-3590 , I understand that you have on-premises network with OSPF and you want to integrate the dynamic routing into Azure using Azure Route Server and NVA that supports both OSPF and BGP.

    Please note that Azure Route Server is a service designed with high availability. Your route server has zone-level redundancy if you deploy it in an Azure region that supports Availability Zones. It means that if you want to achieve redundancy in terms of AZ (Availability Zone), you can deploy Azure Route Server in one region that supports AZ (SouthCentralUS, WestUS2, or WestUS3; WestUS does not support AZ). Azure Route Server will have 2 instances. We recommend peering each NVA with both Route Server instances to ensure that virtual network routes are advertised over the NVA connections and achieve high availability.

    However, if you require region redundancy, then you need Azure Route Server in each of the regions. The BGP peering from your NVA will be to each of the Route Server instance (2 instances in each region, if the region supports AZ). 2 NVAs --BGPpeering-- 4 Route Server instances, total of 8 BGP peerings.

    Another requirement: Your NVA must support multi-hop external BGP.

    Please accept an answer if correct. Original posters help the community find answers faster by identifying the correct answer. Here is how.

  2. KapilAnanth-MSFT 35,086 Reputation points Microsoft Employee
    2024-04-22T16:18:53.2933333+00:00

    @AdamTyler-3590 ,

    Welcome to the Microsoft Q&A Platform. Thank you for reaching out & I hope you are doing well.

    I understand that you are trying to achieve a Redundant connection with 2 regions and it is currently working

    To address your query,

    1."Have I committed a network or Azure crime here?"

    • This is a strong word, customers are free to explore set ups that suit their requirements.
    • With that said, You are using what is called a Multi Hub Model.
      • You have 2 VNETs, each acting as a Hub and has it's own VPN
    • Azure however, recommends the configurations and set-ups follow a Single Hub- Multi Spokes architecture.
    • Almost, all of the products are designed keeping this as reference.

    With this, all other questions should be easy to answer.

    2.Azure Route Server in 2 Peered VNETs

    • See : FAQ - Does Azure Route Server support virtual network peering?
    • You cannot enable "Use the remote virtual network's gateway or Route Server" in a VNET which has it's own Route Server.
    • A VNET can use only one Route Server (either it's own or peered VNET's)
    • This means, having BGP routes exchanged between WUSFG1-FGT and SCUSFG1-FGT is the correct approach (provided they are in 2 different VNET Hubs)

    With your requirement being OSPF as Routing Protocol, I think you should rely on third party NVAs only (such as the one you have currently)

    From @Silvia Wibowo 's answer, I see you have an interest with ARS and it's availability.

    • ARS Supports Availability Zones.
    • Certain Azure Regions are divided further into Availability Zones.
      User's image
    • Here, Region1 and Region2 have AZ but Region3 and Region4 don't.
    • In case of a zonal failure, your ARS will still be available provided the region support AZ.
    • Resources like ARS are what we call a "Zone-redundant" service.

    Zone-redundant resources are spread across multiple availability zones. Microsoft manages spreading requests across zones and the replication of data across zones. If an outage occurs in a single availability zone, Microsoft manages failover automatically.

    More details can be found in : What are availability zones?

    Kindly let us know if this helps or you need further assistance on this issue.

    Thanks,

    Kapil

    Please don’t forget to close the thread by clicking "Accept the answer" wherever the information provided helps you, as this can be beneficial to other community members.

    0 comments