Routing Issues with S2S VPN VNET Peered with ExpressRoute VNET

RahulRana-1085 30

The Context:

I have 3 VNETS (VNET1, VNET2, VNET3). VNET1 has a S2S VPN allowing on-prem devices to connect to Azure. VNET2 has an ExpressRoute allowing another subnet of on-prem devices to connect to Azure. VNET3 also has an ExpressRoute allowing another subnet of on-prem devices to connect to Azure.

VNET1 and VNET2 are currently peered so that infrastructure in VNET2 can talk to the on-prem devices connected via VNET1. Currently there exists an Azure Firewall (oldFirewall) with a * rule on VNET1 to allow on-prem devices in VNET1 access to public internet. VNET3 is not yet peered with VNET2 but that is expected soon.

The Desired Outcome:

I want to delete the oldFirewall and create a newFirewall in VNET2 so that all traffic from on-prem passes through the newFirewall first and then the infrastructure VMs, following a hub-and-spoke model.

The Problem:

I am having trouble with the Routing Tables. How should the routing be configured so that all on-prem traffic routes to the newFirewall first?

@GitaraniSharma-MSFT , I tried following some steps here: https://learn.microsoft.com/en-us/answers/questions/860533/express-route-and-azure-firewall but I wasn't able to ping VNET1 on-prem devices from VNET2 VM after the routing change.

KarishmaTiwari-MSFT 20,207 Reputation points Microsoft Employee

2024-07-02T19:38:59.99+00:00

@RahulRana-1085 Thanks for posting your query on Microsoft Q&A.
Our team is looking into it and will get back as soon as possible.
GitaraniSharma-MSFT 49,586 Reputation points Microsoft Employee

2024-07-03T07:44:05.86+00:00
Hello @RahulRana-1085 ,

Welcome to Microsoft Q&A Platform. Thank you for reaching out & hope you are doing well.Looking at your setup, I would like to point out few things below:

VNET1(S2S VPN) and VNET2(ExpressRoute) are currently peered so that infrastructure in VNET2 can talk to the on-prem devices connected via VNET1.

The above is not possible. If you have ExpressRoute and site-to-site coexistence, transit routing isn't supported.

To enable transit routing between ExpressRoute and Azure VPN, you need to set up Azure Route Server. But the Azure VPN and ExpressRoute gateway must be deployed in the same virtual network as Route Server in order for BGP peering to be successfully established.

Refer: https://learn.microsoft.com/en-us/azure/vpn-gateway/vpn-gateway-vpn-faq#how-do-i-enable-routing-between-my-site-to-site-vpn-connection-and-my-expressroute

https://learn.microsoft.com/en-us/azure/expressroute/how-to-configure-coexisting-gateway-portal#to-enable-transit-routing-between-expressroute-and-azure-vpn

https://learn.microsoft.com/en-us/azure/virtual-network/virtual-network-configure-vnet-connections#configure-transit-routing-in-an-expressroute-connection

https://learn.microsoft.com/en-us/azure/route-server/expressroute-vpn-support#how-does-it-work

The hub and spoke model define the below configuration:

Hub virtual network. The hub virtual network hosts shared Azure services. Workloads hosted in the spoke virtual networks can use these services. The hub virtual network is the central point of connectivity for cross-premises networks.

Spoke virtual networks. Spoke virtual networks isolate and manage workloads separately in each spoke. Each workload can include multiple tiers, with multiple subnets connected through Azure load balancers. Spokes can exist in different subscriptions and represent different environments, such as Production and Non-production.

Refer: https://learn.microsoft.com/en-us/azure/architecture/networking/architecture/hub-spoke?tabs=cli

https://learn.microsoft.com/en-us/azure/cloud-adoption-framework/ready/azure-best-practices/hub-spoke-network-topology

But in your case, all 3 Vnets are hub Vnets.

It is recommended to connect all networks to the same ExpressRoute circuit by using any-to-any ExpressRoute connectivity model and linking all Vnets to the same ExpressRoute circuit.

Or use ExpressRoute and Site-to-Site coexisting connections in the same Vnet to connect to different on-premises sites as below:

https://learn.microsoft.com/en-us/azure/expressroute/how-to-configure-coexisting-gateway-portal#configure-a-site-to-site-vpn-to-connect-to-sites-not-connected-through-expressroute

So, first we need to re-align the existing setup and requirement to proceed further to Azure Firewall configuration.

Regards,

Gita
RahulRana-1085 30 Reputation points

2024-07-03T13:08:53.9333333+00:00

Thank you for your response @GitaraniSharma-MSFT !

Sorry I wasn't clear in what I meant by:

VNET1(S2S VPN) and VNET2(ExpressRoute) are currently peered so that infrastructure in VNET2 can talk to the on-prem devices connected via VNET1.

In VNET2, I have Azure VMs that allows me to manage/SSH into the on-prem devices. The S2S VPN in VNET1 is using one cellular provider and the ExpressRoute in VNET2 is using another. The on-prem devices cannot talk to each other.

Flow for S2S:

On-Prem Device --> Cellular Provider --> S2S VPN --> Azure VNET1 (and public internet)

Flow for VNET2 ExpressRoute:

On-Prem Device --> Cellular Provider --> ExpressRoute --> Azure VNET2 (using private peering)

Now since VNET1 and VNET2 are peered, I can do this:

Azure VM in VNET2 --> (ssh) --> On-Prem Device connected via S2S VPN

So in essence, regardless of the connectivity source, all traffic from on-prem devices should ideally pass through the Azure Firewall in VNET2 first before hitting any VM subnets or public internet.
GitaraniSharma-MSFT 49,586 Reputation points Microsoft Employee

2024-07-03T14:19:34.2566667+00:00

Hello @RahulRana-1085 ,

Now since VNET1 and VNET2 are peered, I can do this: Azure VM in VNET2 --> (ssh) --> On-Prem Device connected via S2S VPN

The above can only be done using gateway transit option, when only one of the peered Vnet has its own gateway.

You can configure the gateway in the peered virtual network as a transit point to an on-premises network. In this case, the virtual network that is using a remote gateway can't have its own gateway. A virtual network could have only one gateway, the gateway should be either local or remote gateway in the peered virtual network.

Refer: https://learn.microsoft.com/en-us/azure/virtual-network/virtual-network-peering-overview#gateways-and-on-premises-connectivity

https://learn.microsoft.com/en-us/azure/vpn-gateway/vpn-gateway-peering-gateway-transit?toc=%2Fazure%2Fvirtual-network%2Ftoc.json#to-add-a-peering-and-enable-transit

If both the Vnets have Virtual network gateways, then you cannot enable gateway transit option and cannot access on-premises networks connected to the peered Vnet. This is because of a virtual network peering limitation.

Refer: https://learn.microsoft.com/en-us/azure/virtual-network/virtual-network-troubleshoot-peering-issues#both-the-hub-virtual-network-and-the-spoke-virtual-network-have-a-vpn-gateway

In your case Vnet1 and Vnet2 both have Virtual network gateways; hence the remote gateway/gateway transit option will not be available to configure in the Vnet peering and you cannot access on-premises resources from the peered Vnet.

If your setup is different from the above, please provide details on the gateways configured in the Vnets.

Regards,

Gita
GitaraniSharma-MSFT 49,586 Reputation points Microsoft Employee

2024-07-05T12:28:35.2966667+00:00

@RahulRana-1085 , could you please provide an update on this post?
RahulRana-1085 30 Reputation points

2024-07-05T14:57:14.2366667+00:00

Hi @GitaraniSharma-MSFT

Here is a flow log I've pulled from my VNET2. This describes the flow from a on-prem device connected via cellular with backbone into Azure via S2S VPN. The destination is a VM in VNET2 and the source is the on-prem device from VNET1

FlowType

S2S

SrcIp

on-prem device IP address connected via S2S in VNET1

DestIp

private vm IP address in VNET2

TargetResourceId

VNET2

TargetResourceType

VirtualNetwork

SrcLocalNetworkGateway

VNET1 local network gateway

ConnectionType

VpnGateway

ConnectionName

VNET1 connection

ConnectingVnets

VNET1

AclGroup

VNET 2 private VM NSG

AclRule

allow on-prem devices from VNET1

The Virtual network gateway type in VNET1 is VPN (route-based) and the virtual network gateway type in VNET2 is ExpressRoute.
GitaraniSharma-MSFT 49,586 Reputation points Microsoft Employee

2024-07-08T12:39:47.1566667+00:00

Hello @RahulRana-1085 ,

Thank you for the update.

I understand your setup and requirements, but I want to emphasize the limitations of Azure VNet peering.

You mentioned VNET1(S2S VPN) and VNET2(ExpressRoute) are currently peered so that infrastructure in VNET2 can talk to the on-prem devices connected via VNET1

The above is only possible when you configure the gateway in the peered virtual network as a transit point to an on-premises network.

Gateway transit allows you to share an ExpressRoute or VPN gateway with all peered virtual networks and lets you manage the connectivity in one place. The virtual network that is using a remote gateway can't have its own gateway. A virtual network could have only one gateway, the gateway should be either local or remote gateway in the peered virtual network.

Refer: https://learn.microsoft.com/en-us/azure/virtual-network/virtual-network-peering-overview#gateways-and-on-premises-connectivity

https://learn.microsoft.com/en-us/azure/virtual-network/virtual-network-manage-peering?tabs=peering-portal#create-a-peering

https://learn.microsoft.com/en-us/azure/vpn-gateway/vpn-gateway-peering-gateway-transit#to-add-a-peering-and-enable-transit

Could you please share the Vnet peering configuration between Vnet1 and Vnet2?

Regards,

Gita
RahulRana-1085 30 Reputation points

2024-07-09T18:24:14.2133333+00:00

Hi @GitaraniSharma-MSFT

VNET 1 Peering Settings

Allow 'VNET1' to access 'VNET2'✔

Allow 'VNET1' to receive forwarded traffic from 'VNET2' ✔

VNET 2 Peering Settings

Allow 'VNET2' to access 'VNET1'✔

Allow 'VNET2' to receive forwarded traffic from 'VNET1' ✔

VNET 1 Old Firewall

Rule1: Source *, Destination *

Rule2: Source VNET1 on-prem IP address range, Destination VNET2 private IP address range

Preexisting Route Table associated with GatewaySubnet of VNET1

Route: Destination: VNET2 private IP address range, Next hop type: old Firewall private IP

Preexisting Route Table associated with old AzureFirewallSubnet of VNET1

Route: Destination: S2S connected on-prem device IP range, Next hop type: Virtual network gateway

Route: Destination: 0.0.0.0/0, Next hop type: Internet

Preexisting Route Table associated with subnet of VNET2 VMs

Route: Destination: S2S connected on-prem device IP range, Next hop type: old Firewall private IP

The idea is to decommission the old firewall in VNET1 and setup a new one in VNET2 filtering all incoming traffic from either S2S or ExpressRoute.

If there is a limitation in communication via VNET Peering, I believe the infras is communicating through the old Azure firewall. Doing a traceroute from a VNET2 Azure VM to an S2S connected on-prem device has only a couple hops, the first hop being an IP address in the range of the AzureFirewallSubnet. Guessing the flow is: VNET2 Azure VM --> VNET1 Azure Firewall --> S2S connected on-prem device

There are more pieces to the architecture but I hope this provides some clarity.
RahulRana-1085 30 Reputation points

2024-07-09T19:11:58.2233333+00:00

Hi @GitaraniSharma-MSFT

VNET 1 Peering Settings

Allow 'VNET1' to access 'VNET2'✔

Allow 'VNET1' to receive forwarded traffic from 'VNET2' ✔

VNET 2 Peering Settings

Allow 'VNET2' to access 'VNET1'✔

Allow 'VNET2' to receive forwarded traffic from 'VNET1' ✔

VNET 1 Old Firewall

Rule1: Source *, Destination *

Rule2: Source VNET1 on-prem IP address range, Destination VNET2 private IP address range

Preexisting Route Table associated with GatewaySubnet of VNET1

Route: Destination: VNET2 private IP address range, Next hop type: old Firewall private IP

Preexisting Route Table associated with old AzureFirewallSubnet of VNET1

Route: Destination: S2S connected on-prem device IP range, Next hop type: Virtual network gateway

Route: Destination: 0.0.0.0/0, Next hop type: Internet

Preexisting Route Table associated with subnet of VNET2 VMs

Route: Destination: S2S connected on-prem device IP range, Next hop type: old Firewall private IP

The idea is to decomission the old firewall in VNET1 and setup a new one in VNET2 filtering all incoming traffic from either S2S or ExpressRoute.

If there is a limitation in communication via VNET Peering, I believe the infras is communicating through the old Azure firewall. Doing a traceroute from a VNET2 Azure VM to an S2S connected on-prem device has only a couple hops, the first hop being an IP address in the range of the AzureFirewallSubnet. Guessing the flow is: VNET2 Azure VM --> VNET1 Azure Firewall --> S2S connected on-prem device

There are more pieces to the architecture but I hope this provides some clarity.
Udayashankar K.N 80 Reputation points Microsoft Employee

2024-07-15T04:55:09.22+00:00

Transit is not possible in your situation as per my understanding

Accepted answer

KapilAnanth-MSFT 47,046 Reputation points Microsoft Employee

2024-07-11T06:09:52.5433333+00:00
Hello @RahulRana-1085 ,

From your verbatim, I see

VNET1 is connected to OnPrem1 via S2S

This VNET1 has a firewall, say OldFirewall

VNET2 is connected to OnPrem2 via ExR

You do not use Gateway Transit and instead, Connectivity between OnPrem1 to VNET2 is provided by routing traffic to the OldFirewall

Essentially, as you said,

VNET2 Azure VM <--> VNET1 Azure Firewall <--> S2S connected OnPrem1

Yes, this configuration would work but note that this is not a typical Hub Spoke.

We suggest all the Gateways be in a single VNET and this VNET should be the Hub

Your situation is something like a multi Hub design.

Observation:

From the logs you shared, I don't see two things happening

OnPrem1 <---> OnPrem2 flow

VNET1 <---> OnPrem2 flow

I think this is expected and doubt you will be able to achieve this with static routing.

This makes sense as from OldFirewall in VNET1, there is no way of routing to OnPrem2 (which is connected to VNET2)

i.e., The Firewall can only learn the OnPrem routes to which it's VNET is connected.

That's why you were able to access "S2S connected on-prem device"

Now, per your requirement, "I want to delete the oldFirewall and create a newFirewall in VNET2 so that all traffic from on-prem passes through the newFirewall first and then the infrastructure VMs, following a hub-and-spoke model"

You will able to achieve connectivity between different VNETs that are peered to the VNET2 via the NewFirewall.

Also, OnPrem2 (connected to VNET2), can access all the peered VNETs via Azure Firewall

However, I don't think OnPrem to OnPrem connectivity can be achieved in this manner

You can instead consider using Azure Virtual WAN for such complex scenarios which provided "Branch connectivity" across all connected sites.

Hope this clarifies.

Cheers,

Kapil
Please sign in to rate this answer.

1 person found this answer helpful.
RahulRana-1085 30 Reputation points

2024-07-12T17:50:37.8033333+00:00

Thank you for the information. I think I have clarity now that the OldFirewall is necessary to manage VNET1 S2S connected on-prem devices through VNET2 Azure VMs. I will not go forward with creating a new Firewall in VNET2 and maintain the firewall in VNET1.

VNET3 is not yet peered with VNET2 but that is expected soon.

In VNET 3, I have on-prem connected via another Azure ExpressRoute. If I wanted to manage those on-prem devices from the same VNET2 VM, I'd need to create an Azure Firewall there similar to "OldFirewall" to route communication?

KapilAnanth-MSFT 47,046 Reputation points Microsoft Employee

2024-07-15T07:20:56.98+00:00

Hello @RahulRana-1085 ,

For naming convention,

VNET3 is connected to OnPrem3 via ExR

VNET2 is peered to VNET3 and VNET2 has a Gateway of it's own.

You are deploying a Firewall in VNET3, call it Firewall3

Now,

VNET2 <---> OnPrem3 communication should be possible with a NVA/Firewall in the VNET3.

Let's say you have a VM in VNET2

You attach a UDR and point OnPrem3's address range traffic to Firewall3

From the VNET3's GatewaySubnet,

Attach a UDR and point the VNET2's address range to the Firewall3

This should enable to and fro communication between VNET2 and OnPrem3

Kindly let us know if this helps or you need further assistance on this issue.

Thanks,

Kapil

Please don’t forget to close the thread by clicking "Accept the answer" wherever the information provided helps you, as this can be beneficial to other community members.

RahulRana-1085 30 Reputation points

2024-09-26T01:22:03.03+00:00

Hi @KapilAnanth-MSFT , I am trying to configure this setup:

VNET3 is connected to OnPrem3 via ExR

VNET2 is peered to VNET3 and VNET2 has a Gateway of it's own.

You are deploying a Firewall in VNET3, call it Firewall3

I am not able to get the address space of VNET2 to show up on the private peering route table of VNET3. I have the appropriate UDRs setup but I cannot communicate from VNET2 VM to OnPrem3 device or vice versa.

Route Tables:

VNET3:

RouteTable1 (associated to AzureFirewallSubnet)

0.0.0.0/0 -> Internet

ONPREM3_IP_ADDRESS_SPACE -> Virtual network gateway

RouteTable2 (associated to GatewaySubnet)

IP_ADDRESS_SPACE_OF_VNET2 -> Firewall3

VNET2:

RouteTable3 (associated to VM subnet)

IP_ADDRESS_SPACE_OF_VNET3 -> Firewall3

ONPREM3_IP_ADDRESS_SPACE -> Firewall3

KapilAnanth-MSFT 47,046 Reputation points Microsoft Employee

2024-09-26T05:12:35.4233333+00:00

Hello @RahulRana-1085 ,

Wrt, "I am not able to get the address space of VNET2 to show up on the private peering route table of VNET3."

This is expected.

BGP will not advertise the VNET2's address space

This is the reason we are creating the manual routes in the first place.

You should "force" the OnPrem to route VNET2's range into the ExpressRoute

Once VNET2's traffic reaches the Gateway in Azure, UDRs will take care of the to and fro traffic from Gateway to the VNET2 and vice versa

Cheers,

Kapil

RahulRana-1085 30 Reputation points

2024-09-26T14:07:47.9166667+00:00

Could you elaborate on this:

You should "force" the OnPrem to route VNET2's range into the ExpressRoute

Is this something I can do on the Azure portal or do I need to do this via the ExpressRoute provider?

KapilAnanth-MSFT 47,046 Reputation points Microsoft Employee

2024-09-29T13:37:23.43+00:00

Hello @RahulRana-1085 ,

This should be done from the OnPrem end.

See : ExpressRoute FAQ
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Routing Issues with S2S VPN VNET Peered with ExpressRoute VNET

0 additional answers

Your answer