Routing Issues with S2S VPN VNET Peered with ExpressRoute VNET
The Context:
I have 3 VNETS (VNET1, VNET2, VNET3). VNET1 has a S2S VPN allowing on-prem devices to connect to Azure. VNET2 has an ExpressRoute allowing another subnet of on-prem devices to connect to Azure. VNET3 also has an ExpressRoute allowing another subnet of on-prem devices to connect to Azure.
VNET1 and VNET2 are currently peered so that infrastructure in VNET2 can talk to the on-prem devices connected via VNET1. Currently there exists an Azure Firewall (oldFirewall) with a * rule on VNET1 to allow on-prem devices in VNET1 access to public internet. VNET3 is not yet peered with VNET2 but that is expected soon.
The Desired Outcome:
I want to delete the oldFirewall and create a newFirewall in VNET2 so that all traffic from on-prem passes through the newFirewall first and then the infrastructure VMs, following a hub-and-spoke model.
The Problem:
I am having trouble with the Routing Tables. How should the routing be configured so that all on-prem traffic routes to the newFirewall first?
@GitaraniSharma-MSFT , I tried following some steps here: https://learn.microsoft.com/en-us/answers/questions/860533/express-route-and-azure-firewall but I wasn't able to ping VNET1 on-prem devices from VNET2 VM after the routing change.
Azure Virtual Machines
Azure VPN Gateway
Azure Firewall
Azure Virtual Network
Azure ExpressRoute
-
KarishmaTiwari-MSFT 18,852 Reputation points • Microsoft Employee
2024-07-02T19:38:59.99+00:00 @RahulRana-1085 Thanks for posting your query on Microsoft Q&A.
Our team is looking into it and will get back as soon as possible. -
GitaraniSharma-MSFT 49,356 Reputation points • Microsoft Employee
2024-07-03T07:44:05.86+00:00 Hello @RahulRana-1085 ,
Welcome to Microsoft Q&A Platform. Thank you for reaching out & hope you are doing well.Looking at your setup, I would like to point out few things below:
VNET1(S2S VPN) and VNET2(ExpressRoute) are currently peered so that infrastructure in VNET2 can talk to the on-prem devices connected via VNET1.
The above is not possible. If you have ExpressRoute and site-to-site coexistence, transit routing isn't supported.
To enable transit routing between ExpressRoute and Azure VPN, you need to set up Azure Route Server. But the Azure VPN and ExpressRoute gateway must be deployed in the same virtual network as Route Server in order for BGP peering to be successfully established.
https://learn.microsoft.com/en-us/azure/route-server/expressroute-vpn-support#how-does-it-work
The hub and spoke model define the below configuration:
- Hub virtual network. The hub virtual network hosts shared Azure services. Workloads hosted in the spoke virtual networks can use these services. The hub virtual network is the central point of connectivity for cross-premises networks.
- Spoke virtual networks. Spoke virtual networks isolate and manage workloads separately in each spoke. Each workload can include multiple tiers, with multiple subnets connected through Azure load balancers. Spokes can exist in different subscriptions and represent different environments, such as Production and Non-production.
Refer: https://learn.microsoft.com/en-us/azure/architecture/networking/architecture/hub-spoke?tabs=cli
But in your case, all 3 Vnets are hub Vnets.
It is recommended to connect all networks to the same ExpressRoute circuit by using any-to-any ExpressRoute connectivity model and linking all Vnets to the same ExpressRoute circuit.
Or use ExpressRoute and Site-to-Site coexisting connections in the same Vnet to connect to different on-premises sites as below:
So, first we need to re-align the existing setup and requirement to proceed further to Azure Firewall configuration.
Regards,
Gita
-
RahulRana-1085 10 Reputation points
2024-07-03T13:08:53.9333333+00:00 Thank you for your response @GitaraniSharma-MSFT !
Sorry I wasn't clear in what I meant by:
VNET1(S2S VPN) and VNET2(ExpressRoute) are currently peered so that infrastructure in VNET2 can talk to the on-prem devices connected via VNET1.
In VNET2, I have Azure VMs that allows me to manage/SSH into the on-prem devices. The S2S VPN in VNET1 is using one cellular provider and the ExpressRoute in VNET2 is using another. The on-prem devices cannot talk to each other.
Flow for S2S:
On-Prem Device --> Cellular Provider --> S2S VPN --> Azure VNET1 (and public internet)
Flow for VNET2 ExpressRoute:
On-Prem Device --> Cellular Provider --> ExpressRoute --> Azure VNET2 (using private peering)
Now since VNET1 and VNET2 are peered, I can do this:
Azure VM in VNET2 --> (ssh) --> On-Prem Device connected via S2S VPN
So in essence, regardless of the connectivity source, all traffic from on-prem devices should ideally pass through the Azure Firewall in VNET2 first before hitting any VM subnets or public internet.
-
GitaraniSharma-MSFT 49,356 Reputation points • Microsoft Employee
2024-07-03T14:19:34.2566667+00:00 Hello @RahulRana-1085 ,
Now since VNET1 and VNET2 are peered, I can do this: Azure VM in VNET2 --> (ssh) --> On-Prem Device connected via S2S VPN
The above can only be done using gateway transit option, when only one of the peered Vnet has its own gateway.
You can configure the gateway in the peered virtual network as a transit point to an on-premises network. In this case, the virtual network that is using a remote gateway can't have its own gateway. A virtual network could have only one gateway, the gateway should be either local or remote gateway in the peered virtual network.
If both the Vnets have Virtual network gateways, then you cannot enable gateway transit option and cannot access on-premises networks connected to the peered Vnet. This is because of a virtual network peering limitation.
In your case Vnet1 and Vnet2 both have Virtual network gateways; hence the remote gateway/gateway transit option will not be available to configure in the Vnet peering and you cannot access on-premises resources from the peered Vnet.
If your setup is different from the above, please provide details on the gateways configured in the Vnets.
Regards,
Gita
-
GitaraniSharma-MSFT 49,356 Reputation points • Microsoft Employee
2024-07-05T12:28:35.2966667+00:00 @RahulRana-1085 , could you please provide an update on this post?
-
RahulRana-1085 10 Reputation points
2024-07-05T14:57:14.2366667+00:00 Here is a flow log I've pulled from my VNET2. This describes the flow from a on-prem device connected via cellular with backbone into Azure via S2S VPN. The destination is a VM in VNET2 and the source is the on-prem device from VNET1
FlowType
S2S
SrcIp
on-prem device IP address connected via S2S in VNET1
DestIp
private vm IP address in VNET2
TargetResourceId
VNET2
TargetResourceType
VirtualNetwork
SrcLocalNetworkGateway
VNET1 local network gateway
ConnectionType
VpnGateway
ConnectionName
VNET1 connection
ConnectingVnets
VNET1
AclGroup
VNET 2 private VM NSG
AclRule
allow on-prem devices from VNET1
The Virtual network gateway type in VNET1 is VPN (route-based) and the virtual network gateway type in VNET2 is ExpressRoute.
-
GitaraniSharma-MSFT 49,356 Reputation points • Microsoft Employee
2024-07-08T12:39:47.1566667+00:00 Hello @RahulRana-1085 ,
Thank you for the update.
I understand your setup and requirements, but I want to emphasize the limitations of Azure VNet peering.
You mentioned
VNET1(S2S VPN) and VNET2(ExpressRoute) are currently peered so that infrastructure in VNET2 can talk to the on-prem devices connected via VNET1
The above is only possible when you configure the gateway in the peered virtual network as a transit point to an on-premises network.
Gateway transit allows you to share an ExpressRoute or VPN gateway with all peered virtual networks and lets you manage the connectivity in one place. The virtual network that is using a remote gateway can't have its own gateway. A virtual network could have only one gateway, the gateway should be either local or remote gateway in the peered virtual network.
Could you please share the Vnet peering configuration between Vnet1 and Vnet2?
Regards,
Gita
-
RahulRana-1085 10 Reputation points
2024-07-09T18:24:14.2133333+00:00 VNET 1 Peering Settings
Allow 'VNET1' to access 'VNET2'✔
Allow 'VNET1' to receive forwarded traffic from 'VNET2' ✔
VNET 2 Peering Settings
Allow 'VNET2' to access 'VNET1'✔
Allow 'VNET2' to receive forwarded traffic from 'VNET1' ✔
VNET 1 Old Firewall
Rule1: Source *, Destination *
Rule2: Source VNET1 on-prem IP address range, Destination VNET2 private IP address range
Preexisting Route Table associated with GatewaySubnet of VNET1
Route: Destination: VNET2 private IP address range, Next hop type: old Firewall private IP
Preexisting Route Table associated with old AzureFirewallSubnet of VNET1
Route: Destination: S2S connected on-prem device IP range, Next hop type: Virtual network gateway
Route: Destination: 0.0.0.0/0, Next hop type: Internet
Preexisting Route Table associated with subnet of VNET2 VMs
Route: Destination: S2S connected on-prem device IP range, Next hop type: old Firewall private IP
The idea is to decommission the old firewall in VNET1 and setup a new one in VNET2 filtering all incoming traffic from either S2S or ExpressRoute.
If there is a limitation in communication via VNET Peering, I believe the infras is communicating through the old Azure firewall. Doing a traceroute from a VNET2 Azure VM to an S2S connected on-prem device has only a couple hops, the first hop being an IP address in the range of the AzureFirewallSubnet. Guessing the flow is: VNET2 Azure VM --> VNET1 Azure Firewall --> S2S connected on-prem device
There are more pieces to the architecture but I hope this provides some clarity.
-
RahulRana-1085 10 Reputation points
2024-07-09T19:11:58.2233333+00:00 VNET 1 Peering Settings
Allow 'VNET1' to access 'VNET2'✔
Allow 'VNET1' to receive forwarded traffic from 'VNET2' ✔
VNET 2 Peering Settings
Allow 'VNET2' to access 'VNET1'✔
Allow 'VNET2' to receive forwarded traffic from 'VNET1' ✔
VNET 1 Old Firewall
Rule1: Source *, Destination *
Rule2: Source VNET1 on-prem IP address range, Destination VNET2 private IP address range
Preexisting Route Table associated with GatewaySubnet of VNET1
Route: Destination: VNET2 private IP address range, Next hop type: old Firewall private IP
Preexisting Route Table associated with old AzureFirewallSubnet of VNET1
Route: Destination: S2S connected on-prem device IP range, Next hop type: Virtual network gateway
Route: Destination: 0.0.0.0/0, Next hop type: Internet
Preexisting Route Table associated with subnet of VNET2 VMs
Route: Destination: S2S connected on-prem device IP range, Next hop type: old Firewall private IP
The idea is to decomission the old firewall in VNET1 and setup a new one in VNET2 filtering all incoming traffic from either S2S or ExpressRoute.
If there is a limitation in communication via VNET Peering, I believe the infras is communicating through the old Azure firewall. Doing a traceroute from a VNET2 Azure VM to an S2S connected on-prem device has only a couple hops, the first hop being an IP address in the range of the AzureFirewallSubnet. Guessing the flow is: VNET2 Azure VM --> VNET1 Azure Firewall --> S2S connected on-prem device
There are more pieces to the architecture but I hope this provides some clarity.
-
KapilAnanth-MSFT 40,256 Reputation points • Microsoft Employee
2024-07-11T06:09:52.5433333+00:00 Hello @RahulRana-1085 ,
From your verbatim, I see
- VNET1 is connected to OnPrem1 via S2S
- This VNET1 has a firewall, say OldFirewall
- VNET2 is connected to OnPrem2 via ExR
- You do not use Gateway Transit and instead, Connectivity between OnPrem1 to VNET2 is provided by routing traffic to the OldFirewall
- Essentially, as you said,
- VNET2 Azure VM <--> VNET1 Azure Firewall <--> S2S connected OnPrem1
Yes, this configuration would work but note that this is not a typical Hub Spoke.
- We suggest all the Gateways be in a single VNET and this VNET should be the Hub
- Your situation is something like a multi Hub design.
Observation:
- From the logs you shared, I don't see two things happening
- OnPrem1 <---> OnPrem2 flow
- VNET1 <---> OnPrem2 flow
- I think this is expected and doubt you will be able to achieve this with static routing.
- This makes sense as from OldFirewall in VNET1, there is no way of routing to OnPrem2 (which is connected to VNET2)
- i.e., The Firewall can only learn the OnPrem routes to which it's VNET is connected.
- That's why you were able to access "S2S connected on-prem device"
Now, per your requirement, "I want to delete the oldFirewall and create a newFirewall in VNET2 so that all traffic from on-prem passes through the newFirewall first and then the infrastructure VMs, following a hub-and-spoke model"
- You will able to achieve connectivity between different VNETs that are peered to the VNET2 via the NewFirewall.
- Also, OnPrem2 (connected to VNET2), can access all the peered VNETs via Azure Firewall
- However, I don't think OnPrem to OnPrem connectivity can be achieved in this manner
You can instead consider using Azure Virtual WAN for such complex scenarios which provided "Branch connectivity" across all connected sites.
Hope this clarifies.
Cheers,
Kapil
- VNET1 is connected to OnPrem1 via S2S
-
RahulRana-1085 10 Reputation points
2024-07-12T17:50:37.8033333+00:00 Thank you for the information. I think I have clarity now that the OldFirewall is necessary to manage VNET1 S2S connected on-prem devices through VNET2 Azure VMs. I will not go forward with creating a new Firewall in VNET2 and maintain the firewall in VNET1.
VNET3 is not yet peered with VNET2 but that is expected soon.
In VNET 3, I have on-prem connected via another Azure ExpressRoute. If I wanted to manage those on-prem devices from the same VNET2 VM, I'd need to create an Azure Firewall there similar to "OldFirewall" to route communication?
Sign in to comment