UDR challange in ExpressRoute and Virtual Appliance Environment

Rzure99 1

Hi,

I have some challenges with setting up network routing in Azure...

We had setup where we had VPN to on-premises and Firewall as virtual appliance in Azure. We used User-Defined Routes where we defined that all internet traffic should go thru the firewall, in other words, the default route was to internal interface of the firewall. All traffic between on-premise network and azure was routed directory thru VPN with UDRs. This worked perfectly, until...

We got an ExpressRoute. The same setup but it doesnt work anymore. All connections will work if we drop UDRs from Azure subnets, but then we have a problem that direct connections from subnets to Internet is possible. We want to direct all internet traffic thru virtual appliance. It seems that using routes with next hop Virtual Network Gateway in ExpressRoute setup is not the same as in VPN setup.

Anyone else has had issues with UDRs and Express route or is it just me?

GitaraniSharma-MSFT 49,651 Reputation points Microsoft Employee

2020-10-09T11:41:00.96+00:00

Hello @Rzure99 ,

Could you please clarify your question and requirement again? In your question, the UDR setup is not clear. You mentioned, in your previous setup the default route was to internal interface of the firewall and in the current setup you have mentioned that using routes with next hop Virtual Network Gateway in ExpressRoute setup is not the same as in VPN setup. What is the exact requirement in your ExpressRoute setup? Is to force tunnel all Azure traffic (including Internet) to your on-premises or force tunnel all VPN traffic to your NVA/Firewall in Azure?

You cannot specify a virtual network gateway created as type ExpressRoute in a user-defined route because with ExpressRoute, you must use BGP for custom routes. You can use user-defined routes for forcing traffic from the Express Route to, for example, a Network Virtual Appliance.

Please refer : https://learn.microsoft.com/en-us/azure/virtual-network/virtual-networks-udr-overview

Thanks,
Gita
Rzure99 1 Reputation point

2020-10-09T13:03:29.68+00:00

Hi, thank you for your comments.

Our requirements in the new setup remains the same as in the old one. So, we want to route all traffic to internet from Azure thru Firewall, which is also in Azure, and all communications to internal address spaces routed directly to ExpressRoute.

One example from our setup is from subnet X. This worked with VPN, but apparently not with Express route

CIDR next hop
0.0.0.0/0 192.168.1.1 (IP of internal interface of Firewall)
10.0.0.0/8 Virtual Network Gateway

From your comments I understand that UDR wont work in our case and we should use BGP for custom routes.
Abraham Colon 1 Reputation point

2020-10-11T15:36:37.767+00:00

I have the same problem.

I want to forward on-prem traffic coming through an express route circuit to servers located in a vnet behind a virtual appliance (Azure firewall).

I configured a User define routes (UDR, Routing Table) in a gateway-subnet were in have a virtual gateway (gateway type ExpressRoute) attach. The UDR has the destination CIRD for the servers using the virtual appliance IP as the next hub.

The on-prem's servers can not communicate with the servers behind the virtual appliance. We configure an allow any to any policy at the virtual appliance for testing connectivity.

As a test, we change the virtual gateway configuration to a VPN type and configure a site-to-site VPN using BGP (similar configuration as the Express Route) and it works using the same UDR.

iI believe that the ExpressRoute Gateways does not support or use UDRs configured in their subnets.

I hope that someone can clarify or confirm this issue.

Thnks Abraham
GitaraniSharma-MSFT 49,651 Reputation points Microsoft Employee

2020-10-12T14:21:36.227+00:00

Hello @Abraham Colon ,

User-defined routes with a 0.0.0.0/0 destination on the GatewaySubnet are not supported.
Also, you cannot create user-defined routes to force traffic to the ExpressRoute virtual network gateway if you deploy a virtual network gateway deployed as type: ExpressRoute. But you can use user-defined routes for forcing traffic from the Express Route to, for example, a Network Virtual Appliance.

NOTE : Propagate gateway routes should be set to "Enabled" on the GatewaySubnet to ensure availability of the gateway and to propagate your on-premises routes to the network interfaces in the subnet.

Thanks,
Gita
GitaraniSharma-MSFT 49,651 Reputation points Microsoft Employee

2020-10-12T14:22:34.403+00:00

Hello @Rzure99 ,

Just checking in to see if the below answer helped. If this answers your query, please don’t forget to click "Accept the answer" and Up-Vote for the same, which might be beneficial to other community members reading this thread. And, if you have any further query do let us know.

Thanks,
Gita
GitaraniSharma-MSFT 49,651 Reputation points Microsoft Employee

2020-10-14T12:19:52.63+00:00

Hello @Rzure99 ,

Could you please provide an update on this post?

Kindly let us know if the below answer helps or you need further assistance on this issue.

----------------------------------------------------------------------------------------------------------------

Please don’t forget to close the thread by clicking "Accept the answer" wherever the information provided helps you, as this can be beneficial to other community members.
Abraham Colon 1 Reputation point

2020-10-14T12:57:00.95+00:00

Gita Thank for your quick response.

But precisely user-defined routes for forcing traffic from the Express Route to Network Virtual Appliance it is what is not working.
The scenario is the following

On-prem Net-Express Route Circ.-ER Virt.Gateway-Gateway Subnet-Gateways Vnet-Vnet peering-Virtual Appliance Vnet-Vnet peering-Servers Vnet (Address Space 172.30.12.0/24)- Servers Subnet (172.30.12.0/27).

The configured UDR at the Gateway Subnet is:
Ip prefix 172.30.12.0/24
Next Hub Type: Virtual Appliance
Next Hub IP: 172.30.3.4

The configured UDR at the Servers Vnet is:
Ip prefix 0.0.0.0/0
Next Hub Type: Virtual Appliance
Next Hub IP: 172.30.3.4
Propagate gateway routes: NO

Form an on-prem server I can ping the virtual appliance, but I cannot ping a server in the servers vnet.
From a server, in the servers vnet I can ping the virtual appliance, but I cannot ping an on-prem server.
Virtual Appliance Firewall policies are open for testing. No restrictions.
GitaraniSharma-MSFT 49,651 Reputation points Microsoft Employee

2020-10-14T14:57:44.71+00:00

Hello @Abraham Colon ,

Why have you set Propagate gateway routes as: NO on your server subnets?

I am not sure if you are using VPN with BGP or without BGP but in case of ExpressRoute, it always uses BGP (dynamic routing protocol) to exchange routes between your on-premises network and your instances in Azure. So, route propagation should not be disabled. If disabled, your on-premises routes will not be propagated to the network interfaces in the subnet and hence connectivity will be affected.

You should set Propagate gateway routes as: YES and check again.

Thanks,
Gita
GitaraniSharma-MSFT 49,651 Reputation points Microsoft Employee

2020-10-16T07:18:46.347+00:00

Hello @Rzure99 ,

Following up again to see if the below answer helped. If this answers your query, please don’t forget to click "Accept the answer" and Up-Vote for the same, which might be beneficial to other community members reading this thread. And, if you have any further query do let us know.

Thanks,
Gita

1 answer

GitaraniSharma-MSFT 49,651 Reputation points Microsoft Employee

2020-10-09T16:16:12.563+00:00
Hello @Rzure99 ,

As I mentioned before, you cannot specify a virtual network gateway created as type ExpressRoute in a user-defined route because with ExpressRoute, you must use BGP for custom routes.
Please refer : https://learn.microsoft.com/en-us/azure/virtual-network/virtual-networks-udr-overview

Hence, in order to achieve your current setup requirement, you will have to make use of both UDR and BGP custom routes in one of 2 following ways:

1)
UDR for Internet traffic 0.0.0.0/0 to next hop Firewall in Azure 192.168.1.1.
Advertise BGP custom route for 10.0.0.0/8.

2)
Advertise a route with the 0.0.0.0/0 prefix via BGP over ExpressRoute. (Default routes are permitted only on Azure private peering sessions.)
Add UDR on all subnets with default route 0.0.0.0/0 to next hop Firewall in Azure 192.168.1.1.

The 2nd option relies on how Azure selects a route:
If multiple routes contain the same address prefix, Azure selects the route type, based on the following priority:

User-defined route

BGP route

System route

NOTE: UDR always takes precedence.

Please refer : https://learn.microsoft.com/en-us/azure/expressroute/expressroute-routing#advertising-default-routes
https://learn.microsoft.com/en-us/azure/virtual-network/virtual-networks-udr-overview#how-azure-selects-a-route

Hope this helps!

Kindly let us know if the above helps or you need further assistance on this issue.

----------------------------------------------------------------------------------------------------------------

Please "Accept the answer" if the information helped you. This will help us and others in the community as well.
Please sign in to rate this answer.

1 person found this answer helpful.

0 comments No comments
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

UDR challange in ExpressRoute and Virtual Appliance Environment

1 answer

Your answer