Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
Summary
This article helps you verify and troubleshoot Azure ExpressRoute connectivity. ExpressRoute extends an on-premises network into the Microsoft Cloud over a private connection facilitated by a connectivity provider. ExpressRoute connectivity involves three distinct network zones:
- Customer network
- Provider network
- Microsoft datacenter
Note
In the ExpressRoute Direct connectivity model, you can directly connect to the port for Microsoft Enterprise Edge (MSEE) routers. This model includes only your network and the Microsoft network zones.
This article helps you identify connectivity issues and seek support from the appropriate team to resolve them.
Important
This article helps you diagnose and fix simple issues. It's not a replacement for Microsoft support. If you can't solve a problem by using this guidance, open a support ticket with Microsoft Support.
Overview
The following diagram shows the logical connectivity of a customer network to the Microsoft network through ExpressRoute.
In the diagram, the numbers indicate key network points:
- Customer compute device (for example, a server or PC).
- Customer edge routers (CEs).
- Provider edge routers and switches (PEs) facing customer edge routers.
- PEs facing Microsoft Enterprise Edge ExpressRoute routers (MSEEs), called PE-MSEEs.
- MSEEs.
- Virtual network gateway.
- Compute device on the Azure virtual network.
This article references these network points by their associated number.
Depending on the ExpressRoute connectivity model, network points 3 and 4 might be switches (layer 2 devices) or routers (layer 3 devices). The connectivity models are cloud exchange colocation, point-to-point Ethernet connection, or any-to-any (IPVPN).
In the direct connectivity model, the architecture doesn't include network points 3 and 4. Instead, CEs (2) directly connect to MSEEs through dark fiber.
If you use the cloud exchange colocation, point-to-point Ethernet, or direct connectivity model, CEs (2) establish Border Gateway Protocol (BGP) peering with MSEEs (5).
If you use the any-to-any (IPVPN) connectivity model, PE-MSEEs (4) establish BGP peering with MSEEs (5). PE-MSEEs propagate the routes received from Microsoft back to the customer network through the IPVPN service provider network.
Note
For high availability, Microsoft establishes fully redundant parallel connectivity between MSEE and PE-MSEE pairs. You should also create a fully redundant parallel network path between the customer network and PE/CE pairs. For more information about high availability, see Designing for high availability with ExpressRoute.
The following sections represent the logical steps in troubleshooting an ExpressRoute circuit.
Verify circuit provisioning and state
Provisioning an ExpressRoute circuit establishes a redundant layer 2 connection between CEs/PE-MSEEs (2/4) and MSEEs (5). For more information about how to create, modify, provision, and verify an ExpressRoute circuit, see Create and modify an ExpressRoute circuit.
Tip
A service key uniquely identifies an ExpressRoute circuit. If you need assistance from Microsoft or an ExpressRoute partner to troubleshoot an issue, provide the service key to readily identify the circuit.
Verification by using the Azure portal
In the Azure portal, go to the page for your ExpressRoute circuit. The Overview section lists the ExpressRoute essentials, as shown in the following screenshot:
In the ExpressRoute essentials, Circuit status shows the status of the circuit on the Microsoft side, and Provider status shows whether the service provider has provisioned the circuit.
For an ExpressRoute circuit to be operational, Circuit status must be Enabled, and Provider status must be Provisioned.
Note
If Circuit status is stuck in Not enabled, contact Microsoft Support. If Provider status is stuck in Not provisioned, contact your service provider.
Verification by using Azure PowerShell
To list all ExpressRoute circuits in a resource group, use the following command.
Get-AzExpressRouteCircuit -ResourceGroupName "Test-ER-RG"
Tip
To find the name of a resource group, use the Get-AzResourceGroup command to list all resource groups in your subscription.
To get details of a specific ExpressRoute circuit in a resource group, use the following command.
Get-AzExpressRouteCircuit -ResourceGroupName "Test-ER-RG" -Name "Test-ER-Ckt"
The following example shows a response.
Name : Test-ER-Ckt
ResourceGroupName : Test-ER-RG
Location : westus2
Id : /subscriptions/***************************/resourceGroups/Test-ER-RG/providers/***********/expressRouteCircuits/Test-ER-Ckt
Etag : W/"################################"
ProvisioningState : Succeeded
Sku : {
"Name": "Standard_UnlimitedData",
"Tier": "Standard",
"Family": "UnlimitedData"
}
CircuitProvisioningState : Enabled
ServiceProviderProvisioningState : Provisioned
ServiceProviderNotes :
ServiceProviderProperties : {
"ServiceProviderName": "****",
"PeeringLocation": "******",
"BandwidthInMbps": 100
}
ServiceKey : **************************************
Peerings : []
Authorizations : []
To confirm that an ExpressRoute circuit is operational, ensure the following fields are set correctly.
CircuitProvisioningState : Enabled
ServiceProviderProvisioningState : Provisioned
Note
If Circuit status is stuck in Not enabled, contact Microsoft Support. If Provider status is stuck in Not provisioned, contact your service provider.
Validate peering configuration
After the service provider provisions the ExpressRoute circuit, you can create multiple routing configurations that use external BGP (eBGP) over the circuit between CEs/MSEE-PEs (2/4) and MSEEs (5). Each ExpressRoute circuit can have one or both of the following peering configurations:
- Azure private peering: for traffic to private virtual networks in Azure
- Microsoft peering: for traffic to public endpoints of platform as a service (PaaS) and software as a service (SaaS)
For more information about creating and modifying routing configurations, see Create and modify routing for an ExpressRoute circuit.
Verification by using the Azure portal
Note
In an IPVPN connectivity model, service providers handle the responsibility of configuring the peerings (layer 3 services). If the peering is blank in the portal after the service provider configures it, try refreshing the circuit configuration by using the refresh button on the portal. This operation pulls the current routing configuration from your circuit.
In the Azure portal, you can check the status of an ExpressRoute circuit on its page. The
section lists the ExpressRoute peerings, as shown in the following illustration.
In the preceding example, Azure private peering is provisioned, but Azure public and Microsoft peerings aren't. A successfully provisioned peering context also lists the primary and secondary point-to-point subnets. The /30 subnets are used for the interface IP addresses of the MSEEs and CEs/PE-MSEEs. The listing also indicates who last modified the configuration.
Note
If enabling a peering fails, check if the assigned primary and secondary subnets match the configuration on the linked CE or PE-MSEE, depending on your connectivity model. Also, verify that the VlanId, AzureASN, and PeerASN values on the MSEEs match those on the linked CE/PE-MSEE.
If you choose MD5 hashing, ensure the shared key is the same on both MSEE and CE/PE-MSEE pairs. For security reasons, previously configured shared keys aren't displayed.
To change any of these configurations on an MSEE router, see Create and modify routing for an ExpressRoute circuit.
Note
On a /30 subnet assigned for the interface, Microsoft uses the second usable IP address for the MSEE interface. Assign the first usable IP address to the CE or PE-MSEE that peers with the MSEE, depending on your connectivity model.
Verification by using Azure PowerShell
To get the configuration details for Azure private peering, use Azure PowerShell and run the following command.
$ckt = Get-AzExpressRouteCircuit -ResourceGroupName "Test-ER-RG" -Name "Test-ER-Ckt"
Get-AzExpressRouteCircuitPeeringConfig -Name "AzurePrivatePeering" -ExpressRouteCircuit $ckt
The following example shows a response for a successfully configured private peering.
Name : AzurePrivatePeering
Id : /subscriptions/***************************/resourceGroups/Test-ER-RG/providers/***********/expressRouteCircuits/Test-ER-Ckt/peerings/AzurePrivatePeering
Etag : W/"################################"
PeeringType : AzurePrivatePeering
AzureASN : 12076
PeerASN : 123##
PrimaryPeerAddressPrefix : 172.16.0.0/30
SecondaryPeerAddressPrefix : 172.16.0.4/30
PrimaryAzurePort :
SecondaryAzurePort :
SharedKey :
VlanId : 200
MicrosoftPeeringConfig : null
ProvisioningState : Succeeded
A successfully enabled peering context lists the primary and secondary address prefixes. The /30 subnets are used for the interface IP addresses of the MSEEs and CEs/PE-MSEEs.
To get the configuration details for Microsoft peering, use the following commands.
$ckt = Get-AzExpressRouteCircuit -ResourceGroupName "Test-ER-RG" -Name "Test-ER-Ckt"
Get-AzExpressRouteCircuitPeeringConfig -Name "MicrosoftPeering" -ExpressRouteCircuit $ckt
If a peering isn't configured, you get an error message. Here's an example response when the stated peering isn't configured within the circuit.
Get-AzExpressRouteCircuitPeeringConfig : Sequence contains no matching element
At line:1 char:1
+ Get-AzExpressRouteCircuitPeeringConfig -Name "MicrosoftPeering ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : CloseError: (:) [Get-AzExpr...itPeeringConfig], InvalidOperationException
+ FullyQualifiedErrorId : Microsoft.Azure.Commands.Network.GetAzureExpressRouteCircuitPeeringConfigCommand
Note
If enabling a peering fails, check if the assigned primary and secondary subnets match the configuration on the linked CE/PE-MSEE. Also, verify that the VlanId, AzureASN, and PeerASN values on the MSEEs match those on the linked CE/PE-MSEE.
If you choose MD5 hashing, ensure the shared key is the same on both MSEE and CE/PE-MSEE pairs. For security reasons, previously configured shared keys aren't displayed.
To change any of these configurations on an MSEE router, see Create and modify routing for an ExpressRoute circuit.
Note
On a /30 subnet assigned for the interface, Microsoft uses the second usable IP address for the MSEE interface. Ensure you assign the first usable IP address to the peered CE/PE-MSEE.
Validate ARP
The Address Resolution Protocol (ARP) table provides a mapping of the IP address and MAC address for a particular peering. The ARP table for an ExpressRoute circuit peering provides the following information for each interface (primary and secondary):
- Mapping of the IP address for the on-premises router interface to the MAC address
- Mapping of the IP address for the ExpressRoute router interface to the MAC address (optional)
- Age of the mapping
ARP tables can help validate layer 2 configuration and troubleshoot basic layer 2 connectivity issues.
Note
Depending on the hardware platform, the ARP results might vary and only display the On-premises interface.
For an ExpressRoute Direct circuit that uses QinQ encapsulation, use the following values when you validate the VLAN configuration:
| Value | Interpretation | Where to find it |
|---|---|---|
InterfaceProperty, IpAddress, MacAddress, and Age |
ARP output fields that show whether an on-premises IP address resolves to a MAC address. An incomplete on-premises MAC address or a result that contains only the Microsoft entry indicates a layer 2 problem to investigate. | Get-AzExpressRouteCircuitARPTable output |
VlanId |
C-Tag, the inner VLAN tag that identifies the peering. | Peering configuration in the Get-AzExpressRouteCircuit output |
Stag |
S-Tag, the outer VLAN tag assigned to the circuit. | Circuit properties in the Get-AzExpressRouteCircuit output |
The ARP output doesn't contain the C-Tag or S-Tag. Run the ARP command for both paths to identify an affected path:
Get-AzExpressRouteCircuitARPTable -ResourceGroupName <ResourceGroupName> -ExpressRouteCircuitName <CircuitName> -PeeringType AzurePrivatePeering -DevicePath Primary
Get-AzExpressRouteCircuitARPTable -ResourceGroupName <ResourceGroupName> -ExpressRouteCircuitName <CircuitName> -PeeringType AzurePrivatePeering -DevicePath Secondary
If either result shows an incomplete on-premises MAC address or only the Microsoft entry, compare the peering VlanId (C-Tag) and circuit Stag (S-Tag) with the inner and outer VLAN tags configured across the QinQ handoff. To retrieve these properties and complete the validation, see Diagnose VLAN C-Tag mismatches.
Validate BGP and routes on the MSEE
To retrieve the routing table from the MSEE on the primary path for the private routing context, use the following command:
Get-AzExpressRouteCircuitRouteTable -DevicePath Primary -ExpressRouteCircuitName <CircuitName> -PeeringType AzurePrivatePeering -ResourceGroupName <ResourceGroupName>
Example response:
Network : 10.1.0.0/16
NextHop : 10.17.17.141
LocPrf :
Weight : 0
Path : 65515
Network : 10.1.0.0/16
NextHop : 10.17.17.140*
LocPrf :
Weight : 0
Path : 65515
Network : 10.2.20.0/25
NextHop : 172.16.0.1
LocPrf :
Weight : 0
Path : 123##
Confirm that the output contains each on-premises prefix that you expect to reach through private peering. If an expected prefix is missing, verify that your on-premises routers are advertising the prefix and that outbound route policies aren't suppressing it. Re-enable or adjust the advertisement before you continue with traffic-flow checks.
Note
If the eBGP peering state between an MSEE and a CE/PE-MSEE is Active or Idle, verify that the primary and secondary peer subnets match the configuration on the linked CE/PE-MSEE. Ensure the VlanId, AzureASN, and PeerASN values are correct on the MSEEs and match those values on the linked CE/PE-MSEE. If you use MD5 hashing, the shared key must be the same on both MSEE and CE/PE-MSEE pairs. For configuration changes on an MSEE router, see Create and modify routing for an ExpressRoute circuit.
Note
If certain destinations are unreachable over a peering, check the MSEE route table for the corresponding peering context. If a matching prefix is present, ensure no firewalls, network security groups, or access control lists (ACLs) are blocking the traffic.
The following is an example response for a nonexistent peering.
Get-AzExpressRouteCircuitRouteTable : The BGP Peering AzurePublicPeering with Service Key <ServiceKey> is not found.
StatusCode: 400
Confirm the traffic flow
To get traffic statistics (bytes in and out) for a peering context, use Azure PowerShell to run the following command.
Get-AzExpressRouteCircuitStats -ResourceGroupName <ResourceGroupName> -ExpressRouteCircuitName <CircuitName> -PeeringType 'AzurePrivatePeering'
The following is example output.
PrimaryBytesIn PrimaryBytesOut SecondaryBytesIn SecondaryBytesOut
-------------- --------------- ---------------- -----------------
240780020 239863857 240565035 239628474
The following is example output for a nonexistent peering.
Get-AzExpressRouteCircuitRouteTable : The BGP Peering AzurePublicPeering with Service Key <ServiceKey> is not found.
StatusCode: 400
Test private peering connectivity
Test private peering connectivity by counting packets arriving at and leaving the Microsoft edge of your ExpressRoute circuit on the MSEE devices. This diagnostic tool uses an ACL to count packets hitting specific rules, confirming connectivity.
Before you run the test, validate BGP and routes on the MSEE and confirm that the expected on-premises prefixes appear. This check helps distinguish a missing route advertisement from traffic-filtering issues before you evaluate packet counts.
Run a test
Follow these steps to run a private peering connectivity test:
In the Azure portal, select Diagnose and solve problems from your ExpressRoute circuit.
Select Connectivity & Performance issues.
In the Tell us more about the problem you are experiencing dropdown menu, select Issues with Private peering.
Expand the Test private-peering connectivity section.
Run the PsPing test from your on-premises IP to your Azure IP, and keep it running during the test.
Fill out the form fields with the same IP addresses used in step 5, select Submit, and wait for results.
Interpret results
Review the results for the primary and secondary MSEE devices:
- Matches sent and received on both MSEEs - Indicates healthy traffic inbound and outbound. Any loss is downstream from the MSEEs.
- Received matches but no sent matches - Traffic is reaching Azure but not returning. Check return-path routing issues.
- Sent matches but no received matches - Traffic is reaching on-premises but not returning to Azure. Work with your provider to resolve.
- One MSEE shows no matches, the other shows good matches - Indicates one MSEE isn't receiving or passing traffic. It might be offline.
- If you're testing PsPing from on-premises to Azure, received results show matches, but sent results show no matches - This result indicates that traffic is coming in to Azure but isn't returning to on-premises. Check for return-path routing issues. For example, are you advertising the appropriate prefixes to Azure? Is a user-defined route (UDR) overriding prefixes?
- If you're testing PsPing from Azure to on-premises, sent results show matches, but received results show no matches - This result indicates that traffic is coming in to on-premises but isn't returning to Azure. Work with your provider to find out why traffic isn't being routed to Azure via your ExpressRoute circuit.
- One MSEE shows no matches, but the other shows good matches - This result indicates that one MSEE isn't receiving or passing any traffic. It might be offline (for example, BGP or ARP is down).
- You can run additional testing to confirm the unhealthy path by advertising a unique /32 on-premises route over the BGP session on this path.
- Run Test your private peering connectivity using the unique /32 advertised as the on-premises destination address and review the results to confirm the path health.
Your test results for each MSEE device look like the following example.
src 10.0.0.0 dst 20.0.0.0 dstport 3389 (received): 120 matches
src 20.0.0.0 srcport 3389 dst 10.0.0.0 (sent): 120 matches
Verify availability of the virtual network gateway
The ExpressRoute virtual network gateway manages connectivity to private link services and private IPs in an Azure virtual network. Microsoft manages this infrastructure and might perform maintenance, which can reduce performance.
To troubleshoot connectivity problems and check for recent maintenance, follow these steps:
In the Azure portal, select Diagnose and solve problems from your ExpressRoute circuit.
Select Performance Issues.
Wait for diagnostics to run and interpret the results.
If maintenance occurs during packet loss or latency, it might contribute to connectivity problems. Follow the recommended steps and consider upgrading the virtual network gateway SKU to support higher throughput and avoid future problems.
Next steps
For more information or help, see the following resources: