Unable to connect to external IP from App Service through 2 VNETs which are peered

Peter Mello 5

Connecting to 172.19.33.32:80 from your App instance - failed, internal error. Please retry. Please submit a feedback if error persists. TCP ping to 172.19.33.32 failed, error code:undefined

I continue to get this error despite multiple troubleshooting efforts.

App Service is part of a App-VNET which is Peered with another VNET that connects to customer networks.

Four months ago we could TCPPING the address above successfully.

Silvia Wibowo 3,411 Reputation points Microsoft Employee

2024-06-12T03:20:28.29+00:00
Hi @Peter Mello , it will be useful to describe more info on the network setup. Is this accurate:

App-Vnet (where App Service is integrated into) <---peered with---> Hub-Vnet <---VPN---> on-premises (customer network)

What App Service Plan are you using? Is it configured with virtual network integration?

Have you checked:

NSG (Network Security Group)?

Firewall rules - both on-premises and in Azure?

Any changes on VPN connection? Is it using BGP or static routes?

Any changes on on-premises router/firewall (termination of VPN tunnel) routing?
Peter Mello 45 Reputation points

2024-06-12T14:04:06.96+00:00

Azure Environment.pdf

Please see the network diagram attached.

Note: the VMSubnet in VNET2 contains a Windows 10 VM that is able to ping the customer's IP (port 80).

No firewall, routing table or NSG rule exists in either VNET.

Peering exists on both sides - I felt it was obvious that if Peering existed in VNET2 the reverse existed in VNET1 so only one config was shown.

The connection (ping) between the App Service and the Customer IP worked 4 months ago
Ben Gimblett 3,825 Reputation points Microsoft Employee

2024-06-13T14:24:50.09+00:00

If I understand correctly you have an App Service which is VNET integrated into VNET-A/Subnet1 which is peered to VNET-B and the VPN is on VNET-B

The VPN is advertising routes to your customer - and the route in question (although private) is on a public prefix (non RFC1918 space)

I'm assuming the VM you mention is on the VNET with the VPN, but it should not matter , if the peering is correctly configured the routes should be advertised in both directions and include VPN routes provided gateway transit is enabled https://learn.microsoft.com/en-us/azure/vpn-gateway/vpn-gateway-peering-gateway-transit

The one thing you need to double check here is that the VNET integration for app service is set to force ALL traffic through the VNET integration. By default "internet" traffic will breakout from the App Service and won't be seen by your VNET and that would be a problem in your scenario as the destination would be considered "internet" as the prefix is not in the private range.
https://learn.microsoft.com/en-us/azure/app-service/overview-vnet-integration#application-routing

If app routing and VNET route-all is enabled ALL traffic will go to the VNET but (and you state you are not using an NVA/firewall/NATGW) the source IP will not be NAT to the subnet prefix - it'll still be the IP from App Service.

If you've ticked all the above boxes I'm just wondering if that last point may pose a problem. I'm going to check with peers what happens to the return packets from the VPN in this case and whether they'd go asymmetrically via the public endpoint , given we state they're not NAT to the subnet prefix
Ben Gimblett 3,825 Reputation points Microsoft Employee

2024-06-13T14:58:50.51+00:00

On that last point; there should be no asymmetric issue. The vnet-route-all means that internet bound traffic is visible to the VNET , but unless there's a valid route on the VNET (for example a UDR towards a Firewall) it'll still break out normally . But, in this case you would have a valid route (again if your VPN is on the peer you'd need the peering setup correct) so the traffic would be NAT and return packets would go back symmetrically.
Peter Mello 45 Reputation points

2024-06-13T20:23:47.67+00:00

Thank for for all that info, @Ben Gimblett . I have checked the box for Outbound internet traffic under Application Routing and also see that Route All is enabled. I am still unable to ping the 172.19 address. What is more puzzling to me is that I also cannot ping (from the App Service) the open port (8080) on the VM that's in the same subnet as the App Service. Outgoing and incoming is enabled.
Silvia Wibowo 3,411 Reputation points Microsoft Employee

2024-06-14T02:35:57.9233333+00:00
Hi @Peter Mello , your diagram mentioned 2 subnets:

App Subnet: 192.168.2.0/27 delegated to Microsoft.Web

VM Subnet: 192.168.2.0/29

Those 2 CIDRs are overlapping.

You mentioned in your comment "VM that's in the same subnet as the App Service" - does that mean it's the same subnet for App Service vnet integration and your VM?
Ben Gimblett 3,825 Reputation points Microsoft Employee

2024-06-14T08:31:48.7466667+00:00

Peter - to add to Silvia's comment
(1) you must not have overlapping address space between two peered VNETs - this will need to be corrected . Without trying I thought there was a validation check when you tried to peer if there's overlapping space. In any case this needs to be fixed as overlapping address space would break intra-vnet-routing via the peering.
(2) When you say "the VM that's in the same subnet as the App Service" the subnet for app service must be exclusive - so the VM would have to be in a different subnet or vnet

Note: A colleague tested the routing for regional VNET integration early this morning and confirms that provided vnet-route-all is enabled (and you confirm it is) then the traffic out from App service will be offered to the VNET/subnet routes and will be SNAT (source ip from subnet prefix) if there's a valid path / route. If there's not it'll breakout as usual (source ip from the App service IP list)
Peter Mello 45 Reputation points

2024-06-17T12:53:58.97+00:00

@Silvia Wibowo I misspoke. The VM is in it's own subnet. That said, I am going to remove the VM and the VM's subnet and try to connect again.
Ben Gimblett 3,825 Reputation points Microsoft Employee

2024-06-17T13:11:20.0166667+00:00

Peter - Quick point to clarify - is your VPN using software on the VM or are you using Az VPN Gateway?
Peter Mello 45 Reputation points

2024-06-17T13:14:28.0233333+00:00

@Ben Gimblett we are using the Azure VPN Gateway. Thanks! The PDF output of the diagram is terrible. I'm going to ensure it looks better.
Peter Mello 45 Reputation points

2024-06-17T14:01:12.06+00:00

I have removed the VM and the VM subnet from the equation and restarted the App Service. I tried to ping the customer endpoint that failed. I ran the Network/Connectivity Troubleshooter and here are the results:
Peter Mello 45 Reputation points

2024-06-17T14:21:35.96+00:00

Config.jpg (Overall configuration)

Peerings.jpg (Peering between VNETs)

Virtual_Network_Integration.jpg (Shows Application Routing enabled)

VNET_Integration.jpg (Shows Route All enabled)

@Ben Gimblett @Silvia Wibowo Updated diagrams - hopefully these are readable!!

Thank you!!!
Ben Gimblett 3,825 Reputation points Microsoft Employee

2024-06-17T14:35:13.4033333+00:00

Peter - assuming there's no longer any overlapping address space, appears not based on the new info

Can you check you have a peering both ways
vnet1=>vnet2
vnet2=>vnet1
And that "use remote gateways" and "gateway transit" are also set appropriately

As per info
https://learn.microsoft.com/en-us/azure/virtual-network/virtual-network-peering-overview
and
https://learn.microsoft.com/en-us/azure/vpn-gateway/vpn-gateway-peering-gateway-transit
Peter Mello 45 Reputation points

2024-06-17T14:54:41.95+00:00

VNET2 -> VNET1 Peering

Full Synchronized: True

Peering State Connected: True

Virtual network gateway or route server: Disabled

If I try to enable the checkbox:

Allow gateway or route server in 'VNET2' to forward traffic to 'VNet1'

I get the error/message:

'VNET2' does not have a VPN gateway or route server. To enable this option, 'VNET2' needs to have a VPN gateway or route server

VNET1 -> VNET2 Peering

Full Synchronized: True

Peering State Connected: True

Virtual network gateway or route server: Enabled
Ben Gimblett 3,825 Reputation points Microsoft Employee

2024-06-17T15:05:23.42+00:00

you only have one vpn gw which is on the vnet which doesnt have your app service linked? (your diagram shows too but i assumed it was a typo)
Peter Mello 45 Reputation points

2024-06-17T15:17:05.5433333+00:00

@Ben Gimblett Forgive my ignorance, but, by your response, it appears that I need a VPN Gateway on the App subnet and that is most likely the issue.
Ben Gimblett 3,825 Reputation points Microsoft Employee

2024-06-17T15:21:20.7033333+00:00

Peter - sorry no. I was trying to clarify the setup as the diagram shows two. The VPN gateway could be on the same VNET but then you would not need the peering.

In your case I think you mean (typo in the drawing aside) that you have a single VPN GW in the remote VNET and you're peering to it and allowing transit . And that is fine.

You should check the routes are being advertised correctly . The quickest way to do this is deploy a linux VM to an adjacent subnet in the same vnet as the app service, then go to the virtual NIC and you should see the route the VPN GW is advertising across the peering. If you dont then something is either wrong with the VPN config or the peering. You need to do this because there's no way of seeing effective routes etc on the App service vnet integration
Ben Gimblett 3,825 Reputation points Microsoft Employee

2024-06-17T18:41:38.63+00:00

OK , so to recap
*your web app is successfully vnet integrated with route-all enabled

*your peering looks correct
*peering allows gateway transit
*vpn is up
*the route you need is showing OK from a VM v-nic on an adjacent subnet to the one the app is integrated with

All of which suggests packets should be reaching at least the VPN GW as per the routes

As a next step it would be worth running a packet capture to confirm this https://learn.microsoft.com/en-us/azure/vpn-gateway/packet-capture#about-packet-capture-for-vpn-gateway
Silvia Wibowo 3,411 Reputation points Microsoft Employee

2024-06-18T02:46:39.5266667+00:00

Hi @Peter Mello , your screenshot showed Network/Connectivity Troubleshooter. By its definition: Connection troubleshoot enables you to test a connection between a virtual machine, a virtual machine scale set, an application gateway, or a Bastion host and a virtual machine, an FQDN, a URI, or an IPv4 address.

It's a part of Network Watcher, which isn't a solution for monitoring PaaS (Platform-as-a-Service) infrastructure.

Did you test the traffic from App Service itself?
Peter Mello 45 Reputation points

2024-06-18T13:23:51.41+00:00

Hi @Silvia Wibowo - From the App Service Console and Advanced Tools, I have used TCPPING to test connectivity. This was effective and successful back in February. (TCPPING 172.19.33.32:80)
Peter Mello 45 Reputation points

2024-06-18T16:46:18.2666667+00:00

@KapilAnanth-MSFT The customer states that port 80 is open. They tried to ping the app service IP and received a time out. Below is the email they sent. "current_peer" is the gateway address.

@Mello-I am pinging to 192.168.2.30 from 172.19.33.32 and we see the packets enter the tunnel but do not receive a response

current_peer: 52.152.60.26

       #pkts encaps: 11, #pkts encrypt: 11, #pkts digest: 11

      #pkts decaps: 0, #pkts decrypt: 0, #pkts verify: 0

      #pkts compressed: 0, #pkts decompressed: 0

      #pkts not compressed: 11, #pkts comp failed: 0, #pkts decomp failed: 0

      #pre-frag successes: 0, #pre-frag failures: 0, #fragments created: 0

      #PMTUs sent: 0, #PMTUs rcvd: 0, #decapsulated frgs needing reassembly: 0

      #TFC rcvd: 0, #TFC sent: 0

      #Valid ICMP Errors rcvd: 0, #Invalid ICMP Errors rcvd: 0
Peter Mello 45 Reputation points

2024-06-24T21:47:37.3733333+00:00

Hi @KapilAnanth-MSFT - I had a call with the customer today which was not successful. I would greatly appreciate some support on this issue. How do I go about setting this up? Thank you!! Peter Mello
KapilAnanth-MSFT 39,051 Reputation points Microsoft Employee

2024-06-25T04:19:06.02+00:00

@Peter Mello ,

Sure.

Please share the subscription ID of the VPN Gateway in private messages.

Cheers,

Kapil
Peter Mello 45 Reputation points

2024-06-25T13:55:49.98+00:00

Hi @KapilAnanth-MSFT I do not see a means of starting a private message. From what I read, that is reserved for Forum Owners. Can I email that to you?
KapilAnanth-MSFT 39,051 Reputation points Microsoft Employee

2024-06-25T13:58:43.7866667+00:00

Yes Peter, the question was created by the user : https://learn.microsoft.com/en-us/users/petermello-0910/.

Can you login to the above account and check the private messages please?
Peter Mello 45 Reputation points

2024-06-25T14:59:09.6433333+00:00

@KapilAnanth-MSFT I still cannot find a private message link anywhere. I sent you an email and hoping I got your email correct. Thanks!
Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.
KapilAnanth-MSFT 39,051 Reputation points Microsoft Employee

2024-06-27T12:04:26.2633333+00:00

@Peter Mello ,

Sure

1 answer

KapilAnanth-MSFT 39,051 Reputation points Microsoft Employee

2024-06-12T05:21:49.3266667+00:00
@Peter Mello ,

Welcome to the Microsoft Q&A Platform. Thank you for reaching out & I hope you are doing well.

"172.19.33.32" is not an "external IP" , i.e., not a Public one

See : RFC 1918

Can you provide more information on this IP?

Where exactly does this IP exist?

Is it associated to a VM in the same VNET or the Peered VNET
OR
A server from OnPremises connected to Azure by VPN or ExpressRoute?

There is a high chance that the IP "172.19.33.32" is not listening in Port 80 to begin with

How are you confirming that the IP 172.19.33.32 is "still" listening on 80?

You can use a VM in the same VNET as the AppService and telnet to 172.19.33.32:80 and test this

If possible, please share a network diagram between your App VNET and the destination "172.19.33.32"

Cheers,

Kapil
Please sign in to rate this answer.

1 person found this answer helpful.
Peter Mello 45 Reputation points

2024-06-17T17:00:29.09+00:00

@Ben Gimblett "In your case I think you mean (typo in the drawing aside) that you have a VPN GW in the remote VNET and you're peering to it and allowing transit"

Yes!

I did as you suggested. Here's the output of the Effective Routes

KapilAnanth-MSFT 39,051 Reputation points Microsoft Employee

2024-06-18T14:12:02.53+00:00

@Peter Mello ,

I see Ben Gimblett and Silvia Wibowo have driven the troubleshooting and we arrive at the following

Summary:

Your target is 172.19.33.32:80, the server "172.19.33.32" is located on OnPrem

There are no Route tables (UDRs) or Network security groups attached to the AppServiceIntegratedSubnet

You have a VM in the same VNET as the App Service integration.

From it's NIC Effective Route, we can see route to "172.19.33.32" is being learnt

Next Steps,

Can you share the result from the VM telnet/tcpping to 172.19.33.32:80?

I understand you mentioned this was working with an old VM, but I would appreciate if you could share the results from the VM as a screenshot.

From the App Service's console, can you try telnet/tcpping to <VM's IP Address> on either port 22 or 3389 (depending on the OS)

Please share the screenshot from App Service's console

Is there any UDR or NSG attached to the GatewaySubnet?

Do you have access to the server "172.19.33.32" ?

Without testing the port from Azure, can you directly login to 172.19.33.32 and confirm that the server is actually listening on Port 80 to begin with?

If step 1 worked, this last step is not necessary

Cheers,

Kapil

Peter Mello 45 Reputation points

2024-06-18T14:57:02.7+00:00

@KapilAnanth-MSFT It looks like the VM can't connect either. In addition to traceroute (below), I've tried ping and telnet and nothing connects. Port 80 is open - both incoming and outgoing

Peter Mello 45 Reputation points

2024-06-18T15:03:26.8133333+00:00

@KapilAnanth-MSFT Was able to ping (TCPPING) from App Service console to VM

192.168.2.30 -> 192.168.2.252 over port 22

KapilAnanth-MSFT 39,051 Reputation points Microsoft Employee

2024-06-18T15:34:19.8166667+00:00

@Peter Mello ,

This means the server "172.19.33.32" may not even be listening on Port 80 in the first place.

What about my other steps in the action plan?

Is there any UDR or NSG attached to the GatewaySubnet?

Do you have access to the server "172.19.33.32" ?

Without testing the port from Azure, can you directly login to 172.19.33.32 and confirm that the server is actually listening on Port 80 to begin with?

You can simply do a netstat -ln | grep '80' ,share a screenshot and confirm the server is actually listening

What about the reverse traffic? From this server "172.19.33.32" to the VM on Port 22?

Also, there is a good chance that the

Server is not accepting connection from Azure IPs - OS level firewalls maybe?

OnPrem VPN Device has a Firewall and is not letting Azure IPs to connect?

Can you confirm the above 2 are not the cases?

KapilAnanth-MSFT 39,051 Reputation points Microsoft Employee

2024-06-18T17:00:40.16+00:00

@Peter Mello ,

Thanks for sharing the info.

As a last resort,

Please ask your client to telnet/tcpping the VM and not App Service.

In fact, I am not sure how you can ping App Service integrated into a VNET via Private IP

AFAIK, VNET integration only provides outbound connectivity from the AppService to private resources via Private Network

VNET integration does not provide inbound connectivity to the AppService from private resources via Private Network. Only Private EndPoint can do that.

Is there any other VMs <---> Server communications happening?

i.e. Can your client telnet the VM (not the app service)

Does your client have any other server (other than 172.19.33.32) that you can telnet/tcpping to from the VM

If not, this whole set up appears to be a misconfiguration on the VPN Part and no connectivity between any resources were even working in the first place.

Can you get this validated.

Cheers,

Kapil

Ben Gimblett 3,825 Reputation points Microsoft Employee

2024-06-18T18:21:49.31+00:00

Kapil-- +1 Vnet integration is outbound only. Also agree on VPN statement. The trouble shooting so far points to the VPN itself (could also be a network/device of Peter mellos customer) not working correctly

Peter Mello 45 Reputation points

2024-06-18T21:00:57.4966667+00:00

@KapilAnanth-MSFT @Ben Gimblett The customer attempted to ping the VM and was unsuccessful. He asked "Do you see any decrypts for the tunnel in your logs?" Do you know where I'd find that?

KapilAnanth-MSFT 39,051 Reputation points Microsoft Employee

2024-06-19T04:44:33.5766667+00:00

@Peter Mello ,

Wrt, "Do you see any decrypts for the tunnel in your logs?"

This seems to be specific to the VPN Device they are using

Azure VPN does not have such logs

Complete list of logs available for VPN Gateway can be found in : Azure VPN Gateway diagnostic logs.

I believe the configuration from Azure end is correct,

As we can see the nextHop as VPN Gateway in the Effective Routes

It's possible either the OnPrem VPN Device drops the packet or packets are lost in transit

To troubleshoot further, we will need a specialized 1:1 session, where a support engineer can have a screen share session to pinpoint the issue.

If you have a support plan you may file a support ticket, else please do let us know, we will try and help you get a one-time free technical support.

P.S : We will be able to provide you a one time free support only if you are not having a CSP Subscription

Cheers,

Kapil
Sign in to comment

Share via

Unable to connect to external IP from App Service through 2 VNETs which are peered

1 answer