Application gateway doesn't work correctly after connecting the private link

vlad_d 40

Hello!
I have an Application Gateway with 2 frontend IP configuration blocks.
One configured for public services with public IP. And another one configured for internal services with IP from the private subnet.

Looks like this schem Screenshot 2023-10-31 101340

(but without private link)

When I add a Private link and attach it to the Private Frontend IP Conf, a short downtime occurs, as written in the documentation.

After which hosts that are listening on the Private Frontend IP Conf becomes available again and works as expected.

But the host that is listening on the Public Frontend IP Conf continues to down, and I receive TCP RST when I try to establish an SSL connection.
tcp_dump

I didn't find any descriptions of this problem, nor did I see any restrictions in the documentation for this architecture. I also couldn’t find any error messages in the logs (or maybe I was looking in the wrong place)

Is this some expected error with this configuration? Or just another bug?

P.S
Oh yes, if I connect the Private Endpoint to this Private Link, then everything works as expected and there are no problems

GitaraniSharma-MSFT 49,351 Reputation points Microsoft Employee

2023-10-31T12:19:03.2733333+00:00

Hello @vlad_d ,

Welcome to Microsoft Q&A Platform. Thank you for reaching out & hope you are doing well.

Are you using both the private and public listeners with the same port number?

And could you please clarify the below statement?

If I connect the Private Endpoint to this Private Link, then everything works as expected and there are no problems

Regards,

Gita
vlad_d 40 Reputation points

2023-10-31T12:43:27.4133333+00:00

Hello @GitaraniSharma-MSFT ,
thank you so much for your answer.

Yes, I use 443 port for both listeners, private and public. But this port is binded to different IPs. For the diagram above it will be 5.10.20.30:443 and 10.10.10.7:443

About If I connect the **Private Endpoint** to this **Private Link**, then everything works as expected and there are no problems I meant, that connectivity via Private_endpoint and Private_link to the Private Frontend IP Conf works correctly.
And only public listener is broken.

For example the next connections scenario:
clinet-->private_ep-->private_link-->private_frontend_ip_config --- works
clinet-->private_frontend_ip_config --- works
clinet-->PUBLIC_frontend_ip_config --- broken
GitaraniSharma-MSFT 49,351 Reputation points Microsoft Employee

2023-10-31T13:22:38.3866667+00:00

Thank you for the update, @vlad_d .

Do you have a NSG on the Application gateway subnet?

If yes, did you add the rule for the inbound flow required to use same port for both listeners?

Refer: https://learn.microsoft.com/en-us/azure/application-gateway/application-gateway-faq#can-i-use-the-same-port-for-both-public-facing-and-private-facing-listeners

https://learn.microsoft.com/en-us/azure/application-gateway/configuration-listeners#frontend-port
vlad_d 40 Reputation points

2023-10-31T13:41:07.3533333+00:00

Thank you for the update, @GitaraniSharma-MSFT
No, I don't have NSG

I just have the following entities in my resource group

As you can see, I use WAF for the application gateway, but this resource doesn't contain any custom rules.
I use just built-in OWASP-3.2 and Microsoft_BotManagerRuleSet-0.1 rules
GitaraniSharma-MSFT 49,351 Reputation points Microsoft Employee

2023-10-31T13:52:23.3466667+00:00

@vlad_d , may I know what error you receive when you try to access the public listener/URL of your Application gateway?

And what is the backend health status of your Application gateway for the backend HTTP setting used with the public listener?

Refer: https://learn.microsoft.com/en-us/azure/application-gateway/application-gateway-backend-health-troubleshooting#how-to-check-backend-health
vlad_d 40 Reputation points

2023-10-31T14:16:50.02+00:00
@GitaraniSharma-MSFT , yes of course.
I received OpenSSL SSL_connect: Connection reset by peer in connection to my.public.com

Which corresponds to what I see in the tсpdump

Full output

* Trying 5.10.20.30:443... * Connected to my.public.com (5.10.20.30) port 443 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * TLSv1.0 (OUT), TLS header, Certificate Status (22): * TLSv1.3 (OUT), TLS handshake, Client hello (1): * OpenSSL SSL_connect: Connection reset by peer in connection to my.public.com:443 * Closing connection 0 * TLSv1.0 (OUT), TLS header, Unknown (21): * TLSv1.3 (OUT), TLS alert, decode error (562): curl: (35) OpenSSL SSL_connect: Connection reset by peer in connection to my.public.com:443

Backend statuses are ALIVE.
And if I attach these backends to an one of private hosts(which uses private_frontend_ip_config), then I can successfully receive a request from them.

Also backend probes finish successfully.

It seems that on the next scheme client ---> appgw ---> backend these problem locates between client and appgw clinet---XX-->appgw------>backend
GitaraniSharma-MSFT 49,351 Reputation points Microsoft Employee

2023-10-31T14:46:15.9766667+00:00

@vlad_d , what is the type of public listener configured in your Application gateway? Is it basic listener or multi-site listener?

May I know how are you accessing the Public listener of your Application gateway? Is it via the Public IP address or a FQDN?

If FQDN, is your Public IP mapped to the FQDN that you are using to access the Application gateway?
vlad_d 40 Reputation points

2023-10-31T15:51:51.7133333+00:00
@GitaraniSharma-MSFT ,
For public listner I use multi-site listener with Server Name Indication (SNI).

I use terraform azurerm module for that, and it looks like this

resource "azurerm_application_gateway" "appgw" { .... ... http_listener { frontend_ip_configuration_name = "public-ip" frontend_port_name = "port-443" host_names = [ "*.public.com", ] name = "public_listner" protocol = "Https" require_sni = true ssl_certificate_name = "wildcard_ssl_for_public_com" } ....... }

I use FQDN for accessing to this public listener. DNS record is correct and I don't have any problems when Private link is disabled.

If I enable Private link and send request by public ip address without header Host, for example like curl https://5.10.20.30:443 then I get the same error Connection reset by peer

If FQDN, is your Public IP mapped to the FQDN that you are using to access the Application gateway?

Sorry, I didn't understand correctly this question.
Did you mean DNS ptr record?
I have only forward(A) record for my host like my.public.com IN A 5.10.20.30, but dont have 30.20.10.5.in-addr.arpa. PTR my.public.com.
GitaraniSharma-MSFT 49,351 Reputation points Microsoft Employee

2023-11-01T09:18:09.3633333+00:00
Hello @vlad_d ,

I would request a bit more clarity on your setup as the below statements seems to be confusing:

DNS record is correct and I don't have any problems when Private link is disabled. If I enable Private link and send request by public ip address without header Host, for example like curl https://5.10.20.30:443 then I get the same error Connection reset by peer.

Please share the below details:

Have you enabled Private link for both Public and Private listeners?

How is your DNS configured? Do you own the domain (my.public.com) used to access the App gateway?

You mentioned you don't have any problems when private link is disabled ---> do you mean that when the private link is disabled for your private listener, you are able to access the Public listener without any issue? Or does this statement mean something else?

Regards,

Gita

vlad_d 40

Hello @GitaraniSharma-MSFT ,

1. Have you enabled Private link for both Public and Private listeners?

   No, for my goals Private link is required only for Private listener(you can see it on the scheme from my 1st message). But I tested different options, please see result below

How is your DNS configured? Do you own the domain (my.public.com) used to access the App gateway?

  `my.public.com` is just example. But yes, I own the domain which I use to access the Application gateway. In the thread above I said that this is a simple A record

You mentioned you don't have any problems when private link is disabled ---> do you mean that when the private link is disabled for your private listener, you are able to access the Public listener without any issue? Or does this statement mean something else?

Correct, I meant that when the private link is disabled for the private listener, I can able to access to the Public listener. But I tested different options, please see result below

Small tests

I tried enable/disable the Private link for listeners in different combinations.
As can be seen from the results, the public listener doesn't work only in one case - when PL is enabled for the private listener and disabled for the Public listener.

PL of private FE config enabled	PL of public FE config enabled	Public listener works	Private listener works
True	False	False :(	True :)
False	True	True :)	True :)
True	True	True :)	True :)

PL - private link of application gateway

FE - FrontEnd
I tested both option - multi-site(hosts = ['*.example.com']) and single site (host="example.com") - the results are the same

GitaraniSharma-MSFT 49,351 Reputation points Microsoft Employee

2023-11-01T13:17:05.2833333+00:00

@vlad_d , thank you so much for the detailed update. Let me check this issue internally and discuss it with the Application gateway Product Group team. I will keep you posted.
GitaraniSharma-MSFT 49,351 Reputation points Microsoft Employee

2023-11-01T13:52:08.2133333+00:00

<<<UPDATE>>>

@vlad_d , I just discussed this issue with the Application gateway Product Group team and looks like there was a bug with "PLS + Floating IP + Multi site" listener config in Application gateway that was recently addressed, and the fix rollout is underway.

In the meantime, could you try to use different ports for both Public and Private multi-site listeners as a workaround and let me know if it helps.

Thanks!
vlad_d 40 Reputation points

2023-11-02T09:00:14.7633333+00:00

Hello @GitaraniSharma-MSFT ,
thank you so much for the update.

I think that we will refuse to use the Private Link functionality for now and try some kind of workaround, since it is necessary for us to use the default https 443 port for both listeners.

Thank you so much for you help and have a nice day!
GitaraniSharma-MSFT 49,351 Reputation points Microsoft Employee

2023-11-02T12:33:35.8433333+00:00

@vlad_d , thank you the update.

May I know which region your Application gateway is deployed in? As some regions already have the fix available.
vlad_d 40 Reputation points

2023-11-02T12:42:06.05+00:00

@GitaraniSharma-MSFT , yes of course. Our services deployed in West Europe location.
GitaraniSharma-MSFT 49,351 Reputation points Microsoft Employee

2023-11-02T12:59:09.4933333+00:00

Thank you for the update, @vlad_d . Let me check this internally and validate if the fix has already been rolled out for West Europe or not. I will keep you posted.
GitaraniSharma-MSFT 49,351 Reputation points Microsoft Employee

2023-11-02T16:25:23.6366667+00:00

@vlad_d , I checked with the Application gateway Product Group team for an ETA on the fix rollout for West Europe region and they mentioned that it is tentatively planned to complete roll-out in the next two weeks (however, keep in mind that the ETA is subject to change).

I will add the answer below for visibility and closure. If you have any other questions, please let me know.

Please don’t forget to close the thread by clicking "Accept the answer" wherever the information provided helps you, as this can be beneficial to other community members.

Accepted answer

GitaraniSharma-MSFT 49,351 Reputation points Microsoft Employee

2023-11-02T16:25:31.68+00:00

Hello @vlad_d ,

I understand that you have an Application Gateway with 2 frontend IP configurations - one Public and one Private, both listening on the same port 443 with multi-site listeners and this setup works. But when you add a private link and attach it to the Private Frontend IP, a short downtime occurs and post that the hosts that are listening on the Private Frontend IP Conf becomes available again and works as expected, however the host that is listening on the Public Frontend IP Conf continues to be down, and you receive TCP RST when trying to establish an SSL connection.

I discussed this issue with the Application gateway Product Group team and found that there was a bug with "Private link service + Floating IP (using same port on both public & private frontend) + Multi site listener" configuration in Application gateway that was recently addressed, and the fix rollout is underway.

The Product Group team suggested that you can try to use different ports for both Public and Private multi-site listeners as a workaround in the meantime.

However, you didn't want to implement the workaround of using different ports for Public and Private listeners, since it is necessary for your setup to use the default HTTPS 443 port for both listeners. Instead, they would stop using the Private Link functionality for now and try some kind of workaround until the issue is fixed.

Your Application gateway is deployed in West Europe region and since some regions already have the fix available, I reached out to the Application gateway Product Group team to get an ETA on the fix rollout for West Europe region.

The ETA for the fix rollout in West Europe region is tentatively planned to complete in the next two weeks (however, keep in mind that the ETA is subject to change).

Kindly let us know if the above helps or you need further assistance on this issue.

Please "Accept the answer" if the information helped you. This will help us and others in the community as well.
Please sign in to rate this answer.

1 person found this answer helpful.

0 comments No comments
Sign in to comment

Share via

Application gateway doesn't work correctly after connecting the private link

Small tests

0 additional answers