telnet/openssl connect to AFD custom domain times out after 5 seconds

Question

telnet/openssl connect to AFD custom domain times out after 5 seconds

Zeinab Khosravi 60

Hello all,

we have an Azure Front Door Premium on one of our customer projects. The customer has high level of incoming data, in average 200.000 per minute. Some of the devices that initiate connection to AFD don't support HTTPs and use HTTP for the connection. All the devices work as follow:

They open a connection to the custom domain defined on AFD, then they prepare their data that needs to be sent, which takes somewhere between 15 to 20 seconds. Then they transfer the data over this connection.

The problem is, some of the devices that initiate the connection using HTTP, receive a timeout after 5 seconds. I also have realized that these connections are not logged under FrontDoorAccessLog so there is no way I can investigate the issue using logs. I tried to re-create the issue myself using telnet command, and I can see that the connection is closed after about 5 seconds with the following message:

Connection closed by foreign host.

In addition to that, I also tried the openssl s_client -connect command to check the connection with TLS, and this request also closes after about 5 seconds.

As far as I know, the AFD TCP connection timeout is 61 seconds by default (I don't know if this could change or not), the keep alive timeout is 90 seconds, and my origin timeout is set to 120 seconds! So I don't see why a request should close/ timeout after 5 seconds. My guess is, since I also see no related logs, it might be that the requests don't reach my AFD profile for some reason!!

Your help is highly appreciated on the matter.

Regards

KapilAnanth 49,876 Reputation points Moderator

2023-11-13T12:31:22.8333333+00:00
@Zeinab Khosravi

Welcome to the Microsoft Q&A Platform. Thank you for reaching out & I hope you are doing well.

I understand that you are receiving frequent time outs while accessing Azure Front Door.

Are you saying that TCP Connection itself is timing out?

Or TCP is successful but further HTTP/HTTPS Connection is timing out?

Do you receive any HTTP Response before the connection times out?

Or the HTTP initiation itself fails and results in time out?

Can you share the failure for a "curl" or "wget" command?

I would like to know if this timeout happens at TCP or at HTTP. I am interested in the headers you receive in the HTTP Response.

Cheers,

Kapil.
Zeinab Khosravi 60 Reputation points

2023-11-13T13:05:25.53+00:00

Hello @KapilAnanth

thank you for replying.

As I am using telnet command to re-create the issue myself, I would say the TCP seems to fail, or at least that is my guess.

A curl wouldn't work, because there is no website on the other side of the URL. An AKS cluster service on the backend of AFD should receive the data sent by the devices. So, when I try to connect to the URL, I get 500 internal error!

I also don't get a timeout message per-se, as I mentioned in the original post, I receive:

Connection closed by foreign host.

The devices however see the following error 5 seconds after they try to establish a connection:

Could not get orders from portal: Post "URL": read tcp LOCAL-IP:PORT ->AFD-IP:443: read: connection reset by peer

I hope this is helpful.
Zeinab Khosravi 60 Reputation points

2023-11-13T13:19:27.0366667+00:00

This comment is removed!
KapilAnanth 49,876 Reputation points Moderator

2023-11-14T10:21:39.48+00:00

@Zeinab Khosravi

In that case, I think we will need a specialized 1:1 session, where a support engineer can have a screen share session to pinpoint the issue.

If you have a support plan you may file a support ticket, else please do let us know, we will try and help you get a one-time free technical support.

Cheers,

Kapil
Zeinab Khosravi 60 Reputation points

2023-11-14T10:23:43.65+00:00

@KapilAnanth

thank you for the information. I do have a support plan, I was just hoping there might be something I am missing which is straight forward.. I will open a support ticket then. Thanks for your help.
KapilAnanth 49,876 Reputation points Moderator

2023-11-14T10:25:21.2433333+00:00

@Zeinab Khosravi

Sure.

Please keep us posted on how it goes.

Cheers,

Kapil
Setthawut Janarphorn 0 Reputation points

2024-02-16T11:40:03.6666667+00:00

Can you update, how was going ?. I facing about this error too. I have mobile device connect to AFD to Storage Account CDN Endpoint, and sometime got error "Connection Reset by Peer". I also have realized that these connections are not logged under FrontDoorAccessLog too. @Shadi

2 answers

Your answer

KapilAnanth 49,876 Reputation points Moderator

2023-11-13T12:31:22.8333333+00:00

@Zeinab Khosravi

Welcome to the Microsoft Q&A Platform. Thank you for reaching out & I hope you are doing well.

I understand that you are receiving frequent time outs while accessing Azure Front Door.

Are you saying that TCP Connection itself is timing out?

Or TCP is successful but further HTTP/HTTPS Connection is timing out?

Do you receive any HTTP Response before the connection times out?

Or the HTTP initiation itself fails and results in time out?

Can you share the failure for a "curl" or "wget" command?

I would like to know if this timeout happens at TCP or at HTTP. I am interested in the headers you receive in the HTTP Response.

Cheers,

Kapil.
Zeinab Khosravi 60 Reputation points

2023-11-13T13:05:25.53+00:00

Hello @KapilAnanth

thank you for replying.

As I am using telnet command to re-create the issue myself, I would say the TCP seems to fail, or at least that is my guess.

A curl wouldn't work, because there is no website on the other side of the URL. An AKS cluster service on the backend of AFD should receive the data sent by the devices. So, when I try to connect to the URL, I get 500 internal error!

I also don't get a timeout message per-se, as I mentioned in the original post, I receive:

Connection closed by foreign host.

The devices however see the following error 5 seconds after they try to establish a connection:

Could not get orders from portal: Post "URL": read tcp LOCAL-IP:PORT ->AFD-IP:443: read: connection reset by peer

I hope this is helpful.
Zeinab Khosravi 60 Reputation points

2023-11-13T13:19:27.0366667+00:00

This comment is removed!
KapilAnanth 49,876 Reputation points Moderator

2023-11-14T10:21:39.48+00:00

@Zeinab Khosravi

In that case, I think we will need a specialized 1:1 session, where a support engineer can have a screen share session to pinpoint the issue.

If you have a support plan you may file a support ticket, else please do let us know, we will try and help you get a one-time free technical support.

Cheers,

Kapil
Zeinab Khosravi 60 Reputation points

2023-11-14T10:23:43.65+00:00

@KapilAnanth

thank you for the information. I do have a support plan, I was just hoping there might be something I am missing which is straight forward.. I will open a support ticket then. Thanks for your help.
KapilAnanth 49,876 Reputation points Moderator

2023-11-14T10:25:21.2433333+00:00

@Zeinab Khosravi

Sure.

Please keep us posted on how it goes.

Cheers,

Kapil
Setthawut Janarphorn 0 Reputation points

2024-02-16T11:40:03.6666667+00:00

Can you update, how was going ?. I facing about this error too. I have mobile device connect to AFD to Storage Account CDN Endpoint, and sometime got error "Connection Reset by Peer". I also have realized that these connections are not logged under FrontDoorAccessLog too. @Shadi

Answer 1

Hey, so according to Azure support, they are moving from FDV2as their AFD working platform to something called Roxy. The new platform has the default timeout of 5 seconds, and will close te connection if no data is sent within that time. Here is the Azure support explanation: "Clients are timing out faster (< 5s) when they connect to Roxy if the client doesn't send any data after establishing TLS connection. This timeout is undocumented. We went with 5s to limit the attack surface on Roxy. On FDv2, the timeout is 60s but FDv2 will go away soon, and we would like to stick with 5s given it's a platform protection. Roxy will be the new platform for AFD and will be serving 100% of all customers traffic soon. There will be no way to move the customer away from Roxy. We are limiting the time to connect from platform protection point of view. We would like to stick with 5s. Can the customer adjust their client connect settings." In my case, we moved those clients to application gateway, to fix the issue.

Answer 2

@Shadi ,

I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this!

Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others", I'll repost your solution in case you'd like to "Accept" the answer.

Issue: Telnet/Openssl connect to AFD custom domain times out after 5 seconds

Solution: You reached out to Azure Support and they updated the below,

Azure is migrating from FDV2as their AFD working platform to something called Roxy.
The new platform has the default timeout of 5 seconds, and will close to connection if no data is sent within that time.
Azure Support's explanation:

"Clients are timing out faster (< 5s) when they connect to Roxy if the client doesn't send any data after establishing TLS connection. This timeout is undocumented. We went with 5s to limit the attack surface on Roxy. On FDv2, the timeout is 60s but FDv2 will go away soon, and we would like to stick with 5s given it's a platform protection. Roxy will be the new platform for AFD and will be serving 100% of all customers traffic soon. There will be no way to move the customer away from Roxy. We are limiting the time to connect from platform protection point of view. We would like to stick with 5s. Can the customer adjust their client connect settings."

In your case, you had moved those clients to Application gateway, to work around this.

Thanks,

Kapil

Please Accept an answer if correct. Original posters help the community find answers faster by identifying the correct answer.

Share via

telnet/openssl connect to AFD custom domain times out after 5 seconds

2 answers

Your answer