consistent 21 second delay in response for ssh connection to Linux Redhat server in Azure

Lenard Khoshaba 0

A consistent 21-second delay is occurring during SSH connections to a RedHat Linux server (OpenSSH_7.4) in Azure. This delay affects all endpoints trying to connect using SSH including ones on the corporate network going through the firewall and ones that are in the same Azure subscription using the same rsource group and subnets. For some reason out of the 50 endpoints tested only 2 endpoints do not experience this problem when connecting to the linux server using SSH and those 2 endpoints are also in Azure in the same resource group, those 2 connect immediately despite identical network infrastructure and server conditions.

wireshark log shows repeated retransmits for both fast connections and slow connections however on the slow connections there is also a 21 second delay where no traffic is detected to the source connection, other traffic is flowing normally during this 21 second delay.

the delay is always 21 seconds. it was 51 seconds before rebooting the linux server and now has slightly improved to 21 seconds however it was a 1 second connection time a few months ago and no other change has been made to the server other than reboot.

all other server in Azurewith SSH connections do not experience this problem.

wondering if any one else has seen this problem.

KapilAnanth-MSFT 47,726 Reputation points Microsoft Employee

2024-11-05T05:29:37.4966667+00:00
@Lenard Khoshaba ,

Welcome to the Microsoft Q&A Platform. Thank you for reaching out & I hope you are doing well.

This behavior is strange.

Is this delay happening only with SSH

Or the same can be observed with ICMP Ping?

It is also possible that the latency is introduced somewhere in the path and not at the actual Server.

As next steps,

Create a new dummyTestVM in the same subnet as the affectedVM

And try to access this dummyTestVM using SSH from both your corporate network (with Firewall) and from VMs in the same VNET/Subnet

And let us know if the issue persists.

Can you please perform traceroute to the dummyTestVM

From both the corporate network (with Firewall) and from a VM in the same subnet/VNET

And share the results please

Can you also elaborate what you mean by "50 endpoints"

Do you refer to Azure VMs as 50 endPoints.

Do you mean the issue is happening across 48 of 50 VMs?

NOTE : For these VMs to be considered "identical network infrastructure", they should be in the same Subnet, or atleast same VNET and not Resource Group

Resource group has no significance when it comes to latency

Cheers,

Kapil
Lenard Khoshaba 0 Reputation points

2024-11-05T08:42:54.58+00:00

Hi Kapil, see below
Lenard Khoshaba 0 Reputation points

2024-11-05T08:54:40.53+00:00
Thanks for the prompt response Kapil

Is this delay happening only with SSH?

Yes, only SSH connections are affected

ICMP ping times are consistent (1-2ms) from all clients

Other services/ports work normally

Path/Latency Testing:

Some servers being tested are in same Azure VNET/subnet (2 of which dont experience this issue), other servers and clients are spread across the network and can be laptops, servers, desktop etc

Network path is identical (single hop) for Azure servers

Fast and slow clients coexist on same infrastructure

TCP initial handshake is fast for ALL clients (~0.5ms)

Regarding Testing with DummyVM:

This issue does not exist connecting to other linux VMs with SSH exposed in the same Azure VNET

Traceroute Results:

Single hop for all VMs are in same subnet\VNET

Regarding "50 endpoints":

Some are Azure VMs in same VNET/subnet, other endpoints spread across the corporate network.

Pattern observed:

~48 VMs experience 21s delay

~2 VMs connect instantly

Same behavior consistent over time

Same network security group for azure vms

Same routing configuration for azure vms

Connection Pattern:

TCP and initial SSH handshake is fast for ALL clients:

TCP handshake: ~0.5ms

SSH version exchange: ~0.03s

Key exchange starts: ~0.27s

Then:

Fast clients: Complete in ~1s

Slow clients: 20s pause, then complete

Server-side evidence (during delay):

Server continues other operations

Fast clients connect successfully

ICMP responses normal

Azure infrastructure communication continues
KapilAnanth-MSFT 47,726 Reputation points Microsoft Employee

2024-11-07T06:18:35.05+00:00
@Lenard Khoshaba ,

Thanks for the information.

When you say

"Other services/ports work normally"

and "Some are Azure VMs in same VNET/subnet, other endpoints spread across the corporate network"

I doubt this is an Azure/platform issue, rather, this looks like OS related issue.

Can you please check if there are any local firewall at OS level performing any validation?

Do you see anything strange while SSH using "-v" parameter?

From Redhat community, see : SSH connections delay about 20 seconds

Cheers,

Kapil
Sai Prasanna Sinde 1,965 Reputation points Microsoft Vendor

2024-11-11T09:24:53.07+00:00

Hi @Lenard Khoshaba ,

Greetings!

Could you please go through the above comment and provide us the information required so that we can assist you further.

Cheers,

Sai.
Sai Prasanna Sinde 1,965 Reputation points Microsoft Vendor

2024-11-12T08:46:01.1066667+00:00

Hi @Lenard Khoshaba ,

Greetings!

I would like to follow up with the thread.

Could you please go through the above comment and provide us the information required so that we can assist you further.

Regards,

Sai.

Share via

consistent 21 second delay in response for ssh connection to Linux Redhat server in Azure

Your answer