Share via

Hybrid Connect preventing clean shutdown of TCP connections

Jasper Reddin 0 Reputation points
2026-04-09T15:53:32.29+00:00

We have several .NET App Services running on Linux App Service plans. They are all connected via Hybrid Connect to different on-prem Windows Servers. I started noticing that after an application has been started up and I send several requests that open a TCP connection to the on-prem servers, they do not close fully and are indefinitely stuck in the CLOSE_WAIT state when using netstat in the App Service SSH console.

Originally I thought it was an issue with SqlClient, but I put together a bare-bones TCP socket server program, and a web application that connects to it, receives data from the server, and closes the connection. And sure enough, pure TCP socket communication is having the same issue.

On the index page of the web application:

var ipHostInfo = await Dns.GetHostEntryAsync("******");
var ipEndpoint = new IPEndPoint(ipHostInfo.AddressList[0], 5000);

var sb = new StringBuilder();        

// establish connection
using var client = new Socket(ipEndpoint.AddressFamily, SocketType.Stream, ProtocolType.Tcp);
await client.ConnectAsync(ipEndpoint);

// send message
var message = "Hi friends! <|EOM|>";
var messageBytes = Encoding.UTF8.GetBytes(message);
await client.SendAsync(messageBytes, SocketFlags.None);
_logger.LogInformation("Sent message: {message}", message);

// response is broken into multiple messages. use sb to build response into a single string
while (true)
{
    var buffer = new byte[1024];
    var receivedLength = await client.ReceiveAsync(buffer, SocketFlags.None);
    var response = Encoding.UTF8.GetString(buffer, 0, receivedLength);
    sb.Append(response);
    if (response.Contains("<|ACK|>"))
    {
        _logger.LogInformation("Received acknowledgement: {response}", response);
        break;
    }
    else
    {
        _logger.LogInformation("Received response: {response}", response);
    }
}

Server code:

var ipEndpoint = new IPEndPoint(IPAddress.Parse("0.0.0.0"), 5000);

using var listener = new Socket(ipEndpoint.AddressFamily, SocketType.Stream, ProtocolType.Tcp);
listener.Bind(ipEndpoint);
listener.Listen(100);

while (true)
{
    Console.WriteLine($"Listening on {ipEndpoint.Address}:{ipEndpoint.Port}");
    using var handler = await listener.AcceptAsync();
    Console.WriteLine($"{handler.RemoteEndPoint} connected");
    while (true)
    {
        // Receive message.
        var buffer = new byte[1_024];
        var received = await handler.ReceiveAsync(buffer, SocketFlags.None);
        var response = Encoding.UTF8.GetString(buffer, 0, received);

        var eom = "<|EOM|>";
        if (response.IndexOf(eom) > -1 /* is end of message */)
        {
            Console.WriteLine(
                $"Socket server received message: \"{response.Replace(eom, "")}\"");

            var ackMessage = "<|ACK|>";
            var echoBytes = Encoding.UTF8.GetBytes(ackMessage);
            await handler.SendAsync(Encoding.UTF8.GetBytes(DateTime.UtcNow.ToString("s")));
            await handler.SendAsync(echoBytes, 0);
            Console.WriteLine(
                $"Socket server sent acknowledgment: \"{ackMessage}\"");

            break;
        }
    }
    handler.Close();
}

Approximately 5 minutes after sending a few requests to the web application after freshly restarting it, this is the result of ss -tnp

State         Recv-Q Send-Q     Local Address:Port             Peer Address:Port     Process
CLOSE-WAIT    0      0             127.0.0.14:5000                127.0.0.1:33672                                                                                                                
CLOSE-WAIT    0      0             127.0.0.14:5000                127.0.0.1:33674                                                                                                                
FIN-WAIT-2    0      0              127.0.0.1:57714              127.0.0.14:5000                                                                                                                 
CLOSE-WAIT    0      0             127.0.0.14:5000                127.0.0.1:57714                                                                                                                
CLOSE-WAIT    0      0             127.0.0.14:5000                127.0.0.1:59332                                                                                                                
ESTAB         0      0          169.254.136.2:2222            169.254.136.3:58672     users:(("sshd",pid=1935,fd=3))                                                                             
CLOSE-WAIT    0      0             127.0.0.14:5000                127.0.0.1:40104                                                                                                                
CLOSE-WAIT    0      0             127.0.0.14:5000                127.0.0.1:53386                                                                                                                
ESTAB         0      0 [::ffff:169.254.136.2]:58030  [::ffff:13.73.253.115]:443       users:(("dotnet",pid=1874,fd=292))

There are no exceptions thrown in the application, and I cannot find anything in the logs or diagnostic tools provided by Azure.

In fact, if I replace the last handler.close() statement in the server code with var len = await handler.ReceiveAsync(new byte[1024], SocketFlags.None);, in a healthy system, the task resolves with 0 bytes read into the buffer when the client code closes the connection from its side. But through the hybrid connection, the task never resolves when the connection is closed.

Here is the full reproducible demo: https://github.com/DrOverbuild/HybridConnectionLeak/tree/pure-tcp-sockets

I found a couple other users in this forum experiencing the same issue:

Azure App Service
Azure App Service

Azure App Service is a service used to create and deploy scalable, mission-critical web apps.

2 answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. Golla Venkata Pavani 6,165 Reputation points Microsoft External Staff Moderator
    2026-04-09T16:36:46.4433333+00:00

    Hi @Jasper Reddin

    Thank you for reaching us regarding the issue.

    At present, our engineering team has identified this as an issue currently under active investigation. Work is in progress, and a fix is being targeted in an estimated timeframe of approximately 4–6 weeks. Current status

    The issue is not related to your application logic and aligns with the behavior observed through Hybrid Connections.

    Our internal teams are actively working on a longer-term resolution.

    Interim recommendation:

    As a temporary mitigation, our backend team has suggested periodically restarting the App Service, which helps clear existing TCP connections and prevent accumulation of stale connections.

    Please note that this is a workaround only, not a permanent fix, and is being recommended to minimize impact until the underlying issue is resolved.

    Kindly let us know if the above helps or you need further assistance on this issue.

    If the answer is helpful,  Please do click "Accept the answer” and Yes, this can be beneficial to other community members.

    Was this answer helpful?

    0 comments

  2. AI answer

    2026-04-09T15:54:07.55+00:00

    Hybrid Connections work at the TCP stream level and terminate TCP on both sides (App Service side and HCM/on-prem side), then tunnel the data over Azure Relay. This extra termination layer means the TCP close sequence is not a direct end‑to‑end FIN/ACK between client and server, and it can leave connections in CLOSE_WAIT/FIN_WAIT states if either side does not complete the close handshake or if the relay path does not propagate the FIN as expected.

    Key points from the platform behavior and documentation:

    1. Hybrid Connections are TCP-only, stream-level, and use Azure Relay
      • App Service connects to Azure Relay over TLS 1.2 on port 443.
      • Hybrid Connection Manager (HCM) also connects outbound over 443.
      • The app’s outbound TCP to the configured host:port is intercepted and redirected through the Hybrid Connection when the DNS name matches the endpoint.
      • Each Hybrid Connection is a single host:port mapping; the relay path is opaque to the app and the on‑prem server.
      This means the TCP session you see in the App Service container is between the app worker and the relay endpoint, not directly to the on‑prem server. Any asymmetry or delay in how the relay/HCM handles FIN/RST can manifest as lingering CLOSE_WAIT/FIN_WAIT sockets on either side.
    2. CLOSE_WAIT/FIN_WAIT2 is a classic symptom of one side not closing properly
      • In general TCP behavior, CLOSE_WAIT on one side and FIN_WAIT_2 on the other indicates that one endpoint has received a FIN but has not closed its socket, or the peer has not completed the final ACK.
      • There is a known class of issues where client libraries do not close connections correctly, leaving ports in CLOSE_WAIT/FIN_WAIT_2 until garbage collection or timeout releases them. The SQL mirroring example in the context shows exactly this pattern when the client stack does not complete the close sequence.
    3. Hybrid Connection troubleshooting guidance
      • The official guidance for Hybrid Connections focuses on DNS resolution, connectivity, and firewall/proxy behavior, not on per‑connection TCP state. However, it explicitly notes that:
        • All traffic is outbound over 443 and goes through Azure Relay.
        • If a firewall/proxy is in the path, it must support HTTP/HTTPS and WebSocket over 443, and you should allow *.servicebus.windows.net or the specific relay namespace hostnames.
      • Misbehaving or stateful middleboxes (firewalls, proxies, IDS/IPS) between HCM and the on‑prem server can interfere with FIN/RST propagation and leave half‑open sessions.
    4. What can be done from the application and infrastructure side Given the behavior is reproducible with raw sockets and not just SqlClient, and appears only when going through Hybrid Connections, the issue is most likely in the relay/HCM path or something in between, not in the application code itself. The following steps are aligned with the documented capabilities and troubleshooting model:
      1. Validate the on‑prem path from HCM to the server
        • From the HCM host, test connectivity to the on‑prem TCP endpoint using Test-NetConnection EndpointDNSname -Port 5000 (or the actual port) to confirm that there is no firewall or host‑based filter interfering with session teardown.
        • Ensure any on‑prem firewall or proxy between HCM and the server allows long‑lived TCP and does not aggressively drop idle connections without sending FIN/RST.
      2. Confirm DNS and endpoint configuration
        • Ensure the Hybrid Connection uses a DNS name, not an IP address, so that the App Service stack performs the DNS lookup and the Hybrid Connection redirection is applied correctly.
        • Verify that the DNS name resolves correctly from the HCM host using nslookup with the exact name used in the Hybrid Connection definition.
      3. Check HCM and proxy configuration
        • If HCM is configured to use a proxy (via appsettings.json on the HCM host), verify that the proxy supports HTTP/HTTPS and WebSocket over 443 and is not interfering with connection teardown.
        • If possible, temporarily bypass the proxy to see if CLOSE_WAIT behavior changes.
      4. Observe from both ends
        • On the App Service side, ss -tnp shows CLOSE_WAIT/FIN_WAIT_2.
        • On the HCM host and on‑prem server, use netstat/ss to see the corresponding states. If the on‑prem server shows many ESTABLISHED or CLOSE_WAIT connections that never fully close, that indicates the FIN/RST is not flowing correctly through the relay path.
      5. Application-level mitigations
        • Ensure sockets are disposed deterministically and not just left to GC. In the sample code, using is correct; also consider explicitly calling Shutdown(SocketShutdown.Both) before Dispose to force a clean FIN from the application side.
        • For high‑volume scenarios, implement connection pooling or reuse where possible to reduce churn and the number of TCP closes.
    5. When to escalate
      • Because Hybrid Connections terminate TCP and tunnel over Azure Relay, and the behavior is reproducible with minimal code, this is likely a platform‑level issue or an interaction with HCM or network devices.
      • The Hybrid Connections documentation explicitly directs support questions for App Service Hybrid Connections and HCM to the dedicated support channel. For persistent CLOSE_WAIT/FIN_WAIT_2 accumulation that cannot be explained by application behavior or on‑prem firewalls, open a support case with the Hybrid Connections team so they can inspect relay/HCM logs and behavior for the specific namespace and connections.

    In summary, the CLOSE_WAIT/FIN_WAIT_2 accumulation is consistent with incomplete TCP close handshakes across the Hybrid Connection relay path rather than a bug in the sample socket code. Validate DNS and network path from HCM to the on‑prem server, confirm that no proxy/firewall is interfering with FIN/RST, consider explicit Shutdown in the client code, and then engage the Hybrid Connections support channel with your reproducible sample and connection details so the relay/HCM behavior can be investigated.

    References:

    AI-generated content may be incorrect. Read our transparency notes for more information.

    Was this answer helpful?

Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.