SSL Handshake Timeout with Azure DevOps Server via SOCKS5 Proxy

Question

                                                                                                                                                                                                               
  Problem Summary                                                                                                                                                                                              

  We're experiencing SSL handshake timeouts when attempting to connect to Azure DevOps Server through a SOCKS5 proxy tunnel. The connection times out after 10 seconds during the TLS handshake
  phase.

  Environment

  - Azure DevOps Server Version: 2022.2 (Build: AzureDevopsServer_20240806.7)
  - Java Version: 21
  - Spring Boot Version:4.0.6
  - HTTP Client: Netty-based (WebClient/Reactor Netty)
  - Proxy Type: SOCKS5 (via Teleport tunnel)

  Error Details

caused by: io.netty.handler.ssl.SslHandshakeTimeoutException: handshake timed out after 10000ms
at io.netty.handler.ssl.SslHandler.lambda$applyHandshakeTimeout$6(SslHandler.java:2257)
at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:160)
at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:148)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:141)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:535)
at io.netty.channel.SingleThreadIoEventLoop.run(SingleThreadIoEventLoop.java:201)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:1195)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Unknown Source)

  What We're Trying to Do

  Connect to an Azure DevOps Server instance through socks5 to do an "connection Data" api call.
  

  What We've Tried

  1.Increased SSL handshake timeout from default (10s → 30s, 60s) - No improvement
  2.Verified SOCKS5 tunnel is active and port is reachable
  3.Direct connection (without proxy) works fine
  

  Configuration

  // WebClient configuration with SOCKS5 proxy
  HttpClient httpClient = HttpClient.create()
      .proxy(proxy -> proxy
          .type(ProxyProvider.Proxy.SOCKS5)
          .host("localhost")
          .port(8888))
      .secure(spec -> spec.handshakeTimeout(Duration.ofSeconds(60)));

  Questions

  1. Is HTTPS over SOCKS5 supported for Azure DevOps Server 2022.2?
    - Does Azure DevOps Server have any known issues with proxied SSL/TLS connections?
  2. Are there specific TLS/SSL requirements for Azure DevOps Server?
    - Required TLS versions (1.2, 1.3)?
    - Required cipher suites?
    - Client certificate requirements?
  3. Has anyone successfully connected to Azure DevOps Server through SOCKS5 proxy?
    - What configuration worked?
    - Any special proxy settings needed?

  Additional Context

  - The same SOCKS5 proxy setup works perfectly with other on-prem SCM providers (GitHub Enterprise Server, GitLab Self-Managed)
  - Direct HTTPS connection to Azure DevOps Server (bypassing proxy) works without issues
  - The handshake timeout occurs consistently at exactly 10 seconds, even after increasing the configured timeout

Answer 1

Hello Akshay C S,

Thank you for reaching out to the Microsoft Q&A platform.

Based on the details and stack trace provided, the issue is not with Azure DevOps Server, but rather with how Reactor Netty handles timeouts and DNS resolution during the SOCKS5 proxy connection phase.

The fact that the timeout occurs at exactly 10 seconds, despite you increasing the handshakeTimeout to 60 seconds, indicates that the default channel connection timeout is being hit. In Reactor Netty, the default connection timeout (ChannelOption.CONNECT_TIMEOUT_MILLIS) is 10 seconds (10000ms). This timeout covers the TCP connect and the proxy connection steps. If the tunnel takes longer than 10 seconds to establish, it fails before your 60-second SSL handshake timeout logic is fully applied.

To directly answer your questions:

1. Is HTTPS over SOCKS5 supported for Azure DevOps Server 2022.2?
Yes, it is fully supported. Azure DevOps Server operates at the application and transport layers (HTTPS/TLS) and is completely unaware of the client-side SOCKS5 tunnel. There are no DevOps-specific limitations for proxied SSL/TLS connections.

2. Are there specific TLS/SSL requirements for Azure DevOps Server?

• TLS Versions: As per Microsoft standards, Azure DevOps Server 2022 defaults to requiring TLS 1.2. Older protocols (TLS 1.0/1.1) are disabled by default.
• Cipher Suites: Standard FIPS and non-FIPS cipher suites for TLS 1.2 are supported. The JVM default cipher suites are perfectly sufficient.
• Client Certificates: Unless explicitly configured by your server administrator for mutual TLS (mTLS), client certificates are not required.

Possible Resolution: To resolve this, you need to increase the Netty channel connection timeout and defer DNS resolution to the proxy. If Netty tries to resolve the Azure DevOps Server hostname locally before routing through the proxy, it can cause routing failures or timeouts.

Please update your HttpClient configuration as follows:

import io.netty.channel.ChannelOption;
import io.netty.resolver.NoopAddressResolverGroup;
import reactor.netty.http.client.HttpClient;
import reactor.netty.transport.ProxyProvider;
import java.time.Duration;

HttpClient httpClient = HttpClient.create()
  // 1. Increase TCP Connect and Proxy Connect timeout to 60 seconds
  .option(ChannelOption.CONNECT_TIMEOUT_MILLIS, 60000)
  // 2. Defer DNS resolution to the SOCKS5 proxy
  .resolver(NoopAddressResolverGroup.INSTANCE)
  .proxy(proxy -> proxy
    .type(ProxyProvider.Proxy.SOCKS5)
    .host("localhost")
    .port(8888))
  // 3. Keep your existing SSL handshake timeout
  .secure(ssl -> ssl.handshakeTimeout(Duration.ofSeconds(60)));

Note: If you ever transition to using the Azure SDK's client builder, you can achieve the same using the NettyAsyncHttpClientBuilder.

References:

• NettyAsyncHttpClientBuilder Class
• Azure DevOps Pipelines - Certificates

Please try this configuration and let me know if it resolves the handshake timeout issue.

Note: This response is drafted with the help of AI systems.

Answer 2

Hello Akshay C S

Greetings! Thanks for raising this question in Q&A forum.

The fact that direct HTTPS works fine but it fails consistently at exactly 10 seconds through SOCKS5 even after raising your handshake timeout to 60s strongly suggests the timeout isn't really coming from your Netty/WebClient config at all. Something in the proxy path (the Teleport SOCKS5 tunnel itself, or an intermediate component) is enforcing its own 10-second timeout and silently dropping the connection before your TLS handshake can complete, so Azure DevOps Server never even sees the handshake attempt.

Here's how to narrow this down step by step:

Step 1: Confirm Whether TCP Connect Succeeds Through the Proxy Use a simple tool like curl with the same SOCKS5 proxy to test raw connectivity:

curl -v --socks5 localhost:8888 https://<your-devops-server>:443

If this also hangs at ~10 seconds, it confirms the issue is at the tunnel/proxy level, not in your Java application code.

Step 2: Test with OpenSSL Directly Through the Proxy Run a manual TLS handshake test:

openssl s_client -connect <your-devops-server>:443 -proxy localhost:8888

This bypasses Netty entirely. If the handshake still times out at 10s, your application code (and the 60s timeout setting) is not the bottleneck — the proxy layer is.

Step 3: Check the Teleport Tunnel's Own Timeout Settings Teleport proxies often have their own connection/idle timeouts independent of your client configuration (commonly defaulting to 10s for connection establishment). Check the Teleport configuration (teleport.yaml) for settings like idle_timeout or connection-level timeouts under the app/proxy service section, and increase them.

Step 4: Compare TLS Versions Negotiated Since GitHub Enterprise Server and GitLab work fine through the same tunnel, capture a packet trace (Wireshark) for both a working connection (GitHub/GitLab) and the failing Azure DevOps Server connection. Compare:

• Does the ClientHello even reach the server in both cases?
• Is there a difference in TLS version offered (1.2 vs 1.3) or cipher suite list?

If the ClientHello never leaves the proxy for Azure DevOps Server but does for GitHub/GitLab, that points to a routing or SNI-based filtering issue in the tunnel rather than a TLS compatibility issue with Azure DevOps Server itself.

Step 5: Check Azure DevOps Server's TLS Configuration On the Azure DevOps Server 2022.2 host (typically running on IIS), confirm:

• TLS 1.2 is enabled (TLS 1.3 support depends on the underlying Windows Server/.NET version — Azure DevOps Server 2022 primarily expects TLS 1.2)
• The SSL certificate binding is correctly configured on the IIS site
• No client certificate is being requested (since GitHub/GitLab through the same proxy don't require one, but DevOps Server might be configured differently)

Step 6: Test SNI Handling Some SOCKS5 proxies/tunnels don't properly pass through the SNI (Server Name Indication) field during the TLS handshake for certain destination hosts. If Azure DevOps Server's IIS binding relies on SNI to select the correct certificate, and the tunnel mishandles SNI for this host specifically, the server may simply never respond — leading to a client-side timeout.

Step 7: Try an Alternate Proxy Type as a Diagnostic As a test, try routing through an HTTP CONNECT proxy instead of SOCKS5 (if available in your environment) to the same Azure DevOps Server endpoint. If HTTP CONNECT works but SOCKS5 doesn't, this strongly confirms a SOCKS5-tunnel-specific issue rather than anything on the Azure DevOps Server side.

To directly answer your questions: Azure DevOps Server 2022.2 has no documented restriction against being accessed via SOCKS5 HTTPS over SOCKS5 is just regular TCP tunneling and is protocol-agnostic from the server's perspective. There are no special TLS requirements beyond standard TLS 1.2 support with a valid certificate binding in IIS, and no client certificate is required by default. The consistent 10-second cutoff is the strongest clue here, and it almost always points back to the proxy/tunnel layer's own timeout configuration rather than Azure DevOps Server.

If this answer helps you kindly accept the answer which will help others who have similar questions

Best Regards,

Jerald Felix.