Troubleshooting TLS / SSL communication problems for ASP.NET applications making HTTP Web Request or WCF queries to SSL endpoints – Scenario 2

This is the second part of our series of articles about troubleshooting TLS / SSL communications problems when you make Http Web Request or WCF queries from your ASP.NET applications to SSL endpoints.

In our first scenario, we troubleshooted a "The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel" error message. We made some basic tests, such as a "browser test" and found that the certificate used was not valid. However fixing the certificate did not solve the issue and we still see the same error message telling us that the existing connection is forcibly closed by the remote host when we run our ASP.NET application which makes an HTTP web request to https://iis85.buggybits.com/ URL.

About "browser tests"...

Although a browser can be used to test the communication, this may not be the safest way to test each and every scenario because this test will use current user's certificate store. When ASP.NET application is used it will use the computer account's store. This difference may cause different behaviors.

Also, when testing with browser, we rely on the browser TLS negotiation settings and its choices during the SSL handshake. When it is an ASP.NET application that runs as a client, depending on the .NET configuration, TLS negotiation in the SSL handshake may behave differently.

A few words about SSL handshake

SSL handshake is explained greatly here , so I will not give all the detail of it and I suggest you to read it if your unfamiliar with it. Basically, after the TCP handshake;

• Client sends a CLIENT HELLO package to the server and it includes the SSL / TLS versions and the cipher suites it supports.
• Then the server responds with a SERVER HELLO package which includes the SSL / TLS versions and the cipher suits that it supports.
• If the client sends a TLS version lower than the server supports the negotiation fails.
• If the server responds with a lower TLS version and if the client supports that TLS version, SSL handshake continues with that TLS version. This is called TLS fallback. For example, if the client supports both TLS 1.0 and TLS 1.2, and the server supports only TLS 1.0, the SSL handshake may start with TLS 1.2 by client, and then it may actually happen in TLS 1.0 when server replies with "I support TLS 1.0 and let's continue with that" message.
• Cipher suite negotiation also happens here.

Now it is a good time to capture a network trace from the machine where the client ASP.net application is hosted on. It is always good practice to capture the network trace from both client and the server, even from the intetmediate devices or servers, such as proxies, where possible. Having multiple network traces collected from different places along the transit could help analyze the issue easier because you can check if the same traffic / package from the client machine arrives at the remote server without a problem. In our case it is OK to capture and take a look at the trace from client machine as we are currently trying to understand if TCP and SSL handshake is done sucessfully.

I personally choose Network Monitor but some of you may prefer Wireshark over Network Monitor. It is up to you to choose the tool to capture and analyze a network trace.

You can see a screenshot of the network trace I captured while I reproduce the same problem:

(Please note that the TMG is just the name of the server, and not the "Threat Management Gateway", sorry for the confusion)

As seen in the filtered trace above, the TCP handshake is successfull (the first three frames). Then the client sends the "Client Hello" in the fourth frame (frame # 818) but instead of a "Server Hello" reply it receives a RESET package from the server as seen in the sixth frame (frame # 822) – you can see the R flag, which stands for TCP RST (reset) is indicated on the frame. So SSL handshake is failing here.

We know that the TLS and cipher suites are negotiated in the SSL handshake so those are the first things to check. If we look at the Client Hello details in the trace we see that the client sends TLS 1.0 request to the server:

This is interesting because the machine runs the ASP.NET application supports TLS 1.2 and we can confirm in the registry at HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols (ref.: /en-us/windows-server/identity/ad-fs/operations/manage-ssl-protocols-in-ad-fs):

As seen in the screenshot above, TLS 1.2 is enabled as both client and server. If we check the other TLS protocols we confirm that TLS 1.0, 1.1 and 1.2 are all enabled.

Here is a theory comes at this point: our ASP.NET client tries communicating via TLS 1.0 but the remote server does not support TLS 1.0. How can we confirm our theory? Remember that we have a working scenario, if we browse the same URL via Internet Explorer, the page loads fine. So let's take a network trace from the working scenario and see what the client requests and what the server responds:

(Please note that the TMG is just the name of the server, and not the "Threat Management Gateway", sorry for the confusion)

As seen above, there is Client Hello and server response with Server Hello and the certificate without its private key then the handshake suceeds. Let's take a closer look at the Client Hello (frame # 256) first:

This time, in sucessfull scenario with Ineternet Explorer, we are seeing that the client starts negotiation with TLS 1.2, actually it is the strongest version of the TLS supported on the machine.

Let's take a look at what Server Hello tells the client:

Bingo! Server supports TLS 1.2. Then it is expected to fail if ASP.NET client requests to use TLS 1.0 which is not supported by the server.

IE just works fine because it uses the same TLS version with server but it is interesting to see that the ASP.NET client goes with TLS 1.0, instead of stronger version of TLS 1.2, even both IE and the ASP.NET client runs on the same server.

A quick word about Internet Explorer's TLS settings

If you open Internet Explorer's advanced settings you can configure which SSL and TLS versions that Internet Explorer can use:

So Internet Explorer uses the strongest version of the protocol. Then why does not ASP.NET use the strongest one? Here comes a security update in the picture:

Microsoft Security Advisory 2960358 - Update for Disabling RC4 in .NET TLS
https://technet.microsoft.com/en-us/library/security/2960358.aspx

This update disables the RC4 and SSL 3.0 for .NET Framework and there are updates for different .NET Framework versions. As described in the article you can use the following registry key to force the usage of the strongest TLS version:

For 32-bit applications on 32-bit systems and 64-bit applications on x64-based systems:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\.NETFramework\v4.0.30319
"SchUseStrongCrypto"=dword:00000001

For 32-bit applications on x64-based systems:

HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\.NETFramework\v4.0.30319
"SchUseStrongCrypto"=dword:00000001

Since we are using .NET 4.6.x, instead of setting the registry key we can just set targetFramework=4.6 in the web.config file to force using the strongest TLS version:

<compilation targetFramework="4.6" /> <httpRuntime targetFramework="4.6" />

After setting the targetFramework in my application the TLS version negotiation started to work as expected, as confirmed in the network trace screenshot below:

(Please note that the TMG is just the name of the server, and not the "Threat Management Gateway", sorry for the confusion)

Now if you check the Client Hello and Server Hello details you would see that TLS negotiation is suceeded as both are using TLS 1.2:

I would like to say "we nailed it" but no, not at all. It is still not working. If you look at closely to the network trace, this time client is sending a FIN package and closing the TCP connection (frame # 185). Why is that? Let's analyze it in our last scenario.

Happy troubleshooting...