TLS Connection Resets, FIPS, and a Catch-22

Question

This is as much of a heads up as it is a question, but thought I would post in the hopes that it might save someone else the same frustration. That said, if anyone has the answer to the problem outlined below, I'd be very interested in hearing the answer.

I ran across a strange issue while troubleshooting a new SharePoint 2019 web server deployment. After installing the IIS and SharePoint components, we were unable to connect to the server via HTTPS. We ran through the typical troubleshooting processes: verifying certificates had private keys, checking the cipher suite configuration, etc. A network capture showed the Client Hello, immediately followed by a TCP RESET from the server:

Interestingly, HTTPS sessions were successful using Internet Explorer. Again, looking at the network capture, one of the primary differences noted was that the IE 11 session did not include any GREASE extension in the Client Hello, whereas all of the modern browsers tested (Chrome, Edge, Firefox) did. Turning up the SChannel logging to the maximum level (7) in the registry, the server's system logs showed errors with Event ID 36888 and cited a fatal error code of 50 (decode_error). There were no other IIS or error logs to examine, as the server simply rejected the session.

After combing through the policies applied to the server, the FIPS setting was identified as being the root cause of the problem. This was verified on a vanilla Server 2019 install running IIS as well. If you have the following registry key value set to disable FIPS, TLS handshakes will immediately fail when trying to connect with a modern browser:
Path - HKLM\SYSTEM\CurrentControlSet\Control\LSA\FipsAlgorithmPolicy
Key - Enabled
Value: 0

Setting the value to 1 to enable FIPS and rebooting fixed the TLS handshake issue. The problem is, SharePoint 2019 specifically states that FIPS must be disabled because there are components that are not FIPS compliant. After enabling FIPS, we noted exception errors when trying to manage SharePoint that read, "This implementation is not part of the Windows Platform FIPS validated cryptographic algorithms". We have already opened a ticket with MS, because this seems to be a catch-22. Either TLS works or SharePoint works, but we can't have both.

I don't know why the server immediately rejects TLS handshakes with FIPS disabled, but I suspect it has something to do with the GREASE extension in the Client Hello. The fatal SChannel error code aligns with this theory, as do online articles stating non-compliant servers will terminate TLS handshakes if unable to process or ignore unknown extensions (like GREASE).

Answer 1

Hello there,

Your suspect GREASE extension can be an man-in-the-middle (MITM) that might have caused this Havoc.

It could be something like a network firewall preventing the connection, or it could be a configuration on an edge device on the server-side network. So, this issue can actually be either a client- or server-side fix depending on the scenario.

In the below article you can find 5 handy tools that can test different phases of SSL/TLS connection so that you can narrow down the cause of SSL/TLS connection issue and locate root cause.

SSL/TLS connection issue troubleshooting test tools https://techcommunity.microsoft.com/t5/azure-paas-blog/ssl-tls-connection-issue-troubleshooting-test-tools/ba-p/2240059

I hope this information helps. If you have any questions please let me know and I will be glad to help you out.

----------------------------------------------------------------------------------------------------------------------------------------------

--If the reply is helpful, please Upvote and Accept it as an answer--

Answer 2

I discovered there were registry keys present to enable TLS 1.3 for both client and server. I removed these, disabled FIPS, rebooted, and was able to verify that TLS handshakes were successful. I think configuring the registry to support TLS 1.3 and disabling FIPS at the same time was the root cause.