Poor Man’s Guide to Troubleshooting TLS Failures
Network security has never been more of a hot topic than it is now. There are many different driving forces making network security an ever increasing topic for discussion and review. Network security using Secure Sockets Layer (SSL) or Transport Layer Security (TLS) are particularly focused on since they are layer of network security which can traverse a corporate on premise network and provide security from that network to a cloud hosted service. Or online shopping ;)
In today’s post we will be focusing on how to determine
- what version of SSL or TLS is occurring in a network trace
- Quickly identifying which is the client and which is the server in the interaction
- How to find an RFC compliant TLS error if present in a network trace
- How to find what the client and server are exchanging in the TLS negotiation with respect to certificates
- How to find what the client and server are exchanging in the TLS negotiation with respect to supported ciphers
- How to view unexpected TLS session shutdowns from client or server
This blog post will make assumptions about a basic understanding of SSL and TLS. That may seem like a huge base of knowledge (and it is!) but you can apply Occam’s Razor as a framework to making things understandable: there’s a client and a server and they must negotiate the best mutually supported security for the session. That’s it in a nutshell for most network security and authentication and everything else is just filling in the blanks.
MSDN has a detailed explanation of how SSL and TLS work at this link.
Getting a network capture
Though I love the WireShark tool like a long lost brother I will be talking about using NetMon 3.x for the purposes of this blog post. I’ve used Message Analyzer a bit and I suspect the same filters and techniques will work with it as well.
If you have NetMon installed then getting a capture is very easy. What many don’t know is that getting a network capture on a Windows computer even without NetMon is easy and scriptable. NetSh.exe, the “Swiss Army knife” of Windows networking, can be used to collect a network trace. More on NetSh.exe can be found on TechNet at this link.
The command I usually use is (at an elevated command prompt):
netsh trace start tracefile=<filenameandpath> capture=yes maxsize=200 filemode=circular overwrite=yes report=no
If you want to include this in a PowerShell script the command translates to (at an elevated PowerShell prompt):
$NetCapFile = $env:SystemRoot + '\temp\netcap.etl'
$StartNetCap = "netsh trace start traceFile=" + $NetCapFile + " capture=yes maxsize=200 filemode=circular overwrite=yes report=no"
Locating SSL or TLS in Network Trace
This is the easiest of filters in NetMon. For SSL of any version the filter is simply “ssl” (without the quotes) and for TLS it is “tls” (also without quotes).
Finding the SSL or TLS Version Used
Finding the SSL or TLS version is very easy since it is exchanged in the Client and Server Hellos. Just expand the packet to view using the NetMon parsers. Keep in mind that this is negotiated between client and server.
Determining Client and Server in Trace
This is the second easiest filter you will use when looking at SSL or TLS. If you filter for SSLA or TLS you can identify the client who is initiating the secure session because that is the computer which is the source of the “Client Hello” message. That’s the quick and easy way to understand who is initiating the conversation and who is the server side of the session, every time.
If you want to get an idea of all of the client connections by filtering just for the Client Hello messages you can apply the filter
TLS.TlsRecLayer.TlsRecordLayer.SSLHandshake.HandShake.ClientHello //TLS 1.x Client Hello filter
Finding SSL and TLS Negotiation Errors
SSL and TLS have a set client to server exchange where how the secure session will take place is ironed out and mutually agreed upon. This negotiation can go wrong for various reasons and the client or the server are allowed to send an error message to the other side of the conversation detailing that things went wrong. In SSL/TLS parlance this is known as an “alert” message. It invariably means that something went wrong. An example would be that a server side (Server Auth) certificate may be expired, or not trusted by the client, and the result is that the client would send a TLS alert message to the server. Servers can send the requesting clients TLS alerts for a variety of reasons as well.
This capture filter will only any TLS errors in the capture so you can quickly see if any are present at all…
TLS.TlsRecLayer.TlsRecordLayer.ContentType== 0x15 //This filter will show TLS Alerts
Viewing Exchanged Certificates
SSL and TLS rely on Public Key Infrastructure (PKI)-in other words certificates and specifically certificates which have the Server Authentication usage (identified by OID 1.3.6.1.5.5.7.3.1) or the Client Authentication Usage (identified by the OID 1.3.6.1.5.5.7.3.2). The server side application decides whether both client and server or only the server side certificate is required.
If the certificate’s issuer is not trusted, or a specified URI in the certificate cannot be checked, or it is expired or not yet valid then the certificate cannot be used. SSL or TLS will fail at that point.
During the SSL/TLS session setup the certificates are exchanged. The server will offer its certificate in the Server Hello, and the client will offer it’s certificate if the server tells it that it is needed. To find those certificates you can use the filter below
TLS.TlsRecLayer.TlsRecordLayer.SSLHandshake.HandShake.HandShakeType == 0xb //This filter will show packets which contain certificates exchanged in TLS negotiation
The things which you can see about the certificate are really helpful to tell you what is actually being offered by the client and server applications:
- Subject Name
- Version (usually indicative of the template version the certificate was initially based on)
- Serial Number
- Issuer (certificate authority which issued the certificate)
- Validity (NotAfter and NotBefore times)
- Public Key Info (cipher)
- Extensions like Subject Alternate Name (SAN), CRL Distribution Points, Key Usage and more.
Determining the Supported Ciphers
With the added attention to transport layer security comes more attention to the ciphers used by the client and server for the security. Not all TLS servers and clients support the same ciphers and it is possible to see incompatibilities in TLS negotiation. This is true within the Windows versions from over the years, but is equally true when looking at network captures between other operating systems or network devices and Windows clients or servers.
In network captures we can filter for specific ciphers if we know that they may be problematic for either the client or the server. When doing so you simply add the hexadecimal value of the offered cipher to the filter below.
The hexadecimal values are the right hand numbers above. For example:
TLS.TlsRecLayer.TlsRecordLayer.SSLHandshake.HandShake.ClientHello.TLSCipherSuites.Cipher == <hexadecimal cipher value>
In support we talked to some customers who saw issues related to newer ciphers after they installed the Microsoft security fix MS14-066 (https://technet.microsoft.com/en-us/library/security/ms14-066.aspx). Some customers reported performance related concerns on the servers or even failed TLS session setups. This was by no means universally saw by all customers in all scenarios but was a complex scenario-specific thing in each and every case.
One of the more difficult things to do is to identify whether that is even a potentially related concern when troubleshooting. To quickly rule in or rule out whether those newer ciphers are being used simply add both lines below to a filter with an “or” between them:
TLS.TlsRecLayer.TlsRecordLayer.SSLHandshake.HandShake.ClientHello.TLSCipherSuites.Cipher == 0xc014 //Filter to find TLS Client Hello’s which are offering ECDHE_RSA_WITH_AES_256_CBC_SHA as an available cipher
TLS.TlsRecLayer.TlsRecordLayer.SSLHandshake.HandShake.ClientHello.TLSCipherSuites.Cipher == 0xc013 //Filter to find TLS Client Hello’s which are offering ECDHE_RSA_WITH_AES_128_CBC_SHA as an available cipher
TLS.TlsRecLayer.TlsRecordLayer.SSLHandshake.HandShake.ClientHello.TLSCipherSuites.Cipher == 0xc0A //Filter to find TLS Client Hello’s which are offering ECDHE_ECDSA_WITH_AES_256_CBC_SHA as an available cipher
TLS.TlsRecLayer.TlsRecordLayer.SSLHandshake.HandShake.ClientHello.TLSCipherSuites.Cipher == 0xc09 //Filter to find TLS Client Hello’s which are offering ECDHE_ECDSA_WITH_AES_256_CBC_SHA as an available cipher
Does simply finding these ciphers in the TLS Client Hello mean that there is a problem? No it doesn't. (I put that in bold to make sure it is not missed). Things can work perfectly well with these advanced ciphers in place, and in fact they can offer better security.
There are some additional techniques which you can apply to find out whether those newer ciphers being present are leading to the server side of the negotiation to close or reset the connection before any real data can be transmitted.
First filter for the TLS traffic you are concerned about. If the capture was taken using the NetSH.exe commands earlier in this blog post and it was taken on the client side you can identify your application in a column named “UTProcessName”. An example of one for Outlook.exe is
UTProcessName == "OUTLOOK.EXE (3014)" //filter for process
Keep in mind that the process ID number above (3014) changes client to client and reboot to reboot. Also, if you filter only for that then you may miss some of the conversation since not all network traffic for any network exchange “goes in or out” of a single process. However, it is a very useful filter to know of if you need to tune into a specific application and how it is behaving on the wire.
If the server is perhaps closing the TLS session with a TCP reset after negotiation-without failing with a TLS alert-then you can help tune into that with a filter like this one, where we are looking for server side TCP resets and showing TLS negotiations as well.
TCP.Flags.Reset == 0x1 and ipv4.address==<IP of Server side> or TLS.TlsRecLayer.TlsRecordLayer.SSLHandshake.HandShake
Keep in mind that TCP resets should always be expected at some point as the client closes out the session to the server. However, if there are a high volume of TCP resets with little or no “Application Data” (traffic which contains the encapsulated encrypted data between client and server) then you likely have a problem. Particularly if the server side is resetting the connection as opposed to the client.
Also, remember that you can quickly find what the IP address of the client and the server in the TLS exchange are by simply filtering for Client Hellos.
Finding the Unexpected
It is very common for people on the Identity support team to be engaged with a vague problem description that is thought to be related to SSL or TLS. It is truly a Monty Python-esque “rumors and portents of things going on” ambiguity where the people asking for us to look for “problems” aren't really sure even if the reason they are asking can be remotely related to SSL or TLS concerns.
The right thing to do is simply look and see what is in the network traces.
Sometimes we discover that the connection is not even using SSL or TLS. Conversely, sometimes we are engaged on a general problem with authentication only to discover that the failure is SSL/TLS related. They key is to approach it with an open mind and simply see what happens on the wire during the failure.
I usually start with the filter captures discussed in this blog post and simply go from there. If I do not find any TLS alerts then I “widen the net” to look for how the TLS sessions are setting up (what version, what ciphers, what certificates) and contextually where the session setups fit in with the client to server scenario.
If nothing SSL or TLS related stand out then I simply select the built in “authentication filter” like below and apply it to start looking for things unrelated to SSL and TLS.
This allows us to quickly look for anything related to authentication between a client and a server including NTLM, Kerberos and transport layer protocols which are authenticated using them.
Thanks for reading and I hope the “Poor Man’s Guide…” helps you better understand and perhaps even fix transport layer security on your network.
Comments
- Anonymous
February 23, 2015
Thanks