How to configure mutual (two way) TLS 1.2 handshake to support large messages (50kb+)?

Question

I am hosting webservice implemented with WCF with two way TLS 1.2. We have a problem when client sends request that is ~50KB in size, handshake is never completed and TCP timeout closes the connection after 2 minutes.

When message is smaller, ~40kb, everything is good, handshake is completed. In case when it is ~50kb I can see that server never receive client certificate.

There is no errors or warning in Schannel events or any alert messages in wireshark. It just seems when message is ~50kb it takes too long for handshake to be completed. (and I thought that payload does not affect the handshake proccess) After the TCP timeout in svc log it says that the client certificate could not be found or validated.

Again, certificates are good because it is all successful when message is smaller.

Answer 1

Hi.

The issue you are facing might be due to network connectivity issues between the client and the server. The server might be experiencing high CPU usage or thread pool starvation, which can cause the server to drop the connection. You can check the thread count to see if there are any spikes at that time. If there are no spikes, you can use code to detect thread pool starvation. You can also dump the memory and analyze the call stack to find the root cause of thread pool starvation.

Another possibility is that the OpenTimeout value might be too small. You can check if the OpenTimeout value is less than about 20 seconds. If it is, you can increase the OpenTimeout value to resolve the issue.

It is also possible that the client certificate could not be found or validated. You can check if the Communication Services token is valid and not expired.

Hope the information is helpful.

Best Regards,

Hania Lian

============================================
If the Answer is helpful, please click "Accept Answer" and upvote it.