Encountering "The client and server cannot communicate, because they do not possess a common algorithm" or SEC_E_ALGORITHM_MISMATCH (0x80090331)

Typically when a Client and Server communicate via SSL/TLS, they must use a common cipher algorithm.  If they are unable to find a common algorithm, the SSL/TLS connection will fail with the exception, "The client and server cannot communicate, because they do not possess a common algorithm" using the SSLStream class.  Internally, SSLStream calls AcquireCredentialsHandle() which fails with SEC_E_ALGORITHM_MISMATCH (0x80090331).

I recently encountered the following issue.  My customer was trying to use one of the following ciphers between a client & server:

  • TLS_RSA_WITH_AES_128_CBC_SHA256
  • TLS_RSA_WITH_AES_256_CBC_SHA256
  • TLS_RSA_WITH_AES_128_CBC_SHA
  • TLS_RSA_WITH_AES_256_CBC_SHA

When attempting to use any of the above ciphers, the server was failing.  If they used any of the following algorithms, it worked fine:

  • TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
  • TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
  • TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA

It turns out you can encounter this error (SEC_E_ALGORITHM_MISMATCH) if the server’s certificate KeySpec poperty is set to 2 (AT_SIGNATURE).   A certificate with this KeySpec is only good for signing.  This is why the (EC)DHE Key Exchange Algorithms worked.

The cipher suites of choice using the RSA key exchange is for the KeySpec to be set to 1 (AT_KEYEXCHANGE). 

Once this change was made, the server was able to use the first 4 specified algorithms.

Follow us on Twitter, www.twitter.com/WindowsSDK.