SNI Extension Server Name in IP Address is Rejected

Question

Hi,

We have 2 Windows Server 2016 installation. In one installation, when a client passed an IP address to the SNI extension, the SSL exchange can proceeds. However, in another installation, if we pass an IP address, the server rejects so we can only go so far to Client Hello. All bindings have no SNI settings and both server can proceed if the SNI extension is left out.

Does anyone knows why the the other installation rejects the connection if the SNI server name has an IP address?

Thanks.

Answer 1

Hi,
Have you tried to switch to using hostnames?
Check if firewall turned off.
Perform a clean boot and disable security software temporarily to check.
https://support.microsoft.com/en-us/help/929135/how-to-perform-a-clean-boot-in-windows
Best Regards,
Carl

Answer 2

Hi,

It has nothing to do with FW because we can see from Wireshark that TCP connection can go through. In the installation where the server rejects the IP address in the SNI, it issues a tcp RST,ACK after Client Hello. But if I pass a FQDN in the SNI, the SSL exchange completes.

I understand that passing a literal IP address in the SNI is a violation of SSL protocol so rejecting this connection maybe the right behavior . What I don't understand is why a different server with the same version of Windows Server would allow this.

Regards.

Al