AD FS is leveraging SNI from TLS. The hostname used to established the TLS tunnel has to match the ADFS farm name that you can see in the administrative console and in the output of Get-ADFSProperties
(it is the name you pick at the installation).
To have the server listen on another hostname, you need to add that name to the HTTP bindings. You can use the following command on the ADFS server:
netsh http add sslcert ipport=adfs.testdomain.com:443 certhash=<hash of the TLS cert> appid={5d89a20c-beab-4389-9447-324788eb944a}
You need to replace <hash of the TLS cert> with the actual hash.
{5d89a20c-beab-4389-9447-324788eb944a} is the App Id of ADFS.
If you are also using a WAP, the following command can be ran on the WAP:
netsh http add sslcert ipport=adfs.testdomain.com:443 certhash=<hash of the TLS cert> appid={f955c070-e044-456c-ac00-e9e4275b3f04}
{f955c070-e044-456c-ac00-e9e4275b3f04} is the APp Id of WAP.