certauth.adfs.mytestdomain.com is the URL used when you do certificate based authentication on port 443 (like user certificates). It is called alternate hostname binding. Historically certificate based authentication is using a different port: 49443. But because this port is often blocked in public network, starting Windows Server 2016 we introduce the ability to do the certificate based authentication over 443 but it has to use a different name.
So you can ignore this error message if you are not using certificate based authentication. If you need it and want to use it, you need to add the Subject Alternate Name certauth.adfs.mytestdomain.com to your TLS/SSL certificate. This is explained here.
Regarding the "type" certificate it is a TLS certificate It is described here. It must be a X509 v3 certificate (CNG keys are not supported) . To update the certificate, import it on the local store of each ADFS nodes, then you need two commands on the ADFS primary server.
Set-AdfsCertificate -CertificateType "Service-Communications" -Thumbprint '<thumbprint of new cert>'
Set-AdfsSslCertificate -Thumbprint '<thumbprint of new cert>'
This is also explained here. You also need to make sure the ADFS service account has the permissions to read the private key of the certificate. If you are using the same certificate on the WAPs (recommended), you also need to run the following on your WAP servers:
Set-WebApplicationProxySslCertificate -Thumbprint '<thumbprint of new cert>'
There is nothing to do on the relying party trusts when you update this certificate.
Token-Signing and Token-Decrypting certificates should not be TLS certificates (they could but there is no value of paying for these two). You can use the default self-signed certificates for these two. Updating those two certificates will break the trust with the relying party trusts and external claim provider trusts (not the Active Directory one though). So if you change them, tell the admins of the parties to update their configuration. This is documented here. By default we use self-signed certificates and they renew automatically.
Have a look and let us know if you need more clarity!