Sounds like the certificate they are using for SSL is invalid. It is possible they gave you a self-signed cert which isn't going to work for a hosted site. Or the cert they created wasn't created to support SSL. You need to go back to the vendor and ask them to create a valid SSL cert and associate it with their site. Certs are normally created by a certificate authority like Verisign or the like. For testing purposes you can generate a free cert using something like openssl but you wouldn't want to go live with that kind of cert.
You said the default SSL site works but are you using the same SSL cert for both the default site and whatever this vendor site is? Note that for internal testing you could also use your companies SSL cert (if you have one) provided the SSL cert is a wildcard cert (e.g. *.mycompany.org) and the site you're trying to host is a subdomain (e.g. myapp.mycompany.org).
Also note that when you create an SSL cert you generally specify the domain name (or perhaps a wildcard). If the actual host name used by the IIS site doesn't match the SSL cert then it'll trigger at least a warning in the browser.