Troubleshooting SSL Certificates in ISA Server 2004 Publishing

  • Article

Secure Sockets Layer (SSL) server certificates are commonly used in the following Microsoft Internet Security and Acceleration (ISA) Server publishing scenarios.

  • Publishing with server publishing rules. ISA Server uses server publishing to process incoming requests to internal servers. Internal servers are protected by defining a network address translation (NAT) relationship between the network on which client requests are received and the network in which the published server is located. Published IP addresses are actually those of the ISA Server computer, protecting internal resources. Server publishing does not provide the same advantages as Web publishing Hypertext Transfer Protocol (HTTP) or Secure Hypertext Transfer Protocol (HTTPS). In particular, there is no application layer traffic inspection on ISA Server. Typically, server publishing rules are used to publish protocols other than HTTP or HTTPS, such as computers running Microsoft SQL Server™. When server publishing over a secure SSL connection, you require an SSL server certificate on the published server. No SSL processing occurs on the ISA Server computer.
  • Publishing with Web publishing rules. Web publishing is the recommended method for publishing HTTP or HTTPS with ISA Server, including Microsoft Outlook® Web Access servers. Web publishing rules provide several advantages over server publishing, including the ability to decrypt and inspect HTTPS traffic before forwarding it to the published Web server, the caching of Web responses from published Web sites, client authentication on ISA Server, and Web application layer filtering to inspect HTTP and HTTPS traffic before allowing it into the networks protected by ISA Server.

Web Publishing Over an HTTPS Connection

Best Practices

Common Issues

Additional Information

Web Publishing over an HTTPS Connection

When you use secure Web publishing rules to publish an internal Web server through ISA Server, client requests for the Web server arrive at the ISA Server computer over an HTTPS connection. They are forwarded (bridged) from ISA Server to the published Web server. You can choose to:

  • Forward HTTPS requests to the published Web server over HTTP. In this scenario, ISA Server authenticates to the client making the request using an SSL server certificate. An SSL certificate is required on the ISA Server computer only.
  • Forward HTTPS requests to the published Web server over HTTPS. This is the more secure configuration, preserving HTTPS throughout the connection. In this scenario, ISA Server authenticates to the requesting client using an SSL server certificate, and the published Web server authenticates to the ISA Server computer using an SSL server certificate. A certificate is required on both the ISA Server computer and on the published Web server.

Best Practices

There are a number of points to be aware of when obtaining, installing, and configuring certificates:

  • To authenticate ISA Server to external clients, you will usually use a certificate from a commercial certification authority (CA). Because Internet Information Services (IIS) is not usually installed on ISA Server, you will request and install a certificate using the Web Server Certificate Wizard in IIS on the Web server you intend to publish, and then export it to ISA Server. Currently, there is no way to request a server certificate from ISA Server to a commercial CA directly. For detailed information about obtaining and configuring SSL server certificates in publishing scenarios, see Digital Certificates for ISA Server 2004. This document also applies to ISA Server 2006.
  • When you specify a common name in the certificate request, there are certain naming guidelines to follow:
    • The name of the certificate on the ISA Server computer must match the name that external clients will specify to reach the published Web site. This limitation does not apply if you are using wildcard certificates. In ISA Server 2004, ISA Server supports the use of wildcard certificates on the Web listener only. The use of wildcard certificates on the backend Web server is not supported. This configuration is supported in ISA Server 2006. A wildcard certificate will specify a wildcard character in the common name, for example *.internal.net. This means that any host name in the internal.net domain is valid in the client request. For example, a client can send a request for owa.internal.net and www.internal.net. Both requests are consistent with the wildcard certificate. ISA Server only allows a single certificate to be bound for a specific Web listener. Wildcard certificates can be a useful means of publishing multiple secure Web sites using a single IP address. For more information on configuring wildcard certificates in ISA Server 2004, see Publishing Multiple Web Sites Using a Wildcard Certificate in ISA Server 2004 (www.microsoft.com).
    • In an HTTPS to HTTPS Web publishing scenario (requiring a server certificate on both the ISA Server computer and on the published Web server), the name of the certificate on the IIS Web site of the published server must match the name by which ISA Server identifies the Web server. (This is the name specified on the To tab of the Web publishing rule properties.)
  • Between creating the request file and installing the certificate, do not do any of the following:
    • Change the computer name or Web site bindings.
    • Change encryption levels (apply the high encryption pack).
    • Delete the pending certificate request.
    • Change any of the Web site’s Secure Communications properties.
  • When you export the certificate from the Web server for import to ISA Server, you must also export the private key. In the Certificate Export Wizard, if you do not select Yes, export the private key, you will not be able to bind the certificate to a Web listener when you create the Web publishing rule.
  • When you export a file from the Web server to import to the ISA Server computer, select the option to Include all certificates in the certification path if possible. This allows both the Web site certificate and any CA certificate in the hierarchy to be included in the export.
  • Server certificates must be imported under the Local Computer account and stored in the Personal certificate store of the ISA Server computer.
  • In an HTTPS to HTTPS publishing configuration (requires a certificate on both the ISA Server computer and the published Web server), you can choose to leave a copy of the certificate you exported to ISA Server on the Web server. In this case, the same SSL certificate is used to authenticate ISA Server to external clients, and the internal Web server to ISA Server. If you choose to use the same SSL certificate on the ISA Server computer and on the published Web server, you must ensure that the certificate name is configured correctly. The name of the certificate must match the name that external users will specify to reach the published site (the name on the Public Name tab of the Web publishing rule), and the name by which ISA Server refers to the internal Web server (the name on the To tab of the Web publishing rule).
  • Alternatively, you can request and install a second certificate on the Web server. You may choose to use an internal certificate in this case, rather than purchasing one from a commercial CA, because no external requests are involved. For more details about the steps required to set up an internal Certificate Server, see Digital Certificates for ISA Server 2004, and Issuing Your Own Server Certificates (www.microsoft.com).
  • Clients must trust the SSL server certificates presented to them. Server certificates are trusted by client browsers when the root certificate of the CA that issued the certificate is present in the Trusted Root Certification Authorities store on the client computer. Ensure the following:
    • The SSL server certificate sent by ISA Server to authenticate itself to external clients must be trusted by those external clients. This is usually a certificate issued by a commercial CA. By default, client browsers such as Internet Explorer have root certificates installed for common commercial CAs.
    • In an HTTPS to HTTPS scenario, ISA Server must trust the CA that issued the certificate that is used by the published Web server to authenticate to the ISA Server computer. If an internal enterprise CA issued the certificate, and ISA Server is located in the same domain as the enterprise CA, the CA root certificate will automatically be installed in the ISA Server Trusted Root Certification Authorities store. If you obtain a certificate for the Web server from an internal stand-alone CA, or ISA Server is not in the same domain as the enterprise CA, you must install the root certificate from the CA in the Trusted Root Certification Authorities store on the ISA Server computer.
    • If internal certificates are issued from an enterprise, Certificate Services, ISA Server, or the Web server must be able to contact the CA to request a certificate or obtain certificate updates using autoenrollment. Ensure that appropriate access rules are enabled between relevant networks. ISA Server must be able to resolve the host name or fully qualified domain name (FQDN) of the enterprise CA to contact it.

Common Issues

The following describes common issues that you may encounter.

When I generate a Certificate Signing Request (CSR), what do I put in the Common Name field?

The Common Name field contains the domain or server name. Do not include https:// before the name or any subfolders indicated by the / after the domain name. Do not add the port number. Valid examples are: www.mydomain.com, mydomain.com, and secure.mydomain.com.

What does this error mean? 500 Internal Server Error – The target principal name is incorrect.

This error occurs when the name in the SSL client request from ISA Server does not match the common name on the Web site certificate. Check that the certificate names follow the guidelines:

  • For the certificate on the ISA Server computer, the name must match the name that the external clients specify to reach the site.
  • For the certificate on the published Web server, the name must match the name that appears on the To tab of the rule.
  • In the case of the certificate on the Web server in a server publishing scenario, the certificate should have the name that users will use to connect to the server.

To troubleshoot, either obtain a new certificatethat matches the required name, or modify the required name to match the certificate’s common name. In addition, make sure that ISA Server can resolve the name to the IP address of the published Web site. If you modify the name on the To tab, one way to ensure that the name can be resolved is to add a Hosts file entry on the ISA Server computer (WINNT\system32\drivers\etc\hosts) to map the name and IP address of the published site.

How can I publish multiple SSL sites using the same IP address and port, with different certificates?

You can only use one SSL certificate per listener. If all sites are published using the same domain name, you can use a wildcard certificate, and then use a single IP address and a single listener to publish multiple sites. For example, if you are trying to publish the following sites: OWA, WebSite1, WebSite2 at domain.com, you can acquire a wildcard certificate for the ISA Server computer for *.domain.com.

I installed a certificate on my IIS 4.0 Web site and exported to ISA Server. When I try to select the certificate in the Web listener, there is a message that no certificates are installed.

When you export a certificate from IIS 4.0 it does not generate a file in .pfx format, and Microsoft Windows Server™ 2003 does not recognize it as a certificate. To resolve this, install IIS 6.0, and then import the certificate into IIS 6.0 from IIS 4.0. Then export the certificate from IIS 6.0 and install on ISA Server.

I am using wildcard certificates and getting the error: 500 Internet Server Error – The target principal name is incorrect .

ISA Server 2004 only supports wildcard certificates on the ISA Server computer. ISA Server 2006 also supports use of wildcard certificates on the published Web server. When using HTTPS to HTTPS bridging, you cannot use wildcard certificates to authenticate the back-end Web server. Instead, on the internal Web server, create a new certificate that matches the name of the internal Web server, as specified on the To tab in the Web publishing rule. For more information about configuring this scenario, see Publishing Multiple Web Sites using a Wildcard Certificate in ISA Server 2004 (www.microsoft.com).

I am publishing remote procedure call (RPC) over HTTPand getting the error: 500Internal Server Error – The target principal name is incorrect ,even though the name in the client request matches the name of the certificate on the ISA Server computer.

When you create a new Outlook profile, on the Connection tab of Exchange Server Settings, you click Exchange Proxy Settings to specify RPC over HTTP settings. In Use this URL to connect to my proxy server for Exchange, ensure that you have typed the same name that appears on the certificate. Select Mutually authenticate the session when connecting with SSL, and then in Principal name for proxy server, again type the name that appears on the common name of the certificate. For example, if the common name is the FQDN used by clients to reach the site, you will type it in the form msstd:common name.

If this error occurs and you are using a wildcard certificate, ensure that the Principal name for proxy server Outlook setting is defined as msstd:*.domain.com, and not server.domain.com.

I receive an error message: 500 Internal Server Error. The certificate chain was issued by an authority that is not trusted.

ISA Server must trust the certificate from the published Web server. Ensure that the CA certificate is in the ISA Server Trusted Root Certification Authorities certificate store.

When I try to create a Web listener with a certificate, I get a message: There are no certificates configured on this server. I have a certificate, so why can I not add it?

This message may also be accompanied by an event in Event Viewer indicating that the certificate private key could not be accessed. This error may occur in the following circumstances:

  • The SSL certificate and its corresponding private key were not imported to the correct certificate store on the ISA Server computer. The SSL certificate has been moved from one certificate store to another, causing the SSL certificate to separate from its corresponding private key.
  • When you exported the certificate from the Web server, you did not indicate that the private key should be exported.

Check that the private key was exported, and that the certificate was imported under the Local Computer account into the Personal store.

I receive an error message: 500 Internal Server Error. The certificate is revoked .

To help maintain the integrity of a CA public key infrastructure (PKI), CA administrators will revoke a certificate under circumstances where the certificate is no longer considered valid. When a certificate is revoked, it is added to the certificate revocation list (CRL) for that CA. CAs periodically publish an updated CRL. CRL distribution points are used to provide a certificate verifier with a location at which it can retrieve the current CRL. This error is issued when the root certificate cannot find a CRL distribution point, or when the certificate has been revoked. If you experience this issuing when using ISA Server 2004 Standard Edition, you can resolve it by installing ISA Server 2004 Standard Edition Service Pack 1. To work around this issue without SP1, manually download the CRL and install it under the Local Computer account into the Intermediate Certification Authorities store. In addition, ensure that the system policy rule to access CRL is enabled.

I want to use a certificate with multiple common names in it, for example, to reference https://server_name and https://www.server_name.com. Can ISA Server handle these multiple common names?

No, ISA Server will only reference the first common name in the certificate, and does not support multiple names.

I am trying to request a server certificate from an internal enterprise CA using the Certificates Microsoft Management Console (MMC) snap-in on the ISA Server computer, and I receive the following error: The certificate request failed because of one of the following conditions: the certificate request was submitted to a Certification Authority (CA) that was not started: you do not have the permissions to request certificates from the available CAs . The issue occurs even when the CA is started and there are sufficient permissions.

By default, ISA Server enforces strict remote procedure call (RPC) compliance on all firewall rules. To request a certificate for the ISA Server computer, you must modify firewall policy rules.

Note

Use the Certificates MMC snap-in to request a certificate from an enterprise CA when ISA Server is a member of an enterprise CA domain. Otherwise, use the Web enrollment site to request a certificate. The CA root certificate of the enterprise CA is automatically entered into the Trusted Root Certification Authorities store for all computers in the same domain as the CA. If the ISA Server is not a domain member, manually place the CA certificate into the Trusted Root Certification Authorities store on the ISA Server computer.

  • Modify the system policy rule to allow DCOM network traffic from the ISA Server to the CA, as follows: In the Firewall Policy node of ISA Server Management, click the Tasks tab, and then click Edit System Policy. In the Authentication Services group, click Active Directory. Clear the Enforce strict RPC compliance check box, and then click OK. Click Apply to save changes.
  • To obtain a certificate on a Web server behind ISA Server from a CA in a different network, modify settings on the rule (or rules) to allow DCOM traffic between the networks, as follows: In the Firewall Policy node of ISA Server Management, click the required rule, and on the Tasks tab, click Edit Selected Rule. On the access rule Protocols tab, click Filtering, and then click Configure RPC Protocol. On the Protocol tab, clear the Enforce strict RPC compliance check box. Then in the Configuration node of ISA Server Management, click Add-Ins. Right-click RPC Filter in the details pane, and then click Disable. You will have to repeat this for any other access rules that are configured between the networks.

Remember to enable all the settings again after requesting the certificate.

Other Issues

If you suspect that there is a certificate issue in a publishing scenario, you can check a number of general certificate settings. Note the following:

  • In server publishing scenarios, you will check the certificate configuration on the Web server (no certificate required on the ISA Server computer).
  • In Web publishing HTTPS to HTTP, you will check the certificate configuration on the ISA Server computer (no certificate required on the published Web server).
  • In Web publishing HTTPS to HTTPS, you will check the certificate configuration on the Web server and on the ISA Server computer.

General troubleshooting steps include the following:

  • Check that the relevant usage of the certificate is configured correctly. To do this, in the MMC Certificates snap-in, right-click the certificate, and then click Properties. Select Enable only the following purposes, and make sure that Server Authentication is selected.
  • Check the validity of the certificate on the General tab of the certificate properties.
  • Ensure that the certificate has a corresponding private key on the Details tab of the certificate properties. If it does not, you will not be able tobind it to a Web listener when you create the Web publishing rule. If it is the certificate you imported to ISA Server from the Web server, export it again fromthe Web server, and repeat the import to the ISA Server computer.
  • Ensure that there is a trust relationship defined between the installed certificate and the issuing CA. To do this, on the Certification Path of the certificate properties, you should see a hierarchical relationship, and a note that says This certificate is OK.
  • Be sure that the certificate is imported under the Local Computer account and stored in the Personal certificates store.
  • For Web publishing over an HTTPS connection, check that there is an SSL listener enabled on the ISA Server computer, and an SSL server certificate installed.
  • Check that ISA Server trusts the CA that issued the certificate used to authenticate the published Web server. To do this, open Internet Explorer on the ISAServer computer, click the Tools menu, and then click Internet Options. On the Content tab, click Certificates. Check that a certificate for the CA appears on the Trusted Root Certification Authorities tab.
  • Check that the certificate names follow the guidelines. For the certificate on the ISA Server computer, the name must match the name that the external clients specify to reach the site. For the certificate on the published Web server, the name must match the name that appears on the To tab of the rule. In the case of the certificate on the Web server in a server publishing scenario, the certificate should have the name that users will use to connect to the server.

Additional Resources

For additional information, see the following Microsoft Knowledge Base articles:

  • Article 891510 (released with SP1): Clients receive a "500 Server" error message if a Web server requires a Certificate Revocation List in ISA Server 2004
  • Article 833401: How to configure RPC over HTTP on a single server in Exchange Server 2003
  • Article 833704: "The certificate request failed because of one of the following conditions" error message when you request a certificate in ISA Server 2004
  • Article 837354: How to publish a Microsoft Exchange server for Outlook Web Access in ISA Server 2004
  • Article 841664: Clients may receive a “Error Code 500 Internal Server Error” error message if you use ISA Server 2004 to publish a Web site to a server that is on the internal network
  • Article 837834: How to publish an SSL Web site by using SSL tunneling in ISA Server 2004
  • Article 884506: How to configure ISA Server 2004 to allow for RPC over HTTP client connections from Office Outlook 2003 to Exchange Server 2003
  • Article 885186: How to publish a Web site directly on your Internet Security and Acceleration Server 2004 computer

Additional Information

Additional ISA Server 2004 documents are available on the ISA Server 2004 Guidance page (https://www.microsoft.com).