Troubleshooting PKI Problems on Windows Vista

CryptoAPI 2.0 (CAPI2) Diagnostics is a feature in Windows Vista® and Windows Server® 2008. This feature provides administrators with the ability to troubleshoot PKI problems by collecting detailed information about certificate chain validation, certificate store operations, and signature verification. With CAPI2 Diagnostics, it is easier to identify the cause of most public key infrastructure (PKI) problems. CAPI2 Diagnostics can reduce the time required to diagnose problems and improve the troubleshooting experience. This document describes CAPI2 Diagnostics and how it can be used to troubleshoot some common PKI error scenarios.

To download a copy of this document, see https://go.microsoft.com/fwlink/?LinkID=85484.

In This Guide

• CAPI2 Overview
• Background
• CAPI2 Diagnostics in Windows Vista
• Troubleshooting Common Problems
• Error Scenarios
• General Considerations for Automated Troubleshooting
• Summary
• Feedback
• Appendices
• Further Information

CAPI2 Overview

A PKI is the combination of cryptography, software, processes, and services that enable an organization to secure its communications and business transactions. X.509 certificates can be used to identify users, devices, or organizations and to enable more secure applications, such as signed e-mail, code signing, and secure Web browsing.

CAPI is the core cryptography and X.509 certificate support in Windows. CAPI1 refers to the support for base cryptographic components, such as encryption, decryption, and hashing functions. CAPI2 refers to the support for PKI and X.509 certificates, such as certificate chain validation, certificate stores, and signature verification. Cryptography Next Generation (CNG), a new application programming interface (API) set in Windows Vista, is positioned to replace the existing use of CAPI1, although CAPI1 is still supported. This document details troubleshooting PKI errors with CAPI2 but does not cover CNG and CAPI1.

Applications call CAPI2 to perform a number of tasks, such as:

• Build and verify certificate chains
• Manage per-user and per-computer certificate stores
• Encrypt/decrypt, encode/decode, and sign/verify messages

Background

PKI problems are difficult to troubleshoot in many PKI enabled applications. Many applications do not display detailed error information. For example, for many networking related errors, CAPI2 returns a "revocation offline" error to the application, which is also the error message that the application displays. Although the general nature of the error reported is discernible, it is unclear how a user can resolve the PKI problem. The API signature was designed to return an error code and string. Since the API is public, it was not possible to extend the function to return more detailed information without breaking existing applications. In addition, some errors are too performance intensive to detect during normal operations and are better left to post processing.

This makes it necessary to have better diagnostic capabilities for troubleshooting PKI in CAPI2. CAPI2 Diagnostics in Windows Vista provides logging of detailed information about certificate validation, network retrievals, revocation, and other low-level API results and errors. This enhanced functionality will help identify the cause of the PKI problem.

For common PKI related errors, there are specific patterns of information in the log. This document presents an overview of CAPI2 Diagnostics, provides guidance to help you interpret the log, and identifies patterns in the log that correspond to error scenarios. This information should help diagnose PKI problems and enable developers to write tools to troubleshoot common PKI problems.

Certificate Path Validation

Understanding the certificate path validation process is a fundamental requirement for troubleshooting PKI-enabled applications that use CAPI2. Consider a certificate chain, as shown in Figure 1. The root CA issues the subordinate CA certificate and the subordinate CA in turn issues the end-entity certificate. The root CA and the subordinate CA issue certificate revocation lists (CRLs) separately. The CRLs contain the serial numbers of any certificates revoked by the signing CA.

The first step in certificate path validation is certificate path discovery, which refers to the process of locating the issuing CA certificates for end-entity certificates and building a certificate path up to a trusted root certificate. Intermediate CA certificates are included as part of the application protocol or are picked up from Group Policy or through URLs specified in the Authority Information Access (AIA) extension. Once the path is built, every certificate in the path is verified for validity with respect to various parameters, such as name, time, signature, revocation status, and other constraints.

For details about certificate path validation, see https://go.microsoft.com/fwlink/?LinkId=27081.

CAPI2 Diagnostics in Windows Vista

CAPI2 Diagnostics is a feature in Windows Vista that utilizes the event logging and Event Viewer to provide better logging and troubleshooting capabilities for PKI applications based on the CAPI2 API set. The event reporting and tracing system in Windows Vista allows applications, components, and drivers to publish schematized events, query log files, and subscribe to events. This system unifies the event logging system and the event tracing framework. Event logging provides the necessary functionality to allow applications to structure and classify their events so that they are can be easily organized and viewed by an administrator. The events are logged in XML format and can be viewed in Event Viewer. By logging diagnostics information in XML, it is easier to write automated troubleshooting tools. Event Viewer provides the necessary user interface to view the events and enables filtering the events based on parameters like source, level and keywords.

For more information about the event reporting and tracing system in Windows Vista, see https://go.microsoft.com/fwlink/?LinkId=82279.

Getting Started With CAPI2 Diagnostics

CAPI2 events are logged through the Microsoft-Windows-CAPI2 channel in the event log. The events are based on the common CAPI, which is part of the certificate path validation process.

Enabling and Saving the CAPI2 Log

The following procedure provides information about how to enable logging, save or clear the log, and increase the log size. An administrator must perform the following procedure.

Enable and save the CAPI2 log from Event Viewer

1. Open Event Viewer. To open Event Viewer, click Start, click Control Panel, double-click Administrative Tools, and then double-click Event Viewer.

2. If the User Account Control dialog box appears, confirm that the action it displays is what you want, and then click Continue.

3. In the console tree, expand Event Viewer, expand Applications and Services Logs, expand Microsoft, expand Windows, and then expand CAPI2.

4. You can now perform following actions:

   1. To enable CAPI2 logging, right-click Operational, and click Enable Log.
   2. To save the log to a file, right-click Operational, and click Save Events as. You can save the log file in the .evtx format (which can be opened through the Event Viewer) or in .xml format.
   3. To disable CAPI2 logging, right-click Operational, and click Disable Log.
   4. If there is data present in the log before you reproduce the problem, it is recommended that you clear the log. This allows only the data relevant to the problem scenario to be collected from the saved log. To clear the log, right-click Operational, and click Clear Log.
   5. The default size for the event log is 1 MB. For CAPI2 Diagnostics, the log tends to grow in size quickly and it is recommended to increase the log size to at least 4 MB to capture relevant events. To increase the log size, right-click Operational, and click Properties. In the log properties, increase the maximum log size.

You can also enable logging and save the log by using the Wevtutil.exe tool. This tool is available in Windows Vista.

To enable and save the CAPI2 log by using Wevtutil.exe

1. Open a command prompt as an administrator. To do this, click Start, click All Programs, and then click Accessories.

2. Right-click Command Prompt, and click Run as administrator.

3. If the User Account Control dialog box appears, confirm that the action it displays is what you want, and then click Continue.

4. At the command prompt, run the following commands:

   1. To enable logging, type wevtutil.exe sl Microsoft-Windows-CAPI2/Operational /e:true.
   2. To save the log to a file, type wevtutil.exe epl Microsoft-Windows-CAPI2/Operational filename.evtx.
   3. To disable logging, type wevtutil.exe sl Microsoft-Windows-CAPI2/Operational /e:false.
   4. To clear the log, type wevtutil.exe cl Microsoft-Windows-CAPI2/Operational.
   5. To increase the log size, type wevtutil sl Microsoft-Windows-CAPI2/Operational /ms:<log-size-in-bytes>.

Run wevtutil -? to learn about all available options with this tool.

Note

Because event logging can affect performance, you should enable logging only when troubleshooting problems.

Events Overview

In CAPI2 Diagnostics, events are logged corresponding to specific tasks being performed. Some events correspond to specific APIs that are called to perform the task. Writers of automated troubleshooting applications and knowledgeable developers may cross-reference information in the event with API and data structure documentation on MSDN. The events are organized as top-level events and child events that are nested under the top-level events. These child events correspond to internal steps and contain more information about the actions performed and objects referenced as part of the top-level events.

For example, certificate path validation would involve the events listed in the following table.

Certificate path validation events

Event	Description
CertGetCertificateChain	Shows the results of building a certificate chain
CertVerifyRevocation	Indicates the results of revocation checking
CryptRetrieveObjectByUrlWire	Logs details about retrieval of objects such as CRLs or Online Certificate Status Protocol (OCSP) responses from over the network
CertRejectedRevocationInfo	Contains detailed error information in cases where Windows obtained invalid revocation information
X509Objects	Contains details of all objects processed as part of certificate path validation

The child events are organized in this manner because they represent steps that may be repeated several times during a top-level event. For example, CertVerifyRevocation might be called multiple times in the same CertGetCertificateChain event to check revocation for different certificates in the chain.

The list of various events logged and their description is available in Appendix A.

For more information about these APIs, see https://go.microsoft.com/fwlink/?LinkId=82278.

CAPI2 logs detailed information about the event in the UserData section of the event data. You can view this through the Details tab in Event Viewer. For APIs that return meaningful extended errors, CAPI2 logs error codes and descriptions in the event in a result field as part of the event data. CAPI2 also logs flags with a value attribute that refers to the DWORD value, and a list of Boolean attributes referring to individual flags that are set. For example:

<ErrorStatus value="1000040" CERT_TRUST_REVOCATION_STATUS_UNKNOWN="true" CERT_TRUST_IS_OFFLINE_REVOCATION="true" />

Logging PKI Objects

Detailed information about PKI objects, such as certificates, CRLs, and OCSP responses, is important for troubleshooting. CAPI2 logs the most relevant information in certificates and other PKI objects in XML in the X509Objects event.

The following is an example of an entry for a certificate in the "X509Objects" event:

<Certificate fileRef="B18D64DA254B2E51F533193E36FF10E91A9198FC.cer" subjectName="ContosoRoot">
 <Subject>
  <CN>ContosoRoot</CN> 
 </Subject>
 <SubjectKeyID computed="true" hash="24CD7CB5D99A6734139189D3828DD11069397CF6" /> 
 <Issuer>
  <CN>ContosoRoot</CN> 
 </Issuer>
 <SerialNumber>49BAC183C61F688F4F237F306950C6A3</SerialNumber> 
 <NotBefore>2005-01-01T00:00:00Z</NotBefore> 
 <NotAfter>2008-12-31T00:00:00Z</NotAfter>
</Certificate>

The fileRef attribute contains the filename that is the SHA1 hash of the object followed by the appropriate extension (.cer, .crl, or .bin).

X.509 objects are often referenced many times, even within a single top-level event. For example, the end-entity certificate is referenced in the "CertGetCertificateChain," "CertVerifyRevocation," and "CryptAIAUrlRetrievalWire" events. CAPI2 logs each reference of the objects using the fileRef attribute followed by the name, such as subjectName in the case of certificates, or issuerName in the case of CRLs and OCSP responses.

<Certificate fileRef="" subjectName="">
<CertificateRevocationList fileRef="" subjectName="">
<OCSPResponse fileRef="" subjectName="">

For the above example, it would be referenced by:

<Certificate fileRef="B18D64DA254B2E51F533193E36FF10E91A9198FC.cer" subjectName="ContosoRoot">

Correlated Events

Most events have a start and stop variant. CAPI2 logs an empty start event with only system information populated in the event log to indicate the beginning of the event. The full information including input parameters, outputs, and other status information is logged in the stop event. Grouping all information with the stop event reduces the number of times you have to search for another event.

The events are hierarchical in nature. All related events that correspond to internal functions that are called as parts of a top-level API are nested between the start and stop events corresponding to the top-level API. In the log, all of these related events (such as all events that are part of the same certificate path validation sequence) are grouped by a common correlation task identifier. Each event in the sequence has a sequence number associated with it, which helps to identify the order of events in the sequence.

For example, consider a certificate path validation that includes checking the revocation status for all certificates in the chain except for the root certificate. Additional revocation information may be retrieved from the network. One of the revocation checks fails. The nested sequence of events in the log is as follows:

CertGetCertificateChainStart

1. CertVerifyRevocationStart
   • CryptRetrieveObjectByUrlWireStart
   • CryptRetrieveObjectByUrlWire
2. CertVerifyRevocation
3. CertVerifyRevocationStart
   • CryptRetrieveObjectByUrlWireStart
   • CryptRetrieveObjectByUrlWire
   • CertRejectedRevocationInfo
4. CertVerifyRevocation

CertGetCertificateChain

CertVerifyCertifcateChainPolicy

X509Objects

Each event has the same correlation task identifier and increasing order of sequence numbers. The first event here would have the fields:

<CorrelationAuxInfo TaskId="{B679BF8C-B26E-401A-A35F-342C7753B6BA}" SeqNumber="9" />

Logging Modes

CAPI2 Diagnostics utilizes Event Viewer features, such as use of error level and keywords, for filtering the data in the log. For example, if you want to look at path validation related errors, you can filter by an event level of "Error" (level 2) and the keywords "chain building," "chain validation," and "revocation." Events are marked level 2 when the API returns an error and level 4 if the API returns a success.

In the default logging mode, only events with levels 2 and 4 are logged. This includes success and error events for the top-level events along with any child error events. For more detailed logging with additional information corresponding to the child events, enable the verbose logging mode. With the verbose mode, events with verbose level 5 are available in the log. For example, with verbose mode, the links to the binary X.509 objects are available in the log. X.509 objects are cached in file system at %USERPROFILE%/AppData/LocalLow/Microsoft/X509Objects.

This is logged as:

<X509Objects>
  <Base path="F:\Users\abby\AppData\LocalLow\Microsoft\X509Objects”/>

Verbose mode is useful for operations that rarely fail, such as store and cache operations, but are sometimes useful to give context to other events. If the operation fails, an error event with a level of 2 will be logged and will be available in the default logging mode.

To enable verbose mode

1. Click Start, click Start Search, type Run, and then press ENTER.

2. Type regedit, and then click OK.

3. If the User Account Control dialog box appears, confirm that the action it displays is what you want, and then click Continue.

4. Navigate to the following registry key: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\crypt32.

5. Add a DWORD (32-bit) value DiagLevel with value 0x00000005.

6. Add a QWORD (64-bit) value DiagMatchAnyMask with value 0x00ffffff.

Privacy

This section describes how CAPI2 Diagnostics handles personally identifiable information (PII). PII refers to information that can be used to identify or contact you. Administrators should be aware of how CAPI2 Diagnostics handles PII to help them comply with legal or corporate privacy guidelines, as needed.

In addition to the event data logged by Event Viewer, the information logged in the CAPI2 event log contains data about the certificates used by the user, the system, and applications. Data logged for diagnostic purposes that is likely to be PII, includes:

• Information contained in the certificates including the X.500 distinguished name of the subject and the certificate issuer. Depending on the application, this could contain:
  • E-mail addresses, while verifying signed e-mail.
  • File names, during signature verification on files.
  • URLs of Web sites visited, as part of certificate path validation in the Secure Sockets Layer (SSL) protocol.
• URLs accessed as part of certificate path validation, including locations for certificate revocation lists and intermediate certificates.
• IP addresses, such as those of the proxy servers contacted during network downloads.

The logging in the Microsoft-Windows-CAPI2 channel is not enabled by default. The logging can be enabled or disabled only by a local administrator and the log can only be read by an administrator. An administrator can save the log to a file or send it to an expert like Microsoft Premier Support for analysis.

For more information on the collection of PII by CAPI2 Diagnostics and the Event Viewer, see https://go.microsoft.com/fwlink/?LinkID=34493.

Troubleshooting Common Problems

Once you have enabled logging, reproduced the scenario, and collected the log, the next step is to interpret the data from the event log to identify the cause the problem. This section describes shows how CAPI2 Diagnostics events can be useful in troubleshooting some common PKI problems. For each problem scenario, you will learn about the conditions that make up the scenario and which events are logged in the Microsoft-Windows-CAPI2\Operational path in Event Viewer. You will learn how to troubleshoot the problem by performing the following:

• Identifying and correlating the relevant events
• Classifying the error based on the top-level error events
• Using the nested events to narrow down the error cause
• Determining the cause of the error from the detailed event data

CAPI2 logs information in a structured XML format that is designed to enable automated troubleshooting. XPATH, which is a language for addressing parts of an XML document, is very useful for querying information in the CAPI2 Diagnostics event log. XPATH can be used for matching nodes, such as checking if a node matches a particular pattern. There is a particular pattern in the data in the log that corresponds to specific scenarios. For example, the proxy list is logged when network retrieval occurs through a proxy. XPATH can be used to check if a node matches a pattern that indicates the presence of a proxy list. These patterns can be described using XPATH expressions, which, when evaluated, result in either a set of nodes that match the expression or a value such as string, a number, or a Boolean.

For more information about XPATH, see https://go.microsoft.com/fwlink/?LinkId=82280.

Problems addressed in this section include:

• Authentication failure while establishing an HTTP session over SSL
• Failure to log on with a smart card
• Certificate warning in Microsoft Outlook when verifying signed e-mail

Authentication Failure While Establishing an HTTP Session Over SSL

Scenario

You connect to a Web site over HTTPS by typing the Web site address, such as https://www.contoso.com, in Internet Explorer. Internet Explorer then blocks the navigation with the error shown in Figure 3.

If you continue to the Web site, the certificate error is indicated to the right of the address bar.

As part of certificate path discovery, the issuing CA certificates need to be located in order to build the certificate path. These certificates can be obtained from the cache or the certificate store on the client. Servers can provide additional information to the client. SSL is one example of this technique. In the SSL negotiation, the server provides the client with its own certificate and the intermediate certificates that the client can use to build the certificate path.

The conditions for this scenario are as follows:

1. The intermediate certificate required to build the certificate path for the server certificate is not present in the cache or the certificate store on the client.
2. As part of the SSL negotiation, only the end-entity server certificate is provided to the client. A valid intermediate certificate is not present in the local computer certificate store on the server, so it is not provided as part of the SSL negotiation.
3. Because an intermediate certificate is not available locally and is not provided by the application, CAPI2 uses the AIA extension in the end-entity certificate to retrieve the intermediate certificate from the network.
4. The certificate is not present at the network location specified in the AIA extension. As a result the network retrieval fails.
5. The intermediate certificate cannot be obtained; therefore, a certificate path cannot be built up a trusted root CA.

Diagnosis

The following image details which events are logged in this scenario.

1. To start troubleshooting, look at the "CertGetCertificateChain" error event (the event with task category Build Chain). The error reported by the "CertGetCertificateChain" API is the error that is generally reported back to the application. The process name in the event data makes it easier to narrow down the particular error event. In this case, you know that Internet Explorer is the application, and you can use the corresponding process name "iexplore.exe" while querying for the events. Determine all the related events that have the same "CorrelationAuxInfo" TaskId. This will help you narrow down the events to look for to identify the cause.
   
2. To classify the error, look at the status flags logged in the "CertGetCertificateChain" event. From the error status, you can identify the reason why the path validation failed. To detect a partial chain error, query for the flag "CERT_TRUST_IS_PARTIAL_CHAIN." In the certificate path discovery process, issuer certificates are located from the certificate stores and the certificates supplied by the application. The AdditionalStore field in the event data (Figure 6) contains additional certificates supplied by the application.
3. In this example, the application is providing only the end-entity certificate to CAPI2. CAPI2 attempts to download the issuer certificate using the AIA extension. This is represented by the "CertAIAUrlRetrievalWire" event (Figure 7) and is nested under "CertGetCertificateChain." To identify a problem with network retrieval of issuer certificates, query for the "CertAIAUrlRetrievalWire" event with Result as unsuccessful (a value other than zero).
4. For network retrieval, "CryptRetrieveObjectByUrlWire" event (Figure 8) is nested under "CertAIAUrlRetrievalWire." In "HTTPResponseHeadersInfo," under "AdditionalInfo," if you see a "404 Not Found" error, it means that the object was not found at the location. In this case, the object is not present on the location specified in the AIA extension in the end-entity certificate. This information can be queried from the "httpStatusCode" attribute in the "AuxInfo" element in this event.

You can similarly use the HTTP status code information to diagnose other HTTP errors. These scenarios are discussed in the HTTP Errors section of this document.

Failure to Log On with a Smart Card

Scenario

You attempt to log on to a computer by using a smart card. However, the smart card logon does not succeed. You then observe the following error.

During a smart card logon, mutual authentication takes place. The domain controller authenticates the client by verifying the certificate on the smart card. The client authenticates the domain controller by building a chain for the domain controller certificate and validating it.

The conditions for this scenario are:

1. The CRL needed to check revocation for the domain controller certificate is not available in the cache or the store on the client.
2. The CRL is not present at the location specified in the CRL distribution point in the domain controller end-entity certificate.
3. Revocation checking for the end-entity certificate fails. As a result, the certificate path validation for the domain controller certificate fails. Since the smart card logon does not work, you will have to log on to the computer by using a user name and password.

The error message indicates that the error is associated with the domain controller certificate. This certificate path validation happens on the client, so to diagnose the problem, enable CAPI2 Diagnostics logging on the client and reproduce the failure.

Diagnosis

In this scenario, the following events are listed in the log.

1. Look for the "CertGetCertificateChain" error event with the process name lsass.exe and identify all correlated events.
2. Query the error status flags of the "CertGetCertificateChain" error event. If valid revocation information cannot be retrieved, error status flags "CERT_TRUST_IS_OFFLINE_REVOCATION" and "CERT_TRUST_REVOCATION_STATUS_UNKNOWN" are set. Look through the chain elements and if for any element, this flag is present, then it means that the revocation check for that chain element failed. You can also find more information on the chain element for which the error is occurring, such as the subject name and hash of the certificate for which the revocation check failed.
3. Corresponding to revocation checking, the event "CertVerifyRevocation" is nested under "CertGetCertificateChain." If revocation checking fails, the "CertVerifyRevocation" event will report an error. To identify the cause of this error, look at the events nested under "CertVerifyRevocation."
4. During revocation checking, whenever network retrieval takes place, the "CryptRetrieveObjectByUrlWire" error event is nested under "CertVerifyRevocation." Similar to the previous scenario use the HTTP status code information to detect the cause of the error. HTTP error code "404" indicates that the object was not found.

You can follow similar steps for troubleshooting other networking failures. Use the top-level events to identify the revocation offline error and the "CryptRetrieveObjectByUrlWire" error event to identify the cause.

Certificate Warning in Microsoft Outlook When Verifying a Signed E-mail

Scenario

You open a signed e-mail in Microsoft Outlook 2007. Once the message is opened, if you click the digital signature icon, you see the following warning.

The conditions for this scenario are:

1. The CRL needed to check revocation for the signing certificate is not available in the cache or the store on the client.
2. The client computer is disconnected from the network, so the CRL cannot be downloaded from the CRL distribution point specified in the signing certificate.
3. Failure to download the CRL results in a "revocation offline" error and the message reported in the Outlook user interface is "The Certificate Revocation List needed to verify the signing certificate is either unavailable or it has expired."

Diagnosis

In this scenario, the following events are logged.

1. Look for the error event corresponding to "Build Chain" and identify all correlated events. To narrow down the search for specific error events, use the process name ("outlook.exe" in this case).
2. Certificate path validation failed because the revocation check for the certificate failed. Similar to the previous scenario, narrow down the revocation check failure by querying the flags in "CertGetCertificateChain" event. Look at the events nested under the corresponding "CertVerifyRevocation" error event.
   
3. Look at the action logged in the "CertRejectedRevocationInfo" error event. Action "Call_IsNetworkAlive" is logged in this scenario. The error means that the computer is disconnected from the network and cannot retrieve the revocation information required to check the certificate.

The "CertRejectedRevocationInfo" event can be useful for diagnosing a variety of errors that relate to failure in validation of the revocation information, either a CRL or an OCSP response. For these errors, you can identify the cause of the error by querying the corresponding action name in the "CertRejectedRevocationInfo" error event. Other diagnosis steps are similar. More detail on different error scenarios in this class of errors and how to troubleshoot them is available in the Error Scenarios section within this document.

These examples show that CAPI2 Diagnostics can help diagnose different application failures by providing information to identify the cause of the error.

To view specific events from the event log, you can manually filter the log using Event Viewer. Appendix D shows some examples of filtering using Event Viewer. If you save the log in an XML file, you can write XPATH queries in code to query the data from the log. See Appendix B for sample queries. Appendix C shows a C# code sample that demonstrates the use of the XPATH queries to identify an error's cause.

Error Scenarios

This section describes different error scenarios and identifies steps for troubleshooting these errors based on the information logged in the CAPI2 Diagnostics event log.

• Path validation errors
• Network retrieval errors
• Revocation check failure errors
• Certificate path discovery errors

Path Validation Errors

CAPI2 logs certificate path validation errors in the "CertGetCertificateChain" error event. You can diagnose these errors by querying the ErrorStatus flag in the "CertificateChain" element in the "CertGetCertificateChain" error event. The table below lists some of the common certificate path validation errors.

Path validation errors

ErrorStatus flag	Description
CERT_TRUST_IS_NOT_TIME_VALID	The end-entity certificate or one of the certificates in the chain is expired. More information about the certificate and its validity dates can be found in the "X509Objects" event.
CERT_TRUST_IS_NOT_SIGNATURE_VALID	The signature for the end-entity certificate or one of the certificates in the chain cannot be verified.
CERT_TRUST_IS_REVOKED	The end-entity certificate or one of the certificates in the chain is revoked. The revocation reason in available in the event data.
CERT_TRUST_IS_NOT_VALID_FOR_USAGE	The certificate is used for a purpose that is different from its enhanced key usage. Enhanced key usage and the application usage information can be obtained from the "CertGetCertificateChain" error event.
CERT_TRUST_IS_UNTRUSTED_ROOT	The certificate chain is built up to a root CA, but the root CA certificate is not present in the Trusted Root CA store on the computer.
CERT_TRUST_IS_CYCLIC	The certificate chain is cyclic (one of the certificates in the chain was issued by a CA that the original certificate had certified).
CERT_TRUST_IS_REVOCATION_UNKNOWN	The revocation status of the end-entity certificate or one of the certificates in the certificate chain is unknown. For more information on revocation check failures, see the Revocation Check Failure Errors section within this document.
CERT_TRUST_IS_OFFLINE_REVOCATION	The revocation status of the end-entity certificate or one of the certificates in the certificate chain is either offline or out of date. This is often caused due to a networking problem. For more information on revocation check failures, see the Revocation Check Failure Errors section within this document.
CERT_TRUST_IS_PARTIAL_CHAIN	The certificate chain is not complete. Certificate path discovery failure results in an incomplete certificate. For more information, see the Certificate Path Discovery Errors section within this document.

There are certain path validation errors such a partial chain or a revocation check failure that can be caused by different problems, including networking failures, invalid objects, or bad configurations. The subsequent sections describe these error scenarios which are, in many cases, the cause of other problems.

Network Retrieval Errors

Many PKI problems can be traced to a networking failure. For example, network retrieval failures can occur while retrieving different type of objects such as CRLs, OCSP responses, and issuer certificates using the AIA extension. Therefore, network download failures can result in failures in revocation checking or certificate path discovery. Network error scenarios can be further classified as the following:

• HTTP errors
• Proxy errors
• Timeout errors
• LDAP errors

HTTP Errors

This section lists and describes HTTP errors that you might encounter, including the following:

• Object not found
• Access denied
• Method not allowed by the Web server
• Proxy authentication required
• Other HTTP errors

Object Not Found

Download of the CRL will fail if it is not present at the location specified in the CRL distribution point extension in the certificate. Similarly, download of the issuer certificate will fail if the certificate is not present at the location in the AIA extension. If the object cannot be found, HTTP status code "404" is returned. This can be identified from the httpStatusCode field in the "CryptRetrieveObjectByUrlWire" event. The httpStatusCode can be used to diagnose similar HTTP scenarios, some of which are described in the following subsections.

Access Denied

If the access to the object on the server is denied, then HTTP response status "401" is returned. To identify this error, query for the action “HTTP_STATUS_DENIED” in the "CryptRetrieveObjectByUrlWire" error event. The URL of the server is logged in the same error event. This error indicates an authentication failure at the server.

Method Not Allowed by the Web Server

If the HTTP method used to request information from the server is not allowed by the server, HTTP status code "405" is returned to indicate this error. This can be queried by looking for httpStatusCode field with value "405" in "CryptRetrieveObjectByUrlWire" event. An example of this scenario is when CAPI2 makes a request for an OCSP response. CAPI2 first tries a GET method, and if the GET fails, then the POST method is used. If the server does not support POST, it will return an "HTTP 405" error.

Proxy Authentication Required

Another scenario that can be identified by using the HTTP status codes is the proxy authentication failure scenario. This is similar to the "Access denied" error. The difference here is that the authentication failure occurs at the proxy server used to access the Web resource and the proxy server returned HTTP status code "407." To identify this error, query for the action name "HTTP_STATUS_PROXY_AUTH_REQ" in the "CryptRetrieveObjectByUrlWire" event. This error scenario happens only in specific configurations such as when the proxy does not support integrated Windows authentication or the user or computer does not have permissions to access the Internet through the proxy. For more information, see https://go.microsoft.com/fwlink/?LinkId=82288.

Other HTTP Errors

Other HTTP errors can similarly be identified by looking at the HTTP status code in the "CryptRetrieveObjectByUrlWire" event. For more information, see https://go.microsoft.com/fwlink/?LinkId=82289.

Timeout Errors

Whenever a request is made to the Web server to retrieve an object, there is a timeout period associated with the network retrieval. The default timeout is 15 seconds, but this value can be configured through Group Policy or specified by the application. If the object is not retrieved within the timeout period, the "NetworkRetrievalTimeout" action is logged in the "CryptRetrieveObjectByUrlWire" event.

• Object is too large
• Request timed out
• Increasing the timeout period

Object is Too Large

A scenario in which a timeout can occur is if the object being downloaded is large and cannot be downloaded within the timeout period. CAPI2 automatically starts a background thread to retrieve objects. If the retrieval does not complete within the timeout period, the original foreground thread exits with the "NetworkRetrievalTimeout" error but the download is completed in the background thread. Once the download is complete, a "CryptRetrieveObjectByUrlWire" event is logged with action "PendingNetworkRetrievalComplete." This event also records the object that was retrieved.

For example, consider a smart card logon scenario. The logon fails because the CRL download times out. If you try to log on again after some time, the logon succeeds because the CRL download was completed in the background.

To diagnose this error from the event log:

1. Look for the "CryptRetrieveObjectByUrlWire" error event with the "NetworkRetrievalTimeout" action logged.
2. Identify the URL for which the download timed out from the URL element in this error event.
3. Look through the log for another "CryptRetrieveObjectByUrlWire" event with the same URL element value which has a "PendingNetworkRetrievalComplete" action logged. This corresponds to the background thread and it has a separate "CorrelationAuxInfo" TaskId from the foreground thread that does the path validation.
4. A Result Value of "0" means that the background thread successfully completed.

Note

If the calling process exits before the background thread completes the download, then the error event with "PendingNetworkRetrievalComplete" will not be logged.

Request Timed Out

A request made to the server to retrieve an object can timeout in different scenarios, such as:

• Server is not online and does not respond to the request
• DNS server timeout
• Proxy is not responsive
• Auto proxy discovery is slow, resulting in a timeout

In these scenarios, the "CryptRetrieveObjectByUrlWire" error event "NetworkRetrievalTimeout" action is logged to indicate that the timeout period was exceeded. However, the background thread does not succeed in retrieving the object either. The event with the "PendingNetworkRetrievalComplete" action logs the error as reported by the underlying networking component. For example, if a connection cannot be established to HTTP server, an action "Call_WinHttpSendRequest" is logged with an error value "2EFD," which is the hexadecimal value for the error message constant "ERROR_WINHTTP_CANNOT_CONNECT."

To diagnose this error, follow steps similar to diagnosis steps in the Object is Too Large section. The difference here is that the error event with "PendingNetworkRetrievalComplete" has a non-zero value in the Result element. To identify the cause, query the action that refers to the underlying API error such as "Call_WinHttpSendRequest" in the above example. Also, note that when an action prefixed by "Call_" is logged, it refers to the call to an underlying API that failed. For example, "Call_WinHttpSendRequest" indicates that the call to "WinHttpSendRequest" failed, and the error code returned by this API is also logged.

Increasing the Timeout Period

If you see a timeout followed by a successful retrieval in the background, it means that the object is large or that a slow network connection is causing the timeout. To address this problem, you can increase the default network retrieval timeout period. If you know size of the object and the bandwidth, you can estimate the time required to download it. The size of the object is available in the "Content-Length" header field in the "HTTPResponseHeadersInfo" element in "CryptRetrieveObjectByUrlWire" event data.

The following procedure details the steps to increase the retrieval timeout.

To increase the retrieval timeout

1. Click Start, click Start Search, type mmc, and then press ENTER.

2. If the User Account Control dialog box appears, confirm that the action it displays is what you want, and then click Continue.

3. On the File menu, click Add/Remove Snap-in.

4. In Add or Remove Snap-ins:

   1. If you are editing the Group Policy object (GPO) for the local computer, click Local Group Policy Editor, click Add, and then click Finish.
   2. If you are editing the GPO for the domain, click Group Policy Management Console, click Add, and then click Finish. In the console tree, double-click Group Policy Objects in the forest and domain containing the Default Domain Policy GPO that you want to edit. Right-click the Default Domain Policy GPO, and then click Edit.
   3. If you have no more snap-ins to add to the console, click OK.

5. In the console tree, expand Local Security Policy or Default Domain Policy depending on the selection in the previous step. Expand Computer Configuration, expand Windows Settings, expand Security Settings, click Public Key Policies, and then select Certificate Path Validation Settings.

6. Click the Network Retrieval tab.

7. Under Configure Default Retrieval Timeout, click Default URL retrieval timeout (in seconds).

8. Enter the desired value of the timeout.

9. Click Apply to apply the new settings.

Failure to Connect Through a Proxy Server

When connecting to the network through a proxy server, the proxy server must be correctly configured. If the proxy server is incorrectly specified, the retrieval of the revocation information over the network fails. This scenario can be identified by looking at the data logged in the "CryptRetrieveObjectByUrlWire" event. If a proxy is being used, it contains an action “ProxyListAndBypass” followed by the proxy list. If no proxy is configured locally, then it contains the action "NoProxy."

If the proxy server is incorrect, an action "BadProxy" is logged in the "CryptRetrieveObjectByUrlWire" event data. This is followed by the system error message explaining the error. For example, if DNS reports an error because the DNS name of the proxy server cannot be resolved, an action "Call_WinHttpSendRequest" with error code "2EE7," which is the hexadecimal value for error constant "ERROR_WINHTTP_NAME_NOT_RESOLVED," is logged. If the connection to the proxy server cannot be established, an action "Call_WinHttpSendRequest" with error code "2EFD," which is the hexadecimal value for error constant "ERROR_WINHTTP_CANNOT_CONNECT," is logged.

Whenever the network connection happens through a proxy server, the list of proxy servers and bypass sites is present in the log. This is logged in the "CryptRetrieveObjectByUrlWire" event in the action element "ProxyListAndBypassList." The parameter attribute of this action element lists the proxy servers and the bypass sites. "BadProxy" usually refers to a problem with the first proxy in the proxy list.

LDAP Errors

In the case of retrieving objects by using Lightweight Directory Access Protocol (LDAP), some errors are similar to other network retrieval scenarios. However, there are certain additional errors that are specific to LDAP.

Whenever an LDAP URL is encountered, CAPI2 first parses the URL to identify various components. If this fails, then "CrackLdapUrl" is logged in the "CryptRetrieveObjectByUrl" error event.

Errors can occur at various stages of the LDAP connection process. CAPI2 calls different LDAP APIs to establish the connection and download the object. Depending on which API fails, an error event is logged with an action name that includes the API name prefixed by "Call_." The error code returned by the LDAP API is also logged, and that can help to identify the cause of the problem. Depending on which LDAP API failed, actions "Call_ldap_connect," "Call_ldap_bind," or "Call_ldap_search" may be present in the log.

Other Network Retrieval Errors

This section details other network retrieval errors, including the following:

• Failed to retrieve contents of object
• Failed to connect to the server
• Protocol not supported
• Repeat network retrieval failures

Failed to Retrieve Contents of Object

When an object is retrieved from the network, the CryptQueryObject function is called to retrieve information about the contents of the object. If this fails, CAPI2 logs a "CryptRetrieveObjectByUrlWire" error event with action "Call_CryptQueryObject" and the extended error information for this failure. For example, if the CRL is corrupted and it cannot be decoded, this error is logged. Note that decoding of OCSP responses is done separately from this step. For more information, see the Decoding of OCSP Response Failed section within this document.

Failed to Connect to the Server

This error scenario refers to a failure to connect to the server, which can occur due to problems, such as:

• Server is not online
• URL specified is incorrect
• URL points to an incorrect IP address

The "CryptRetrieveObjectByUrl" API calls the WinHttpSendRequest API to send a request for the CRL to the HTTP server. In this scenario, if the connection to the server cannot be established, WinHttpSendRequest fails with the error "ERROR_WINHTTP_CANNOT_CONNECT." This error is logged in the "CryptRetrieveObjectByUrlWire" event as an action element called "Call_WinHttpSendRequest" with error value "2EFD," which is the hexadecimal value for the error message constant "ERROR_WINHTTP_CANNOT_CONNECT."

Protocol Not Supported

While retrieving an object, if the protocol specified in the URL is not supported, then an error "System cannot find the file specified" is logged in the "CryptRetrieveObjectByUrlWire" event. This can be queried by looking of a result of a value of 2 in the "CryptRetrieveObjectByUrlWire" error event. For example, CryptoAPI does not support the FTP protocol for network retrieval. If the CRL distribution point or the AIA extension has an FTP URL then the network retrieval will fail.

Repeat Network Retrieval Failures

CAPI2 avoids contacting the server too often for repeated offline responses. If a request happens prior to the next earliest time the server will be reached per a progressive back-off, then the repeat network retrieval is inhibited. In this scenario, the "CertRejectedRevocationInfo" error event with an action “CanRetrieveFromNetwork” is logged. The next earliest time the server will be reached is listed under the "EarliestOnlineTime" parameter.

In summary, in order to troubleshoot networking failures, fix the underlying component (such as network, proxy settings, Web server, or LDAP) that is causing failure.

Revocation Check Failure

In this section, you will learn how to diagnose revocation check failures. A revocation check could fail for a variety of reasons. Primarily, it could be because valid revocation information could not be retrieved or that revocation information was not available because no CRL distribution point and OCSP AIA extensions were in the certificate. The "CertGetCertificateChain" event contains an "ErrorStatus" flag "CERT_TRUST_REVOCATION_STATUS_UNKNOWN," which indicates a revocation check failure.

Most of the revocation check failures are due to a failure in retrieving valid revocation information. In this case, an additional "ErrorStatus" flag "CERT_TRUST_IS_OFFLINE_REVOCATION" is set to indicate that the revocation status of one of the certificates in the chain is offline or is no longer current. This could be because revocation information could not be retrieved or that the information could be retrieved but was invalid.

Results of revocation checking are logged in the "CertVerifyRevocation" event. To find the cause, you need to look at the "CertRejectedRevocationInfo" and "CryptRetrieveObjectByUrlWire" events, which are nested under the "CertVerifyRevocation" event.

Failure in Retrieval of Revocation Information

Network retrieval could fail for many reasons as listed in the Network Retrieval Errors section of this document.

Invalid Revocation Information

CRL is Expired

If the current date is later than the date in the nextUpdate field in the CRL, it means that the CRL has expired. This scenario is indicated by the action "CheckTimeValidity" in the "CertRejectedRevocationInfo" event. This event contains details about rejected revocation information such as the CRL or the OCSP response, as well as the certificate for which the revocation checking failed. This event also logs the action that failed and resulted in the revocation information being rejected.

The date in the nextUpdate field of the CRL can be identified from the event log. In the corresponding "X509Objects" event, look for the "CertificateRevocationList" element with the same fileref as the one logged in the corresponding "CertRejectedRevocationInfo" event. This element contains information about the CRL validity period as indicated by the thisUpdate and nextUpdate fields. To fix this error, publish a time-valid CRL on the server.

As mentioned in the Repeat Network Retrieval Failures section, CAPI2 avoids contacting the server too often by using progressive back-off. If the server repeatedly responds with a time invalid CRL, and the next request happens prior to the next earliest time the server will be reached per the back-off, then an action "CanCheckServerForUpdatedCrl" is logged. The next earliest time the server will be reached is listed under the "EarliestOnlineTime" parameter.

CRL Signature is Invalid

If the signature of the CRL cannot be verified, a "CertRejectedRevocationInfo" error event is logged with action name "IsCrlSignatureValid." It also logs the error code corresponding to the error that occurred and the location of the CRL. To fix this problem, the CRL on this server needs to be replaced with a CRL with a valid signature.

OCSP Response Cannot be Validated

As part of validating the OCSP response, the signature of the OCSP response is verified. However, if this signature verification fails, then the revocation checking will fail as a result. In this case, the action "IsResponseSignatureValid" is logged in the "CertRejectedRevocationInfo" error event. Additionally, CAPI2 fails to validate an OCSP response if it is not a basic signed response. If the response does not contain an object identifier (also known as OID), a "CertRejectedRevocationInfo" error event with an action element of "IsMissingResponseOID" is logged. If the response contains an object identifier that is different than the object identifier for a basic signed OCSP response, then an action of "IsUnsupportedResponseOID" is logged.

Unauthorized OCSP Signer

An OCSP response needs to be signed and the OCSP signer certificate is required to contain the OCSP signing enhanced key usage. If this enhanced key usage is not present, an error occurs. This error is indicated by the action "GetOCSPSignerCertificate" in the "CertRejectedRevocationInfo" event. Other errors that might occur in the context of an OCSP signer certificate also result in the same action being logged. These include:

• A time-invalid certificate
• A certificate that has a CRL distribution point extension or OCSP URL and does not have the "Revocation_No_Check" extension.

For these errors, CAPI2 also logs the OCSP signer certificate information.

Decoding of OCSP Response Failed

An OCSP response consists of data structures that need to be recursively decoded to obtain the necessary information. If the decoding of the OCSP response fails, CAPI2 logs a "CertRejectedRevocationInfo" error event. Depending on which step the decoding failed, CAPI2 logs this event with the action "Call_CryptDecodeObject_OCSP_RESPONSE," "Call_CryptDecodeObject_OCSP_BASIC_SIGNED_RESPONSE," or "Call_CryptDecodeObject_OCSP_BASIC_RESPONSE."

CRL or OCSP Response Has Unsupported Critical Extension

CAPI2 fails to verify revocation if the CRL or OCSP response contains a critical extension that is not supported. In this case, CAPI2 logs:

• A "CertRejectedRevocationInfo*"* error event with action "IsCrlCriticalExtensionSupported" if the error happens with a CRL
• An action "IsResponseCriticalExtensionSupported" if the OCSP response contains an unsupported critical extension

OCSP Response Status is Not Successful

An OCSP response consists of a "responseStatus" field that indicates the processing status of the prior request. If the processing is successful, it is indicated by a response status of "successful."

If there is an error, the OCSP "responseStatus" can have one of the following values:

• MalformedRequest
• InternalError
• TryLater
• SigRequired
• Unauthorized

If the "responseStatus" is not "successful," then a "CertRejectedRevocationInfo" event with action name “CheckResponseStatus” is logged. To query the status of the OCSP response, look at the corresponding "X509Objects" event. The status element of the "OCSPResponse" contains the response status.

For more details on the status values and the scenarios in which these would be set, refer to the RFC 2560 on the Internet Engineering Task Force Web site.

Other Errors

Certificate Does Not Contain Revocation Information

The presence of a CRL distribution point is necessary for any certificate except for a root CA certificate because the CRL distribution point provides a pointer to locate the revocation information. If the CRL distribution point is absent, the OCSP download location should be available in the AIA extension. The absence of both these pointers means that CAPI2 has no conclusive evidence that the CA actually published revocation information. This results in a revocation checking failure and a "revocation unknown" error status. In this case, CAPI2 logs a "CertRejectedRevocationInfo" error event with action "GetCrlOrOcspUrls" and the extended error "Cannot find object or property."

To troubleshoot this problem, check the certificate that is causing the error from the event log. Unless this is a root CA certificate, the corresponding issuing CAs must be reconfigured, and the certificates in question reissued with the correct CRL distribution point present. If this happens to be a root CA certificate, then an application requesting a revocation check on the root CA certificate is not the best practice.

CAPI2 does not support CRLs partitioned according to reasons. If the CRL distribution point in the certificate references a CRL partitioned according to reasons, CAPI2 will ignore that particular CRL distribution point location during processing. If no other CRL distribution point is available, it results in the same error scenario as the certificate not containing CDP information.

For best practices, see https://go.microsoft.com/fwlink/?LinkId=25612.

IDP in the CRL is Not Valid for the Subject Certificate

If the CRL contains an Issuing Distribution Point (IDP) extension, then CAPI2 checks that the IDP is valid for the subject certificate that is being checked for revocation. If this check fails and the IDP is not valid for the subject certificate, CAPI2 logs a "CertRejectedRevocationInfo" error event with the "Call_CertIsValidCRLForCertificate" action. For example, if the IDP in the CRL does not match the CDP in the subject certificate, this error is logged.

Certificate Path Discovery Errors

Server Configuration Error Causing Issuer Certificate Retrieval

As part of certificate path discovery, the intermediate certificates are either picked from the certificate store or cache on the client or are obtained from the application protocol. For example, in the case of SSL, along with the server authentication certificate, intermediate certificates are sent down to the client as part of the handshake. If the issuer certificate is not available locally or provided as part of the application protocol, then the AIA extension in the end-entity certificate is used to retrieve the issuer certificate from the network. CAPI2 logs the issuer certificate download in the "CertAIAUrlRetrievalWire" event. Network retrieval affects performance; therefore, this AIA retrieval should be avoided.

In the case of SSL, if you see a "CertAIAUrlRetrievalWire" event in the event log, it means that the server is not configured correctly. You can further mine the information in the log to identify missing or expired certificates. An administrator can deploy the intermediate CA certificates on clients using Group Policy but usually intermediate CA certificates are not installed locally on clients. To prevent the network retrieval, all servers should be properly configured with appropriate intermediate CA certificates. Valid intermediate certificates which are part of the certificate chain for the server certificate should be added to the server local computer Intermediate CA store. Once you have added the valid intermediate certificate, reestablish the server certificate binding. This will ensure that intermediate certificates are passed down as part of the handshake.

For more information, see article 834438 at https://go.microsoft.com/fwlink/?LinkId=82290.

Failure to Retrieve Issuer Certificates

If the issuer certificates cannot be retrieved, then a chain cannot be built from the end-entity certificate to a trusted root certificate. This causes a partial chain error (the certificate chain could not be built up to a trusted root authority). In the case of a partial chain error, the error status flag "CERT_TRUST_IS_PARTIAL_CHAIN" is set.

Retrieval of issuer certificates can fail for a variety of reasons. Network retrieval of issuer certificates could fail for many reasons as listed in the Network Retrieval Errors section of this document.

Additionally, there is a limit to the maximum number of URLs that can be tried as part of issuer certificate retrieval. Similarly, the number of certificates retrieved through AIA can also be limited. These limits are in place to provide the ability to limit the time spent in issuer certificate retrieval. If these limits are exceeded, then a partial chain error can result. If the issuer certificate can be retrieved but cannot be decoded successfully, then a partial chain error can also occur.

General Considerations for Automated Troubleshooting

The following process outlines the steps for mining the data in the log and identifying the cause of the error.

Step 1: Identify correlated events and group them together

1. Locate and record each unique "CorrelationAuxInfo" TaskId in the log.
2. For each TaskId, identify events with the same TaskId and group them together in a set.

Perform the following steps on each set of correlated events.

Step 2: Classify the error

In this step, classify the error scenario based on the top-level error event.

1. Look for the "CertGetCertificateChain" event with an error. If a "CertGetCertificateChain" error event is present, use the "ErrorStatus" flags to identify the error scenario
   • If the flag "CERT_TRUST_REVOCATION_STATUS_UNKNOWN" is set, then the path validation failed due to revocation check failure. Revocation check failures are usually due to retrieval errors, and an additional flag is set to indicate this. If this flag is set by itself, it could refer to revocation check failures for other reasons, such as the CDP having a Reasons field that is not supported.
   • If the flag "CERT_TRUST_IS_REVOCATION_OFFLINE" is also set, then path validation failed due to a retrieval error during a revocation check. Either the revocation information could not be retrieved or the information was retrieved, but is invalid.
   • If the "CERT_TRUST_IS_UNTRUSTED_ROOT" flag is set, then the certificate chain was built up to a root CA, but the root CA certificate was not present in the Trusted Root CA store.
   • If the "CERT_TRUST_IS_PARTIAL_CHAIN" flag is set, then a complete certificate chain up to a trusted root could not be created. This could happen if the issuer certificates were not found and the issuer certificate retrieval using AIA failed.
   • If any other flag is present, depending on the flag, this indicates that a specific path validation error has occurred. See the Path Validation Errors section of this document, which describes common path validation error scenarios and the corresponding "ErrorStatus" flags.
2. Look for the "CryptAcquireCertificatePrivateKey" event with an error. This event indicates errors related to obtaining a private key for a certificate.
3. Look for the "CertOpenStore" event with an error. This event indicates errors related to certificate store operations.
4. Look for other errors.

The error status flags in the "CertGetCertificateChain" event help to identify the symptom and limit the search for the cause of the error. To detect the cause of the error, look at other APIs called as part of path validation. However, in certain instances, these APIs may be called directly and there may not be a "CertGetCertificateChain" event in the correlated set of events. In this case, look directly for the cause of the errors using event based on these APIs. These error events could be any of other events described in the Error Scenarios section of this document.

Step 3: Identify the cause of the error

Step 2 identified the symptoms that limit the number of cause errors to look for in each particular scenario. In step 3, you identify this cause. This is can be achieved by looking at specific actions or status codes in the event log. The actions and status codes for various scenarios are described in the Error Scenarios section. Based on the class of errors, look for these action names and status codes in specific events. For example, for revocation failures, look at the "CertRejectedRevocationInfo" error events; for network retrieval errors, look at the "CryptRetrieveObjectByUrlWire" error events.

A flowchart corresponding to this algorithm is shown in following figure.

To automate the troubleshooting process, you can write code to query the log following the steps described above. Review the code sample in Appendix C; it illustrates how this process can be applied to diagnose errors. Appendix B includes some XPATH queries for different error scenarios, similar to the queries used in the code sample.

Summary

CAPI2 Diagnostics enables you to collect detailed information for various operations associated with certificates, such as certificate path validation and signature verification. This information is valuable for troubleshooting PKI problems in PKI-enabled applications. You can use the information about the scenarios and the queries from this document to interpret data from the log, search the event log for specific patterns, and identify the cause of the problem. CAPI2 Diagnostics offers a way to improve PKI troubleshooting and will help to reduce the time spent in troubleshooting common problems.

Feedback

Please post any questions or feedback you have about CAPI2 Diagnostics on the microsoft.public.security.crypto newsgroup.

Appendices

This section includes:

• Appendix A: Events in the CAPI2 diagnostic event log
• Appendix B: Sample XPATH queries
• Appendix C: Code sample for identifying patterns
• Appendix D: Filtering using the Event Viewer
• Appendix E: Certificate Related Tools for Windows

Appendix A: Events in the CAPI2 Diagnostics Event Log

The following table details the top-level events that are logged in the CAPI2 Diagnostics log.

CAPI2 Diagnostics: Top-level events

Event Name	Task	Description
CertGetCertificateChain	Build chain	Shows the results of a certificate path validation.
CertVerifyRevocation	Verify revocation	Shows the results of a revocation check on one or more certificates. In most cases this is nested under "CertGetCertificateChain" and is only triggered as a top-level event only if an application calls the "CertVerifyRevocation" API directly.
CertVerifyCertificateChainPolicy	Verify chain policy	Shows the results of an application evaluating a certificate chain according to predefined policies in Windows.
CertOpenStore	Open store	Shows the results of opening certificate stores.
CryptRetrieveObjectByUrlWire	Retrieve object from network	Shows the results of a network download.
CryptRetrieveObjectByUrlCache	Retrieve object from cache	Shows the results of retrieving an object from the disk cache.
CryptAcquireCertificatePrivateKey	Acquire certificate private key	Shows the results of an attempt to obtain a private key for a certificate.
WinVerifyTrust	Verify trust	Shows the results of signature verification by using "WinVerifyTrust," as well as certificate chain validation. It contains information about why the signed code failed to validate.
CryptCATAdminEnumCatalogFromHash	Find security catalog for file	Contains information about catalog look-ups. As part of signature verification, the hash of the file is obtained, searched in the catalog, and then the signature of the catalog is verified. This logs the hash of the file, catalog file path, and target file path. The file hash can be used as a link between this and the corresponding "WinVerifyTrust" event.

The following are lower-level events that are the nested under the top-level events in the CAPI2 Diagnostics log.

CAPI2 Diagnostics log: Lower-level events

Event Name	Task	Description
CertAIAUrlRetrievalWire	Retrieve issuer certificate from network	Shows the results of retrieval of the issuer certificate over the network.
CertAIAUrlRetrievalCache	Retrieve issuer certificate from cache	Shows the results of retrieval of the issuer certificate from the cache.
CertAutoRootUrlRetrievalWire	Retrieve non-Microsoft root certificate from network	Shows the results of download of a non-Microsoft root certificate as part of the automatic root update.
CertAutoRootUrlRetrievalCache	Retrieve non-Microsoft root certificate from cache	Shows the results of retrieving the non-Microsoft root certificate from the cache.
CertCrossCertUrlRetrievalWire	Retrieve cross-certificate from network	Shows the results of download of the cross-certificate.
CertCrossCertUrlRetrievalCache	Retrieve cross-certificate from cache	Shows the results of retrieving the cross-certificate from the cache.
CertRejectAIAUrlRetrieval	Inhibit issuer certificate retrieval	Contains information about why the issuer certificate retrieval was inhibited. This could be due to reasons similar to the maximum URL count being exceeded.
CertRejectedRevocationInfo	Reject revocation information	Contains information about why a specific CRL or OCSP response is rejected. The event will indicate whether the CRL or OCSP was retrieved from the cache, store, or wire. This is nested under the "CertVerifyRevocation" event.
X509Objects	X509 objects	Contains details of the certificate objects used in the activity. Details of X.509 objects, such as certificates, CRLs, and OCSP responses, are logged as part of this event.

Appendix B: Sample XPATH Queries

XPATHs can be used to query information in the event log to troubleshoot problems. If you save the data from the event log in an XML file, you can use the following XPATHs to identify specific information from the log.

The following table lists tasks detailed in the Troubleshooting Common Problems section of this document and their corresponding XPATH expressions.

XPATH Queries: Troubleshooting Common Problems section

Task	XPATH Expression
Look for Build Chain error event	//Event[System/Level=2]//CertGetCertificateChain
Look for Build Chain error event associated with a particular application (such as Internet Explorer)	//Event[System/Level=2]//CertGetCertificateChain[EventAuxInfo[@ProcessName='iexplore.exe']]
Locate all events with a specific task ID	//CorrelationAuxInfo[@TaskId='" + taskid + "']/parent::*
Query chain element error status flags	//CertGetCertificateChain[ChainElement[TrustStatus[ErrorStatus[@CERT_TRUST_IS_OFFLINE_REVOCATION]]]]

The following table lists errors detailed in the Path Validation Errors section of this document and their corresponding XPATH expressions.

XPATH Queries: Path Validation Errors section

Error	XPATH Expression
Certificate or one of the certificates in the certificate chain is expired	//CertGetCertificateChain/CertificateChain[TrustStatus[ErrorStatus[@CERT_TRUST_IS_NOT_TIME_VALID]] For other path validation errors, replace the corresponding ErrorStatus flag in the above expression.

The following table lists errors detailed in the Network Retrieval Errors section of this document and their corresponding XPATH expressions.

XPATH Queries: Network Retrieval Errors section

Error	XPATH Expression
HTTP 404 error	//CryptRetrieveObjectByUrlWire[AuxInfo[@httpStatusCode='404']] Look for other HTTP errors by replacing 404 with the corresponding status code.
Timeout error	//CryptRetrieveObjectByUrlWire[AdditionalInfo[Action[@name='NetworkRetrievalTimeout']]] If the object is too large, look at events for the background thread. objecturl=//CryptRetrieveObjectByUrlWire[AdditionalInfo[Action[@name='NetworkRetrievalTimeout']]]/URL //CryptRetrieveObjectByUrlWire[AdditionalInfo[Action[@name='PendingNetworkRetrievalComplete']] and URL=’{objecturl}’ and Result[@value=’0’]] If the server is not available, the call to WinHttpSendRequest fails. //CryptRetrieveObjectByUrlWire[AdditionalInfo[Action[@name='Call_WinHttpSendRequest']]]
Failure to connect through a proxy	//CryptRetrieveObjectByUrlWire[AdditionalInfo[Action[@name='BadProxy']]] To identify the bad proxy: //CryptRetrieveObjectByUrlWire/AdditionalInfo/Action[@name='BadProxy']/@parameter1
LDAP error	Querying failures in calls to LDAP APIs //CryptRetrieveObjectByUrlWire[AdditionalInfo[Action[@name='Call_ldap_connect']]] //CryptRetrieveObjectByUrlWire[AdditionalInfo[Action[@name='Call_ldap_bind']]] //CryptRetrieveObjectByUrlWire[AdditionalInfo[Action[@name='Call_ldap_search']]]
Error	XPATH Expression
HTTP 404 error	//CryptRetrieveObjectByUrlWire[AuxInfo[@httpStatusCode='404']] Look for other HTTP errors by replacing 404 with the corresponding status code.
Timeout error	//CryptRetrieveObjectByUrlWire[AdditionalInfo[Action[@name='NetworkRetrievalTimeout']]] If the object is too large, look at events for the background thread. objecturl=//CryptRetrieveObjectByUrlWire[AdditionalInfo[Action[@name='NetworkRetrievalTimeout']]]/URL //CryptRetrieveObjectByUrlWire[AdditionalInfo[Action[@name='PendingNetworkRetrievalComplete']] and URL=’{objecturl}’ and Result[@value=’0’]] If the server is not available, the call to WinHttpSendRequest fails. //CryptRetrieveObjectByUrlWire[AdditionalInfo[Action[@name='Call_WinHttpSendRequest']]]
Failure to connect through a proxy	//CryptRetrieveObjectByUrlWire[AdditionalInfo[Action[@name='BadProxy']]] To identify the bad proxy: //CryptRetrieveObjectByUrlWire/AdditionalInfo/Action[@name='BadProxy']/@parameter1
LDAP error	Querying failures in calls to LDAP APIs //CryptRetrieveObjectByUrlWire[AdditionalInfo[Action[@name='Call_ldap_connect']]] //CryptRetrieveObjectByUrlWire[AdditionalInfo[Action[@name='Call_ldap_bind']]] //CryptRetrieveObjectByUrlWire[AdditionalInfo[Action[@name='Call_ldap_search']]]

The following table lists errors detailed in the Revocation Check Failure Errors section of this document and their corresponding XPATH expressions.

XPATH Queries: Revocation Check Failure section

Error	XPATH Expression
CRL is expired	//CertRejectedRevocationInfo[Action[@name='CheckTimeValidity']]
CRL signature cannot be verified	//CertRejectedRevocationInfo[Action[@name='IsCrlSignatureValid']]
OCSP response cannot be validated	If the signature of the OCSP response cannot be validated: //CertRejectedRevocationInfo[Action[@name='IsResponseSignatureValid']] If the decoding of the OCSP response failed: //CertRejectedRevocationInfo[Action[@name='Call_CryptDecodeObject_OCSP_RESPONSE']]
Unauthorized OCSP signer	//CertRejectedRevocationInfo[Action[@name=’GetOCSPSignerCertificate']]
Decoding of OCSP response failed	//CertRejectedRevocationInfo[Action[@name=’Call_CryptDecodeObject_OCSP_RESPONSE']]
CRL or OCSP response has unsupported critical extension	//CertRejectedRevocationInfo[Action[@name='IsCrlCriticalExtensionSupported’]] //CertRejectedRevocationInfo[Action[@name='IsResponseCriticalExtensionSupported']]
OCSP response status is not successful	//CertRejectedRevocationInfo[Action[@name='CheckResponseStatus']]

Appendix C: Code Sample for Identifying Patterns

This code sample illustrates how to detect the cause of the error in case of the scenario described in the Failure to Log On with a Smart Card section. It shows an implementation of the process outlined in General Considerations for Automated Troubleshooting section. You can extend this to include more error scenarios. To run this sample, save the events from the CAPI2 Diagnostics event log in an XML file. Provide the name of this XML file as a command line argument while running the sample.

The main functions in this sample code are:

• IdentifyPattern: The starting point for identifying the error pattern, for a specific sequence of correlated events.
• ClassifyError: Top-level classification of the path validation error based on the "ErrorStatus" flags in "CertGetCertificateChain."
• CheckHTTPErrorPattern: Check for the class of retrieval errors due to HTTP errors; there could be similar functions, such as "CheckProxyErrorPattern" or "CheckTimeoutErrorPattern."
• Helper functions: The rest of the functions are helper functions to query specific information from the event log.

Using System;
using System.IO;
using System.Xml;
using System.Xml.XPath;
using System.Collections;
using System.Text;

namespace CAPIDiagPatternIdentifier
{
 class PatternIdentifier {
  // The top-level classification of error symptoms 
  const int OTHER_ERROR = 0;
  const int REVOCATION_FAILURE = 1;
  const int RETRIEVAL_ERROR = 2;
  const int SIGNATURE_INVALID = 3;
  const int UNTRUSTED_ROOT = 4;
  const int PARTIAL_CHAIN = 5;
  const int REVOKED = 6;    

  const string filePath = ".\\";
  //The log namespace
  const string logns = "https://schemas.microsoft.com/win/2004/08/events/event";    

  // Define error queries and message strings here
  // This can be used to look for specific patterns in the log

  const string err404qry = "//ns:CryptRetrieveObjectByUrlWire[ns:AuxInfo[@httpStatusCode='404']]";
  const string err404msg = "CRL could not be found at the CDP URL :";     

  private XmlDocument doc;
  private XPathNavigator xpath;
  private XmlNamespaceManager nsmanager;
  
  public PatternIdentifier(string[] args) {
   // Create the XmlDocument object
   doc = new XmlDocument();
   // Open the log specified by the command line
   FileStream fs = new FileStream(args[0], FileMode.Open);
   // Load the log into an XML DOM
   doc.Load(fs);
   xpath = doc.CreateNavigator();      
   nsmanager = new XmlNamespaceManager(xpath.NameTable);
   nsmanager.AddNamespace("ns", logns);   
  }

  public String GetTaskId(XmlNode eventNode) {
   String TaskId = null;
   XPathNavigator navigator = eventNode.CreateNavigator();
   XPathNodeIterator itr = navigator.Select("@TaskId");
   if (itr.MoveNext())  {
    TaskId = itr.Current.Value;
   }
   return TaskId;
  }

  private String GetProcessName(XPathNavigator currentNode)  {
   String processName = null;
   XPathNodeIterator itr = currentNode.Select("//@ProcessName");
   if (itr.MoveNext())   {
    processName = itr.Current.Value;
   }
   return processName ;
  }

  // A common function for checking all network errors
  private bool CheckNetworkRetrievalPattern(XPathNavigator currentNode, String evtQuery, String errString)  {
   XPathExpression evtqry = xpath.Compile(evtQuery);
   evtqry.SetContext(nsmanager);
   if (currentNode.Matches(evtqry) && (currentNode is IHasXmlNode))   {
    XmlNode node = ((IHasXmlNode)currentNode).GetNode();
    string URL = (node.SelectSingleNode("ns:URL", nsmanager)).InnerXml;
    System.Console.WriteLine("Network Retrieval failed. "+ errString + URL);
    return true;
   }
   return false;
  }

  // Check for HTTP Errors
  private bool CheckHTTPErrorPattern(XPathNavigator currentNode)  {
   if (CheckNetworkRetrievalPattern(currentNode, err404qry, err404msg)) return true;
   // Add other HTTP errors here
   return false;
  }

  private bool isInArray(ArrayList al, string val)  {
   for (int i = 0; i < al.Count; ++i)   {
    if (val == (string)al[i])    {
     return true;
    }
   }
   return false;
  }
     
  public void IdentifyPattern(XPathNodeIterator eventNodes, string TaskId)  {
   if (eventNodes.Count > 0)   {
    while (eventNodes.MoveNext())    {
     String fileRef = null;
     String subjectName = null;
     String processName = null ;
     int errorType = 0;
     // Need to look at which chain element has an error           
     XPathExpression errQuery = xpath.Compile("//ns:CertGetCertificateChain[ns:Result[@value!='0']]");
     errQuery.SetContext(nsmanager);
     if (eventNodes.Current.Matches(errQuery))     {
      // We know that a chain element has an error
      // Get more data about the element and the error
      errorType = ClassifyError(eventNodes, ref fileRef, ref subjectName, ref processName);
     }     
     // We now try to identify the cause of this error.
     switch (errorType)    {
      case RETRIEVAL_ERROR:
       {
        while (eventNodes.MoveNext())     {
         if (CheckHTTPErrorPattern(eventNodes.Current)) break;
         //Check for other network errors here
        }
        break;
       }
      //Add more cases per classification
      case OTHER_ERROR:
       {
        //Any other errors not covered by the cases above.        
        break;        
       }         
     }     
    }
    eventNodes = null;
   }
  }

//Top-level classification of error symptoms (step 2 of the algorithm)
//Find the chain element corresponding to the specific error symptom
  private int ClassifyError(XPathNodeIterator eventNodes, ref String fileRef, ref String subjectName, ref String processName)  {
   processName = GetProcessName(eventNodes.Current);
   System.Console.WriteLine("Application causing error: " + processName);
   XPathNodeIterator chainElement = eventNodes.Current.SelectDescendants("ChainElement", logns, false);
   while (chainElement.MoveNext())   {
    XPathExpression statusQuery1 = xpath.Compile("//ns:ChainElement[ns:TrustStatus[ns:ErrorStatus[@CERT_TRUST_REVOCATION_STATUS_UNKNOWN]]]");
    XPathExpression statusQuery2 = xpath.Compile("//ns:ChainElement[ns:TrustStatus[ns:ErrorStatus[@CERT_TRUST_IS_OFFLINE_REVOCATION]]]");
    XPathExpression statusQuery3 = xpath.Compile("//ns:ChainElement[ns:TrustStatus[ns:ErrorStatus[@CERT_TRUST_IS_NOT_SIGNATURE_VALID]]]");
    XPathExpression statusQuery4 = xpath.Compile("//ns:ChainElement[ns:TrustStatus[ns:ErrorStatus[@CERT_TRUST_IS_UNTRUSTED_ROOT]]]");
    XPathExpression statusQuery5 = xpath.Compile("//ns:ChainElement[ns:TrustStatus[ns:ErrorStatus[@CERT_TRUST_IS_PARTIAL_CHAIN]]]");
    XPathExpression statusQuery6 = xpath.Compile("//ns:ChainElement[ns:TrustStatus[ns:ErrorStatus[@CERT_TRUST_IS_REVOKED]]]");
    statusQuery1.SetContext(nsmanager);
    statusQuery2.SetContext(nsmanager);
    statusQuery3.SetContext(nsmanager);
    statusQuery4.SetContext(nsmanager);
    statusQuery5.SetContext(nsmanager);
    statusQuery6.SetContext(nsmanager);
    chainElement.Current.MoveToChild("Certificate", logns);
    fileRef = chainElement.Current.GetAttribute("fileRef", "");
    subjectName = chainElement.Current.GetAttribute("subjectName", "");
    chainElement.Current.MoveToParent();
    if (chainElement.Current.Matches(statusQuery5))     {
     System.Console.WriteLine("Path validation failed for the certificate of subjectName:" + subjectName + " (" + fileRef + ") because a complete chain could not be built");
     return PARTIAL_CHAIN;
    } 
    else if (chainElement.Current.Matches(statusQuery4))     {
     System.Console.WriteLine("Path validation failed for the certificate of subjectName:" + subjectName + " (" + fileRef + ") because it did not chain up to a trusted root");
     return UNTRUSTED_ROOT;
    }
    else if (chainElement.Current.Matches(statusQuery3))     {
     System.Console.WriteLine("Path validation failed for the certificate of subjectName:" + subjectName + " (" + fileRef + ") because its signature could not be verified.");
     return SIGNATURE_INVALID;
    }
    else if (chainElement.Current.Matches(statusQuery2))     {      
     System.Console.WriteLine("Revocation check failed for the certificate of subjectName:" + subjectName + " (" + fileRef + ") because revocation server was offline");
     return RETRIEVAL_ERROR;
    }
    else if (chainElement.Current.Matches(statusQuery1))     {
     System.Console.WriteLine("Revocation status for he certificate of subjectName:" + subjectName + " (" + fileRef + ") could not be determined");
     return REVOCATION_FAILURE;
    }     
    else if (chainElement.Current.Matches(statusQuery6))    {
     System.Console.WriteLine("Certificate is revoked");
     return REVOKED;
    }
   }
   return OTHER_ERROR;
  }
  
//The XML event log file is command line argument to the program
  static void Main(string[] args)  {
   PatternIdentifier mainProg = new PatternIdentifier(args);
   XmlNode root = mainProg.doc.DocumentElement;
//Identify the TaskId for all error events
   XmlNodeList errEvents = root.SelectNodes("//ns:Event[ns:System/ns:Level=2]//ns:CorrelationAuxInfo",mainProg.nsmanager);
   ArrayList tasks = new ArrayList(); 
   foreach (XmlNode errEvent in errEvents)   {
    string taskId = null;
    taskId = mainProg.GetTaskId(errEvent);
    if (!mainProg.isInArray(tasks, taskId))    {
     tasks.Add(taskId);
    }     
   }
//For each TaskId, correlate all events with same taskId and identify an error pattern in the correlated set of events.
   for (int i = 0; i < tasks.Count; ++i)   {
    string TaskId = (string)tasks[i];
    XPathExpression eventQuery = mainProg.xpath.Compile("//ns:CorrelationAuxInfo[@TaskId='" + TaskId + "']/parent::*");
    eventQuery.SetContext(mainProg.nsmanager);
    XPathNodeIterator eventNodes = mainProg.xpath.Select(eventQuery);
    mainProg.IdentifyPattern(eventNodes,TaskId);          
   }       
   System.Console.Read();
  }
 }
}

Appendix D: Filtering Using the Event Viewer

You can write manual queries in the Event Viewer filter to view specific events.

To write manual queries in the Event Viewer filter

1. Open Event Viewer. To open Event Viewer, click Start, click Control Panel, double-click Administrative Tools, and then double-click Event Viewer.

2. If the User Account Control dialog box appears, confirm that the action it displays is what you want, and then click Continue.

3. In the console tree, expand Event Viewer, expand Applications and Services Logs, expand Microsoft, expand Windows, and then expand CAPI2.

4. Right-click Operational, and click Filter Current Log.

5. Click the XML tab, and select the Edit query manually check box.

6. To filter the log, type a corresponding query in the text box.

   1. To look for all "CertGetCertificateChain" error events with a processname of "iexplore.exe," enter the following query in the text box:

      <QueryList>
      <Query Id="0" Path="Microsoft-Windows-CAPI2/Operational">
      <Select Path="Microsoft-Windows-CAPI2/Operational">Event[System[(Level=2)] and UserData[CertGetCertificateChain[EventAuxInfo[@ProcessName='iexplore.exe']]]]</Select>
      </Query>
      </QueryList>
      

   2. To identify all correlated events corresponding to the particular "CertGetCertificateChain" error event, you can use the following query in the Event Viewer filter (Here {taskid} is the Task ID for the "CertGetCertificateChain" error event):

      <QueryList>
      <Query Id="0" Path="Microsoft-Windows-CAPI2/Operational">
      <Select Path="Microsoft-Windows-CAPI2/Operational">*[*[*[CorrelationAuxInfo[@TaskId='{taskid}']]]]</Select>
      </Query>
      </QueryList>
      

Appendix E: Certificate-Related Tools for Windows

CAPI2 Diagnostics is available on Windows Vista. There are existing diagnostic tools that you can use for troubleshooting problems on previous versions of Windows. CryptoAPI Monitor (CAPIMON) is a tool that allows the administrator to monitor an application's CryptoAPI calls and the results. CAPIMON was written to troubleshoot existing applications that do not report errors properly by capturing the input and output of specific APIs instead of relying on the application's error reporting or logging. CAPIMON is a good troubleshooting option on previous versions of Windows in situations when application's error reporting is not indicative of the cause of the failure. To download CAPIMON, see https://go.microsoft.com/fwlink/?LinkId=82293.

To verify certificates, key pairs, and certificate chains, you can use the Certutil.exe command-line tool. Certutil is also useful for configuring certificate services and displaying the configuration information. For more information about Certutil, see https://go.microsoft.com/fwlink/?LinkId=82294.

Additional Resources

For more information, see the following documents:

1. RFC 3280: "Internet Public Key Infrastructure Certificate and Certificate Revocation List (CRL) Profile"
2. RFC 2560: "X.509 Internet Public Key Infrastructure Online Certificate Status Protocol – OCSP"
3. Troubleshooting Certificate Revocation and Status Checking
4. Best Practices for Implementing a Microsoft Windows Server 2003 Public Key Infrastructure
5. Windows Vista Privacy Statement
6. Cryptography Functions
7. Reducing Support Costs with Windows Vista
8. Addressing Infosets with XPATH