The return of PAC-mania [AKA some reasons why PAC verification can fail]

There's tons of good stuff out there on Kerberos PAC verification - but with current trends showing an increase in incoming cases related to this type of issue I though it would do me good to brush up on this and link the most relevant articles together.

In short; PAC verification is the process where a member server sends a verification request to a DC to verify the Kerberos ticket of an incoming user to confirm they are members of the groups they claim to be members of.

A simplified version of PAC verification is as follows:

1. Incoming user presents a Kerberos ticket that states the user is a member of a group (say, the Domain Admins group)
2. The member server wants to verify that the PAC for the Kerberos ticket hasn't been modified (i.e. to make sure the user hasn't modified the ticket and inserted a group they aren't actually members of)
3. The member server attempts to contact a DC in the domain that issued the Kerberos ticket and asks it to verify that the PAC for it hasn't been modified
4. The DC responds back to the member server that the PAC is OK *or* that the PAC has been modified and is NOT OK.

I.e. the purpose of the PAC verification is to confirm that the PAC hasn't been tampered with.

If the attempt to verify the PAC fails for some reason, you'll see a Kerberos Event ID 7 error logged on the server (similar to the one below):

Event Type: Error
Event Source: Kerberos
Event Category: None
Event ID: 7
User: N/A
Description: The Kerberos subsystem encountered a PAC verification failure. This indicates that the PAC from the client <computer name> in realm <AD DNS domain name> had a PAC which failed to verify or was modified. Contact your system administrator.
Data: 0000: c000005e

Note the additional data error code above may be different depending on whether you're looking at this from a member server or a domain controller:

5e 00 00 c0	C000005e	-1073741730	STATUS_NO_LOGON_SERVERS	There are currently no logon  servers available to service the logon request.
6d 00 00 c0	c000006d	-1073741715	STATUS_LOGON_FAILURE	The attempted logon is invalid. This is either due to a bad username or authentication information
50 00 02 c0	c0020050	-1073610672	RPC_NT_CALL_CANCELLED	The remote procedure call was  cancelled.

 
However, by itself it doesn't mean that there is a problem with the actual PAC in the Kerberos ticket.  If the PAC verification failed it might have failed because of the following:

• The PAC we asked the DC to confirm had actually been tampered with and the DC told us so.
• The DC we asked to verify the PAC was unable to verify it because it was unable to obtain the original password for the account whose PAC is being verified
• The DC we asked to confirm the PAC didn't respond within the 45 second timeout period (because it was too busy)
• We were unable to reach any DC to send a PAC verification request to (i.e. network issues)

Of these, only the first indicates an issue with the PAC itself - the others are failing because of external factors.

There are two other important factors that come into play in PAC verification - besides network issues (typically followed by a Netlogon 5719 event which may be temporary and resolve itself or may need active troubleshooting from the networking people).

One is the default MaxConcurrentAPI setting and the other is the LsaLookupRestrictIsolatedNameLevel setting which controls how DC's treat Name2Sid requests which dont't contain a domain prefix to qualify which domain they belong to.

The default settings for both are fine for a typical AD setup.  However, care should be taken to evaluate whether you need to tweak these in larger installations where you have high volume web servers (f.x. Sharepoint) that are generating a large number of authentication requests and PAC verifications.

• In essence, the MaxConcurrentAPI setting controls how many threads are spun up to take care of the authentication of users (via API calls - i.e. Maximum concurrent authentication API calls).
• The LsaLookupRestrictIsolatedNameLevel setting controls if DC's that receive an unknown name without a domain prefix (i.e. (null)USER instead of DOMAINUSER) do with the results - by default the DC makes a serial query to each and every domain it has a trust relationship with.

The two settings become important when you have multiple trusts, multiple authentication attempts that don't prefix the username with an identifying domain name and massive Kerberos PAC verifications caused by busy web servers.

In this scenario, the default MaxConcurrentAPI setting effectively creates a bottleneck on both the member server and the DC side that is exacerbated further by chasing isolated names across trusts and even further if one or more of those trusts is broken.

If you couple that together with the DC's being constantly busy walking the list of trusts to locate isolated usernames then you have a scenario where the member server starts failing PAC verification because the DC eventually starts taking longer and longer to respond with the PAC verification message as the queue of incoming authentication and PAC verification requests grows - eventually exceeding the 45 second timeout limit.

In the case where there is a problem with one of the trusts then this becomes an even bigger problem - as it will cause the time the DC takes to respond to increase even further is it must wait for DC's at the other end of the trusts to respond or time out.

• Increasing the MaxConcurrentAPI limit on the member server side allows the member server to spin up more authentication threads - if the DC is busy because of the scenario above then this will by itself only have a limited effect however.
• Increasing the MaxConcurrentAPI limit on the DC side allows the DC to serve more simultaneous authentication / PAC verification requests - if most of the DC threads are busy waiting on responses from DC's on the other ends of trusts then this will by itself only have a limited effect also.
• Turning off LsaLookupRestrictIsolatedNameLevel so that the DC's only check specific trusts or unknown user accounts if a domain prefix or UPN is specified (i.e. DOMAINUSER or user@domain.com) will in combination with increasing the MaxConcurrentAPI settings on both member servers and DC's have the greatest effect.
  Note that turning off isolated name lookup *will* have a negative effect on any users or applications that don't properly prefix their username with the appropriate domain name or use a UPN.

Note: there is a registry value that makes it possible to disable PAC verification if you want to trust the PAC without verifying it (ValidateKdcPacSignature) but this really only applies to processes that are running as a service.
The IIS worker processes run as application pools that are started by the WWW service so they effectively don't run as a service.
I.e. you cannot turn off Kerberos PAC verification for IIS (or Sharepoint which runs on IIS) using the registry entry (but see the first link below for a User Right which can accomplish that).

[Tons of] Links:

Understanding Microsoft Kerberos PAC Validation
http://blogs.msdn.com/b/openspecification/archive/2009/04/24/understanding-microsoft-kerberos-pac-validation.aspx

Kerberos PAC Validation… what is it?
http://blogs.msdn.com/b/spatdsg/archive/2007/03/07/pac-validation.aspx

You are intermittently prompted for credentials or experience time-outs when you connect to Authenticated Services
http://support.microsoft.com/kb/975363

More Kerberos fun with PAC’s- decrypt the PAC
http://blogs.msdn.com/b/spatdsg/archive/2009/03/26/more-kerberos-fun-with-pac-s.aspx

MaxConcurrentAPI - or, how fast can you authenticate users?
http://blogs.msdn.com/b/spatdsg/archive/2006/01/05/507299.aspx

Utilizing the Authorization Data in Kerberos Tickets for Access Control to Resources
http://msdn.microsoft.com/en-us/library/aa302203.aspx

Trusts and isolated names and logon performance
http://blogs.technet.com/b/instan/archive/2009/07/07/trusts-and-isolated-names-and-logon-performance.aspx

You experience a delay in the user-authentication process when you run a high-volume server program on a domain member in Windows 2000 or Windows Server 2003
http://support.microsoft.com/kb/906736

Updated: NTLM and MaxConcurrentApi Concerns
http://blogs.technet.com/b/ad/archive/2008/09/23/ntlm-and-maxconcurrentapi-concerns.aspx

How to configure an ISA Server computer for a very large number of authentication requests
http://support.microsoft.com/kb/326040

Improving Web Proxy Client Authentication Performance on ISA Server 2006
http://technet.microsoft.com/en-us/library/bb984870.aspx