Troubleshooting Credential Roaming

Hi. Jim here again from Directory Services with a follow up to my Understanding Credential Roaming blog post. To review, credential roaming makes it possible to roam the user's credentials in a manageable, secure manner that is ultimately transparent to the user. What follows is a deeper dive into the inner workings of Credential Roaming. I will point out what must be setup initially and verified if you are planning to deploy Credential Roaming in your enterprise. Or you may have already deployed Credential Roaming and are interested in finding out what to check if your users report issues that would indicate their credentials are not roaming.

Potential credential roaming problems can be broken down into three major areas:

1. Initial setup requirements
2. Client Side Problems
3. Active Directory database size growth

I will give you guidance on how to avoid some common problems with initial setup if you are currently planning to test and deploy credential roaming in your environment.

To help with all the different things that have to be checked Rob Greene has created a vbscript data gathering tool that can be used in a user logon script or run as a “one off” on a workstation to collect this information for you about the users in your environment. Of course, this script is provided “as is” without any warranties and it is not supported by Microsoft. This script requires CAPICOM.DLL to be deployed and registered on all workstations. I have included the link to the “CollectUserStore.vbs” script at the bottom of this post.

Let’s begin now with the initial setup.

Initial Setup

In this section I will illustrate the preliminary steps that must be taken to prepare your environment for credential roaming. These steps must be taken prior to the deployment of the credential roaming group policy.

If you are using Windows XP (pre-SP3) or Windows Server 2003 (Pre-SP2) you will need to deploy the latest credential roaming hotfixes to these operating systems before credential roaming will function properly. If you are using Windows Vista (pre-SP1) you must deploy SP1 or SP2 or you are unsupported. The following is the most recent Credential Roaming hotfix for XP at the time this blog was published:

907247 - Description of the Credential Roaming service update for Windows Server 2003 and for Windows XP - https://support.microsoft.com/default.aspx?scid=kb;EN-US;907247

and

973502 The size of the Ntds.dit file becomes larger on one or more domain controllers that are running Windows Server 2003 or Windows Server 2008 after you enable the credential roaming feature for the domain - https://support.microsoft.com/default.aspx?scid=kb;EN-US;973502

If your workstations and servers are running the latest versions of service packs then you only need install KB973502.

If you “turn on” credential roaming by configuring and applying the credential roaming group policy before deploying the 907247 hotfix or XP service pack 3 to your environment, your users might enroll for new certificates when they logon to computers where the 907247 hotfix has not been installed. This will result in differences within the certificate stores between workstations that the user logs into.

Before setting up credential roaming, one thing that we have found is that it’s best if you clean up unused user profiles from workstations and servers. If you are running Windows XP/2003 we recommend that you use DelProf.exe to clean up old or unused user profiles. This can be deployed via System Center or other management applications. If you do not have this ability you can use DelProf with a computer startup script. When a user starts to logon to multiple computers in the environment if they already have an existing profile on the computer they will start collecting old certificates and DPAPI master keys which will cause your Active Directory database size to grow. If you are using Windows Vista SP1 / 2008 or later there is a group policy setting that can be implemented in your environment that will delete older user profiles based on the number of days:

Computer Configuration\Policies\Administrative Templates\System\User Profiles\Delete user profiles older than a specified number of days on system restart

Client Side Problems

Profiles

It is also necessary to determine if you are using roaming profiles or folder redirection in your environment. In a large enterprise with many administrators this could prove to be a daunting task. It will be necessary to modify these policies to configure an exclusion list in Group Policy for the roaming folders required for credential roaming.

User Configuration\Administrative Templates\System\User Profiles\Exclude directories in roaming profile: Enabled

Windows XP/2003:

Application Data\Microsoft\Crypto
Application Data\Microsoft\Protect
Application Data\Microsoft\SystemCertificates

Windows Vista and higher:

AppData\Roaming\Microsoft\Credentials
AppData\Roaming\Microsoft\Crypto
AppData\Roaming\Microsoft\Protect
AppData\Roaming\Microsoft\SystemCertificates

The CollectUserStore.vbs script that is linked at the bottom of this post will detect if the folders have not been excluded as required and will report the following:

User Profile type is: Roaming
ERROR: You are not excluding the following directory from your roaming profile: Application Data\Microsoft\Crypto
ERROR: You are not excluding the following directory from your roaming profile: Application Data\Microsoft\Protect
ERROR: You are not excluding the following directory from your roaming profile: Application Data\Microsoft\SystemCertificates

Credential roaming will not work properly when folder redirection has been enabled for “Application Data” when running Windows XP or Windows Server 2003. The user can face unexpected results if the previously mentioned folders are being redirected to file servers. This issue does not occur in Windows 7, Windows Server 2008 R2, Windows Vista, or Windows Server 2008. Folder redirection in these operating systems automatically excludes the listed folders.

You can check the following locations in the registry on the client computer manually. This is where the aforementioned exclusion settings can be verified. You must check both locations below because the profile code merges the two locations.

Standard registry location:

HKEY_Current_User\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon
Value name: ExcludeProfileDirs
Data type: REG_SZ

Group Policy registry location:

HKEY_Current_User\SOFTWARE\Policies\Microsoft\Windows\System
Value name: ExcludeProfileDirs
Data type: REG_SZ

Verify the installation of the credential roaming DLL:

Another thing to do prior to implementation is to validate the client side settings before applying the Credential Roaming group policy in test or production. If you have Windows XP or Windows Server 2003 clients you will want to make sure that DIMSNtfy.dll is listed under Winlogon in the registry as illustrated in Figure 1 below. In Windows Vista and later a new dll called dimsjob.dll replaces the dimsnotfy.dll. This new dimsjob.dll as well as dimsroam.dll and pautoenr.dll are loaded within the Task Engine Process as illustrated later.

HKEY_Local_Machine\Software\Microsoft\Windows NT\CurrentVersion\Winlogon\Notify\dimsntfy

The dimsntfy key exported –

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon\Notify\dimsntfy
"Asynchronous"=dword:00000001
"DllName"=%SystemRoot%\system32\dimsntfy.dll”
"Startup"="WlDimsStartup"
"Shutdown"="WlDimsShutdown"
"Logon"="WlDimsLogon"
"Logoff"="WlDimsLogoff"
"StartShell"="WlDimsStartShell"
"Lock"="WlDimsLock"
"Unlock"="WlDimsUnlock"

We are looking for the existence and values for Winlogon Notification packages specifically the dimsntfy registry key and the values underneath it. If this key is missing, certificates are not being downloaded or uploaded from AD. It also means that the hotfix 907247 was not installed.

The CollectUserStore.vbs script will look for the existence of the DLLName value.

The error below will be displayed if the credential roaming hotfix has not been applied:

ERROR: The Credential Roaming Hotfix has not been applied or is not registered with Winlogon

On Vista and Windows 7 verify the scheduled task UserTask-Roam for credential roaming is in the ready status:

Verify Group Policy settings are applied as a post configuration check –

Verify that the group policy settings are being applied to the user. Right click - export the following registry keys while logged on as the user:

HKEY_Current_User\Software\Policies\Microsoft\Cryptography\AutoEnrollment

The screenshot below is what you should expect to see on a client machine if the credential roaming group policy settings have applied successfully.

The AutoEnrollmentkey exported –

HKEY_CURRENT_USER\Software\Policies\Microsoft\Cryptography\AutoEnrollment
"AEPolicy"=dword:00000007
"DIMSRoaming"=dword:00000001
"DIMSRoamingTombstoneDays"=dword:0000003c
"DIMSRoamingMaxNumTokens"=dword:000007d0
"DIMSRoamingMaxTokenSize"=dword:0000ffff

After the client side settings have been validated you can begin issuing the Credential Roaming group policy to test computers and verify the policy is being applied successfully.

NOTE: DIMSRoaming registry value might be a value of 0x9 if you have Windows Vista or higher and enabled Stored User name and password roaming. Here we are looking for the group policy keys being applied to the user.

The “CollectUserStore.vbs” script will also collect this information:

Credentials Roaming Group Policy Settings appear OK
DIMSRoaming = 1
DIMSRoamingMaxNumTokens = 2000
DIMSRoamingMaxTokenSize = 65535
DIMSRoamingTombstoneDays = 60

Verify State.dat file creation:

When credential roaming is enabled there are two files created on the client machine. As a post configuration task after Credential Roaming is configured you should verify that there is a state.dat and state.da~ files in the following directory:

• Windows XP/2003: %USERPROFILE%\Local Settings\Application Data\Microsoft\DIMS
• Windows Vista/2008: %LOCALAPPDATA%\Microsoft\DIMS

NOTE: You have to make sure that you can see Hidden and System files. The State.dat file acts like a traffic cop between the client machine and AD. It reconciles the certificates that are present in the user certificate store with the information that is stored in Active Directory. We can verify the last roaming time stamp on the State.dat file which will be modified whenever the credential roaming series of events occur as explained here.

Testing credential roaming:

If you are testing credential roaming before deploying it throughout the environment you DO NOT delete the certificates using certmgr.msc or from Internet Explorer’s internet options - content - security – certificates menu.

This effectively deletes the certificates from the local store. When this is done, now that credential roaming is enabled the certificate is now tombstoned. The settings for “Maximum tombstone credentials lifetime in days” is configured within the .adm template for credential roaming. It is 60 days by default. In Vista/2008 this group policy setting is integrated into the built in security settings under User Configuration. Essentially the certificate is gone because Credential Roaming not only brings the certificates down from Active Directory but also replicates deletions up to Active Directory.

The proper way to test if credential roaming is working optimally is to:

1. Use a test user account instead of a production user account.
2. Backup your certificates by exporting them.
3. Logon as an administrator of the computer and delete the test users’ profile.
4. Logoff as administrator account.
5. Logon again as the test user.
6. Verify the certificates are replicated down to the certificate store.

Active Directory Database Size Growth

Before deploying Group Policy to enable credential roaming for client computers you should plan for the estimated growth of the Active Directory database.

Click here for more information on key sizes and a formula to determine the potential growth of the NTDS.DIT ahead of time. Keep in mind that the use of the word “Token” refers to a certificate private key, certificate public key, DPAPI Master key, and if enabled for Windows Vista each stored user name and password.

Again, make sure that you have deployed at least Vista SP1. Vista RTM is not supported, and had problems with credential roaming that could make your NTDS.DIT endlessly grow.

Taking the necessary preliminary steps to configure credential roaming and taking the time to perform initial testing in a controlled deployment will help to eliminate headaches later on.

Jim Tierney

CollectUserStore.zip