How do I find out what changes are going on in my Active Directory?
Herbert here. Here are some common questions asked by AD Administrators:
- Why has my AD database size increased by 500MB in the last three weeks?
- I see lots of AD replication in Domain Controller monitoring. What are all these changes?
Both symptoms can be severe enough to impair the operations of your AD forest. Here are examples of past occurrences that we tracked down:
312403 Distributed Link Tracking on Windows-based domain controllers
https://support.microsoft.com/default.aspx?scid=kb;EN-US;312403
318774 Removing duplicate and unwanted proxy addresses in Exchange
https://support.microsoft.com/default.aspx?scid=kb;EN-US;318774
940262 The Active Directory database size increases unexpectedly because a Windows Server 2003-based DNS server inappropriately creates several SerialNo objects
https://support.microsoft.com/default.aspx?scid=kb;EN-US;940262
In order to find the cause for the problems, you should find what has changed in the AD database recently. Now Active Directory assigns an "Update Sequence Number" (USN) to each change. These USNs are 64 Bit Integers and are specific to a Domain Controller. The DC GUID and USN together uniquely identify a database change. A USN is both assigned to originating changes and replicated changes. So even for read-only GC content, you see local USNs getting written.
You can use these USNs to identify recent changes in the database of each DC. Each AD Server (includes AD/AM and LDS) has an attribute named “highestCommittedUSN” on its RootDSE object. Here’s an example output from LDP:
...
12> supportedLDAPPolicies: MaxPoolThreads; MaxDatagramRecv; MaxReceiveBuffer; InitRecvTimeout; MaxConnections; MaxConnIdleTime; MaxPageSize; MaxQueryDuration; MaxTempTableSize; MaxResultSetSize; MaxNotificationPerConn; MaxValRange;
1> highestCommittedUSN: 175389104;
4> supportedSASLMechanisms: GSSAPI; GSS-SPNEGO; EXTERNAL; DIGEST-MD5;
...
Based on this number, you can query for the most recently changed Objects using an LDAP query. As an example, I’m using LDIFDE and I’m subtracting 10000 from the “highestCommittedUSN” value seen on RootDSE:
Ldifde /d dc=contoso,dc=com /s contoso-DC1 /r "(usnchanged>=175379104)" /f domain-NC-last-10000-080919.txt
This file now contains the names of the objects that were changed or created recently. The object names give you a hint as to what area of AD you need to look at, but it may not be enough of a clue yet. For users, you can use attributes that tell you about recent changes:
Pwdlastset
Lastlogontimestamp
e.g. pwdLastSet: 129333360374989750
The attributes have a 64 bit time format. You can convert them to readable timestamps using:
C:> w32tm /ntte 129333360374989750
149691 09:20:37.4989750 - 04.11.2010 10:20:37
You can then grep the LDIF export based on this knowledge about this time stamp, e.g.:
C:>w32tm /ntte 129333000000000000
149690 23:20:00.0000000 - 04.11.2010 00:20:00
Command would be something like:
Ldifde /c:" pwdlastset: 129333" domain-NC-last-10000-080919.txt | wc
This would print the list of users and computers who changed their password in the last ten hours. In most cases, these should all considered false positives.
If they are not all new objects (very recent whenCreated attribute), you may want to look at what attributes have been changed. Also, you want to know from which DC the object change is originating.
Maybe the DC that writes all the changes is the primary DC your provisioning system is working against, or it’s a DC you don’t expect to see. To get this information, retrieve the object meta-data using:
repadmin /showobjmeta <DC name> <Object-DN>
The output looks like this:
Loc.USN Originating DC Org.USN Org.Time/Date Ver Attribute
======= =============== ========= =============
...
175389437 HQcontoso-DC1 175389437 2008-09-16 18:12:46 2 name
...
The leftmost column is the local USN; the more interesting fields are to the right, where you see the originating DC information and change time-stamp, attribute version and name. If the version is really high, it could mean excessive updates to this attribute which deserves more investigation.
You should also look out for changes seen for linked attributes (Windows Server 2003 Forest Mode and higher):
Type Attribute Last Mod Time Originating DC Loc.USN Org.USN Ver Distinguished Name
============================================================================
ABSENT member 2008-09-19 15:14:01 HQcontoso-DC1 175384020 175384020 2 CN=test-user1,OU=Test-OU,DC=contoso,DC=com
PRESENT member 2008-09-16 18:22:29 HQcontoso-DC1 175379684 175379684 1 CN=test-user2,OU=Test-OU,DC=contoso,DC=com
Note: High values for USNs will distort the table view.
Many “ABSENT” and high version numbers indicate high activity with linked values. “ABSENT” indicates a deleted link, so you can think of it as a value tombstone. It’s treated just like an object tombstone in the database. During replication it means that the value is deleted from the attribute, in this case a group membership.
Attributes that can contain lots of data deserve special attention. This often applies to attributes containing binary values, including the security descriptor for AD or Exchange, or attributes containing certificates. Note that by default, LDIFDE does not dump “ntSecurityDescriptor”. If any of these attributes show high version numbers or a recent update time stamp on many objects, you should investigate further. It will depend on the attribute on how you investigate the changes, for example for “ntSecurityDescriptor” you can dump it using DSACLS and check out any excess Access Control Entries.
Excessive changes to “ntSecurityDescriptor” are not so much a problem regarding database size because there is single instance storage for these since Windows Server 2003. But they can take lots of replication bandwidth.
The information on objects, attributes and originating DC you collected so far should give you good hints regarding the originator of the changes. If it’s not clear yet, you can enable auditing on successful changes to these attributes to find out the process that is making these changes. It may be necessary to make the attribute viewable in ACL Editor so you can define auditing for it. See the guide in:
296490 How to modify the filtered properties of an object
https://support.microsoft.com/default.aspx?scid=kb;EN-US;296490
But what if there is no pattern evolving while you get the data?
One approach is to repeat the LDIFDE export and reduce the window until you see a pattern. Maybe the problematic changes only happen at certain times of the day, so it would also play a role when you create the export. Or the changes happen on a branch office that only replicates at a certain time of day.
But there are also more naming contexts that may have excessive changes, such as Configuration or the DNS partitions ForestDnsZones and DomainDnsZones, and on GCs. Hopefully, the admins of the other domains are already aware of the excessive changes. This is how you search the whole of the GC data:
Ldifde /d "" /s contoso-DC1 /t 3268 /r "(usnchanged>=175379104)" /f GC-last-10000-080919.txt
Hint: Keep in mind that this query only shows changes for attributes that are present in the GC.
And finally, the problem may not be with existing objects that are changed, but with objects that are deleted and re-created all the time. Deleted objects still take database space for the tombstone, and the new objects cause replication traffic. LDIFDE can include deleted objects in the query when you pass the “/x” option:
Ldifde /d dc=contoso,dc=com /s contoso-DC1 /x /r "(usnchanged>=175379104)" /f domain-NC-last-10000-deleted-080919.txt
If the combined size of the tombstones is a problem, you have to wait until the garbage collection is done before you can reduce the size of the database file using an offline defragmentation. We advise against shortening Tombstone Lifetime for the sole purpose of kicking out these objects earlier. When you have strict replication enabled and replication quarantine is enforced, this shortening TSL to a few days can have a drastic impact on the availability of your Active Directory.
I hope you’re having fun investigating all your ongoing AD changes. I think you’re up to a few interesting findings.
- Herbert Mauerer