Fixing Replication Lingering Object Problems (Event IDs 1388, 1988, 2042)
Applies To: Windows Server 2003, Windows Server 2003 R2, Windows Server 2003 with SP1, Windows Server 2003 with SP2
If a domain controller does not replicate for a period of time that is longer than the tombstone lifetime and the domain controller is then reconnected to the replication topology, objects that were deleted from Active Directory while the domain controller was offline can remain on the domain controller as lingering objects.
Tombstone Lifetime and Replication of Deletions
When an object is deleted, Active Directory replicates the deletion as a tombstone object, which consists of a small subset of the attributes of the deleted object. By inbound-replicating this object, other domain controllers in the domain and forest become aware of the deletion. The tombstone is retained in Active Directory for a specified period called the tombstone lifetime. At the end of the tombstone lifetime, the tombstone is deleted from the directory permanently.
The default value of the tombstone lifetime depends on the version of the operating system that is running on the first domain controller that is installed in a forest, as follows:
Windows 2000 Server or Windows Server 2003: The default value is 60 days.
Windows Server 2003 with Service Pack 1 (SP1): The default value is 180 days.
Note
The tombstone lifetime value that is in effect when a domain controller is upgraded to Windows Server 2003 SP1 is not changed by upgrading. The existing value is maintained until you change it manually.
After the tombstone is removed permanently, the object deletion can no longer be replicated. Therefore, the tombstone lifetime defines how long domain controllers in the forest retain knowledge of a deleted object and thus the time during which a unique deletion must be received by all direct and transitive replication partners of the originating domain controller.
How Lingering Objects Occur
When conditions beyond your control cause a domain controller to be disconnected for a period that is longer than the tombstone lifetime, one or more objects that are deleted from Active Directory on all other domain controllers might remain on the disconnected domain controller. Such objects are called lingering objects. Because the domain controller is offline during the entire time that the tombstone is alive, the domain controller never receives replication of the tombstone.
When it is reconnected to the replication topology, this domain controller acts as a source replication partner that has an object that its destination partner does not have.
Replication problems occur when the object on the source domain controller is updated. In this case, when the destination attempts to inbound-replicate the update, one of the following happens:
If the destination domain controller has strict replication consistency enabled, it recognizes that it cannot update the object and locally halts inbound replication of the directory partition from that source domain controller.
If the destination domain controller has strict replication consistency disabled, it requests the full replica of the updated object. In this case, the object is reintroduced into the directory.
Lingering objects can reside in writable or read-only partitions that are potentially replicated between domain controllers in the same or different domains in the same forest.
Causes of Long Disconnections
Unexpectedly long disconnections can be caused by the following conditions:
A domain controller is left in a storage room and forgotten, or shipment of a prestaged domain controller to its remote location takes longer than a tombstone lifetime.
Replication fails and monitoring is not in place. Failures can occur as follows:
A domain controller is started and connected to the corporate intranet but experiences inbound replication failure as a result of an underlying network connectivity failure, name resolution failure, or authentication failure that prevents replication from occurring.
A bridgehead server is overloaded, and replication becomes backlogged. Excessively high replication load on a global catalog server, in combination with a short intersite replication interval, can result in updates not being replicated.
Note
Global catalog servers replicate read-only replicas of all domain directory partitions in the forest. The replication of read-only replicas has a lower priority than the replication of writable replicas. In addition, global catalog servers are often bridgehead servers, which adds to the replication load. If the replication load on global catalog servers acting as bridgehead servers is too high as a result of an extremely short replication interval, excessive numbers of concurrent outbound replication partners, or a combination of both, the replication queue can become backlogged. If the condition persists, read-only replicas can remain in the queue indefinitely. These conditions can result in lingering objects on a global catalog server.
Wide area network (WAN) connections are unavailable for long periods. For example, a domain controller onboard a cruise ship might be unable to replicate because the ship is at sea for longer than the tombstone lifetime.
The reported event is a false positive because an administrator shortened the tombstone lifetime to force tombstone deletion (garbage collection).
The reported event is a false positive because the system clock on the source or destination domain controller is improperly rolled forward or back in time. Clock skews are most common following a system reboot and can have the following causes:
System clock battery or motherboard problems.
The time source for a computer is improperly configured, including a time source server configured with Windows Time service (W32time), third-party time servers, and network routers.
The system clock is advanced or rolled back by an administrator attempting to extend the useful life of a system state backup or accelerate the garbage collection of deleted objects. Make sure that the system clock reflects the actual time and that event logs do not contain events from the future or invalid past.
Indications That a Domain Controller Has Lingering Objects
An outdated domain controller can store lingering objects with no noticeable effect as long as an administrator, application, or service does not update the lingering object or attempt to create an object with the same name in the domain or with the same user principal name (UPN) in the forest. However, the existence of lingering objects can cause problems, especially if the object is a security principal.
Symptoms Associated with Lingering Objects
The following symptoms indicate that a domain controller has lingering objects:
A deleted user or group account remains in the global address list (GAL) on Exchange servers. Therefore, although the account name appears in the GAL, attempts to send e-mail messages result in errors.
Multiple copies of an object appear in the object picker or GAL for an object that should be unique in the forest. Duplicate objects sometimes appear with altered names, causing confusion on directory searches. For example, if the relative distinguished name of two objects cannot be resolved, conflict resolution appends "*CNF:GUID" to the name, where * represents a reserved character, CNF is a constant that indicates a conflict resolution, and GUID represents the objectGUID attribute value.
E-mail messages are not delivered to a user whose Active Directory account appears to be current. After an outdated domain controller or global catalog server becomes reconnected, both instances of the user object appear in the global catalog. Because both objects have the same e-mail address, e-mail messages cannot be delivered.
A universal group that no longer exists continues to appear in a user’s access token. Although the group no longer exists, if a user account still has the group in its security token, the user might have access to a resource that you intended to be unavailable to that user.
A new object or Exchange mailbox cannot be created, but you do not see the object in Active Directory. An error message reports that the object already exists.
Searches that use attributes of an existing object incorrectly find multiple copies of an object of the same name. One object has been deleted from the domain, but it remains in an isolated global catalog server.
If an attempt is made to update a lingering object that resides in a writable directory partition, events are logged on the destination domain controller. However, if the only version of a lingering object exists in a read-only directory partition on a global catalog server, the object cannot be updated and this type of event will never be triggered.
Registry Setting That Determines Whether Lingering Objects Are Replicated
If a writable lingering object exists in your environment and an attempt is made to update the object, the value in the strict replication consistency registry entry (type REG_DWORD) in HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\NTDS\Parameters determines whether replication proceeds or is stopped, as follows:
1 (enabled): Inbound replication of the specified directory partition from the source is stopped on the destination.
0 (disabled): The destination requests the full object from the source domain controller, and the lingering object is revived in the directory as a new object.
Default Settings for Strict Replication Consistency
The default value for the strict replication consistency registry entry is determined by the conditions under which the domain controller was installed into the forest.
Note
Raising the domain or forest functional level does not change the replication consistency setting on any domain controller.
Strict replication consistency enabled
The value of strict replication consistency on domain controllers that are installed into a forest defaults to enabled (1) under the following conditions:
The forest root domain of a new forest is created by upgrading the Windows NT 4.0 primary domain controller (PDC) to Windows Server 2003 by using the Windows Server 2003 version of Winnt32.exe.
The forest root domain of a new forest is created by installing Active Directory on a server running Windows Server 2003.
Strict replication consistency disabled
The value of strict replication consistency on domain controllers defaults to disabled (0) under the following conditions:
A domain controller running Windows 2000 Server is upgraded to Windows Server 2003.
A server running Windows 2000 Server is promoted into a Windows Server 2003 forest.
If you have a domain controller that is running Windows Server 2003 with SP1, you do not need to edit the registry to set strict replication consistency. Instead, you can use Repadmin to set the value for one or all domain controllers in the forest. To set strict replication consistency for specific domain controllers or for all domain controllers, see Event ID 1388 or 1988: A lingering object is detected.
For more information about strict replication consistency, see "How the Active Directory Replication Model Works" in the Windows Server 2003 Technical Reference on the Microsoft Web site at https://go.microsoft.com/fwlink/?LinkId=27636.
Tool for Removing Lingering Objects
On domain controllers running Windows Server 2003 or Windows Server 2003 with SP1, use Repadmin.exe (in Windows Support Tools) to remove the lingering object or objects. Windows Support Tools are available on the operating system CD in the Support\Tools folder. The version of Repadmin that ships with Windows Server 2003 provides the option /removelingeringobjects, which safely removes instances of lingering objects from both writable directory partitions and read-only directory partitions.
The repadmin /removelingeringobjects command does the following:
Compares the directory database objects on a reference domain controller with the objects on the target domain controller, which contains (or is suspected to contain) lingering objects. There must be connectivity between the reference domain controller and the target domain controller.
Either removes the lingering objects or logs the potential deletions to the Directory Service event log, as follows:
If you use the /advisory_mode parameter, events are logged in the Directory Service event log for the objects that are found.
If you do not use the /advisory_mode parameter, the found objects are deleted without replicating the deletions; that is, the deletions occur only on the target domain controller.
Choose the problem that best describes your situation from the following list, and then step through the suggested fix:
A deleted account remains in the Address Book, e-mail is not received, or a duplicate account exists
Event ID 2042: It has been too long since this machine replicated