Active Directory: Lingering Objects
What are lingering objects?
There are some specific concepts in active directory which may put your environment in trouble if you do not attend to fix them as soon as you notice. One of them is lingering objects. Defining the meaning of Lingering Objects (LO) is not difficult. Basically, if an object in your active directory partitions exist in one or more domain controllers and not exist in the rest of the domain controllers in the same partition. So you may ask yourself how that is possible to have an object in a DC and not having the same object in another DC? So what is the responsibility of replication? Wasn’t it designed in order to have a synchronized AD database in your environment? We are going to cover your questions as well.
How are they created?
The main reason behind creation of LOs is an object removal. Let’s have a look at the below pictures to better understand how they are created.
- We have three domain controllers in our domain. An administrator creates an account ‘Ed Price’ and it is automatically replicated to all the DCs.
- We assume DC3 experience a server problem. By server problem we mean different reasons such as hardware failures, lack of network connectivity for a long time and etc.
- While DC3 is not functional, User ‘Ed Price’ is deleted from DC2 and the change is replicated to all the DCs. So DC2 will remove ‘Ed Price’ from its database but replication traffic will not receive by DC3 due to server problem.
- In this step we assume the ‘Ed Price’ has been tombstoned and as a result it is completely deleted from the environment. Notice that ‘Ed Price’ still exist in DC3 ever after the object is tombstoned.
- After DC3 is brought up and becomes functional it tries to replicate to its partners. That is the time where LO’s detected. DC1 simply says “Ed Price? Hmmm… I don’t have him in my database. Sorry I can’t accept this via replication”. This behavior is because of Strict Replication which is set on every domain controller. Strict replication does not decline replicating an object if it does not have the actual object in its database.
What are the symptoms? When they show themselves?
They mostly present themselves in event viewer and repadmin messages after running /syncall or other related commands. For example in a scenario where we have 3 DCs, you notice two of the DCs can replicate to each other successfully but third DC cannot accept the replication from DC1 and DC2. As a matter of fact, you are experiencing Inbound Replication problems on DC3. For a better understanding take a look at the picture below:
Another factor in the creation of lingering objects is restoring backups. What do you think will happen if you restore an old backup to a DC and some objects in that backup do not exist in the rest of DCs? By old backup, we mean a backup when the creation date is older than the tombstone. The answer is clear. In that case, you will experience lingering objects as well.
If you are curious to find out whether you have lingering objects in your partitions or not, open up Event Viewer and apply a filter for 1388 and 1988 event IDs in Directory Service tree.
How to remove them?
Because removing lingering objects take place in every domain controller, the process of deleting might takes sometimes, depending on the scale of your environment. In order to remove lingering objects, you have to use the repadmin command. It is highly recommended making sure that reported lingering objects in event viewer are real lingering objects and not false positives. If you want to follow the best practice you should go through these two steps to find out how many lingering objects you have:
- Run repadmin with** /ADVISORY_MODE** to report the number of LO’s
- Delete lingering objects
So you want to generate a report to find out how many lingering objects you have in your DC. There is a command which helps you to find lingering objects in your environment. The command is as follows:
Repadmin /removelingeringobjects DC_Containing_LO, DC_Containing_NO_LO, Partition_Of_LO /ADVISORY_MODE
Where in the command you have to follow these rules:
- DC_Containing_LO is the DC which holding the lingering objects. You must enter the domain controller name instead of DC_Containing_LO.
- DC_Containing_NO_LO is a source domain controller with correct data and no lingering objects. Note that you must enter the GUID instead of name. Refer to this link if are not aware how to find your domain controller's GUID : Determining the Server GUID of a Domain Controller
- Partition_Of_LO is the partition which you want to search for lingering objects. For example: dc=Contoso,DC=Com. Do not forget to run this against all the partitions except Schema.
An example of above command would be something like this:
repadmin /removelingeringobjects DC1.Contoso.com 5200642e-32fe-485c-bc8d-babbd543f029 dc=Contoso,dc=Com /ADVISORY_MODE
Basically in such scenarios you do not know which one of your DCs has no lingering objects and which one contain lingering objects. In that case you have to run the command for every DC and check events on event viewer. After running this command you have to check event viewer for these event IDs:
- Event ID 1938 – This event ID means the querying process for finding LO’s has been started.
- Event ID 1946 – This event will be created for every LO found in DC. It means if you have 100 1946 events you have 100 LO’s on the DC which you ran the command.
- Event ID 1942 – This event ID means the querying process for finding LO’s has been finished.
You run this command on every DC and then check the events on event viewer for mentioned event ids. Since you do not know which DC holding the correct database you have to replace DC_Containing_LO and DC_Containing_NO_LO in the command. For an environment with three DCs it goes like this:
- Repadmin /removelingeringobjects DC1, GUID of DC2, dc=contoso, dc=com /ADVISORY_MODE
- Check events and note them down
- Repadmin /removelingeringobjects **DC1, **GUID of DC3, dc=contoso, dc=com /ADVISORY_MODE
- Check events and note them down
- Repadmin /removelingeringobjects **DC2, **GUID of DC1, dc=contoso, dc=com /ADVISORY_MODE
- Check events and note them down
- Repadmin /removelingeringobjects **DC2, **GUID of DC3, dc=contoso, dc=com /ADVISORY_MODE
- Check events and note them down
- Repadmin /removelingeringobjects **DC3, **GUID of DC1, dc=contoso, dc=com /ADVISORY_MODE
- Check events and note them down
- Repadmin /removelingeringobjects **DC3, **GUID of DC2, dc=contoso, dc=com /ADVISORY_MODE
- Check events and note them down
After completing this process and verifying all the lingering objects, this time you have to run the command without /ADVISORY_MODE parameter. For the last thing, do not forget to search all the partitions except schema in your environment.