(2006-05-08) Lingering Objects

UPDATE 2017-01-07: The information below is still valuable. Somewhere at 3/4 of the blog post a link is available to get scripts to clean lingering objects. However, someone else at Microsoft build a tool the Lingering Objects Liquidator. More info through the following links:

• Remove Lingering Objects that cause AD Replication error 8606 and friends
• Description of the Lingering Object Liquidator tool (LOL)
• Download link to LOL (if this fails check out description on how to download in one of the previous links)

————————-

For those of you who have experienced lingering objects on a number of DCs/GCs will be able to tell that lingering objects are a real PITA. For those who don’t, trust me they are! Why? If you have them on a number of DCs/GCs, it takes quite a lot of manual work to clean them and besides that several issues can occur, like e-mail issues or replication issues or bogus entries in the GAL, that can make you and your users experience loss of functionality.

 

First let’s dive into some technical stuff of how AD works with deleted objects (tombstones)….

When some object is deleted on a DC it is changed into a tombstone. When an object is transformed into a tombstone ALL almost attributes values, except those mandatory and additionally manually configured, are stripped from the deleted object.

NOTE:

• The values of the following attributes are preserved during a deletion:
  • objectGUID, objectSid, nTSecurityDescriptor and uSNChanged.
  • Additionally on W2K3 SP1 DCs: sIDHistory
• Additional attributes can be configured to be preserved by:
  • Using ADSIEDIT.MSC and connecting to the schema partition
  • Configure the "searchFlags" property of the attribute which consists of bits, each with a certain meaning! Don’t just change this without knowing what you are doing!!!
  • Enabling the FOURTH bit on the "searchFlags" property preserves the attribute value in the tombstone of the deleted objects.
    • 1st bit: 2^0=1
    • 2nd bit: 2^1=2
    • 3rd bit: 2^2=4
    • 4th bit: 2^3=8 <——this one!

 

After that the new object form (the tombstone) replicates (at least… it should!) to all DCs/GCs. The lifetime of the tombstone within the directory service depends on the "tombstone lifetime period", which is a forest-wide configuration. The value for the "tombstone lifetime period" can be seen on:

• Object: "CN=Directory Service,CN=Windows NT,CN=Services,CN=Configuration,DC=<domain>,DC=<tld>"
• Attribute: "tombstoneLifetime"
• Value:
  • Fresh install of AD with W2K DCs (all SPs): 60 days
  • Upgrading AD with W2K DCs to W2K3 DCs: 60 days
  • Upgrading AD with W2K DCs to W2K3 SP1 DCs: 60 days
  • Fresh install of AD with W2K3 DCs (all SPs): 60 days
  • Upgrading AD with W2K3 DCs to W2K3 SP1 DCs: 60 days
  • Fresh install of AD with W2K3 SP1 DCs (all SPs): 180 days
  • <not set> means: 60 days

 

A process on each DC called the "Garbage Collection Process" runs each 12 hours and checks, amongst other activities, if tombstones are older than the configured "tombstone lifetime period" they are removed from the directory forever! (nothing to worry as this is a normal process and it should work this ways!) That removal does not replicate to others DCs, but each DC on its own does this check.

 

So now let’s look at HOW lingering objects occur!

Lingering objects occur when a DC/GC is not available (e.g. due to network connectivity issues or DNS issues or it has been shutdown or whatever) for more than the “tombstone lifetime period”.

• Fixing Replication DNS Lookup Problems (Event IDs 1925, 2087, 2088)
  • http://technet2.microsoft.com/WindowsServer/en/Library/43e6f617-fb49-4bb4-8561-53310219f9971033.mspx
• Fixing Replication Connectivity Problems (Event ID 1925)
  • http://technet2.microsoft.com/WindowsServer/en/Library/7fcaa311-bc19-479d-9a4e-179704dfe08f1033.mspx?mfr=true
• Fixing Replication Topology Problems (Event ID 1311)
  • http://technet2.microsoft.com/WindowsServer/en/Library/062e8eaa-27e0-4c5e-bc2b-2913ecce24b81033.mspx

 

Because of that, that particular DC/GC is not able to replicate at least once within the “tombstone lifetime period” to receive the tombstones. If the unavailable/disconnected DC/GC becomes available again after the “tombstone lifetime period” has passed, inconsistencies (may… but probable will) exist between DCs/GCs, which might cause security related issues, replication issues and even loss of functionality.

Whether or not the lingering objects replicate back to the other DCs/GCs depends on how the two defense mechanisms (for W2K3 and later) against lingering objects are configured. W2K DCs only have one defense mechanism, being option 2.

1. Replication time between DCs exceeds the “tombstone lifetime period”
2. Strict and loose replication

 

(1): Replication time between DCs exceeds the “tombstone lifetime period”

When a DC/GC does not replicate at least once within the “tombstone lifetime period” with other DCs/GCs, replication is halted with that DC.

This defense is NOT in place when the following registry name is available and set to a value of 1 (one).

This defense is in place when the following registry name is set to a value of 0 (zero) or is not available:

• Registry Key: HKLMSystemCurrentContro
  lSetServicesNTDSParameters
• Registry Name: Allow Replication With Divergent and Corrupt Partner
• Registry Type: REG_DWORD

 

(2): Strict and loose replication

When strict replication is enabled on a DC/GC the inbound replication of lingering objects is halted and reported (event ID 1988 in the Directory Service event log) for the partition that contains the lingering object. When loose replication is enabled on a DC/GC the inbound replication of lingering objects is NOT halted but it is reported (event ID 1388 in the Directory Service event log). Lingering objects can be detected on DCs by either looking for event ID 1388 or event ID 1988 on DCs/GCs using, amogst others, the EventCombMT tool (Resource Kit).

This defense is NOT in place when the following registry name is NOT available or it is set to a value of 0 (zero).

This defense is in place when the following registry name is available AND set to a value of 1 (one):

• Registry Key: HKLMSystemCurrentControlSetServicesNTDSParameters
• Registry Name: Strict Replication Consistency
• Registry Type: REG_DWORD

 

When STRICT replication is ENABLED:

• Event ID 1988 is registered in the Directory Service event log of the DC NOT containing the lingering (while replication of the lingering object was prevented), which specifies:
  • The transport-specific network address of the "source DC" (the DC containing the lingering object)
    • Example: ed0c6501-28c1-47e9-b3db-5dcf281e9e31._msdcs.ADCORP.LAN)
  • The "distinguishedName" of the "lingering object"
    • Example: CN=JORGE DE ALMEIDA PINTO,OU=Users,OU=ORG,DC=ADCORP,DC=LAN
  • The "objectGUID"  of the "lingering object"
    • Example: a46ccb2a-2214-475e-b59a-c2aa3b357196
• This event ID will only be logged on the DC when an attempt is made to inbound replicate the lingering object from the DC containing the lingering object. That attempt will be triggered when a changed is made on the lingering object on the DC containing the lingering object.

 

When LOOSE replication is ENABLED:

• Event ID 1388 is registered in the Directory Service event log of the DC which previoudly did NOT contain the lingering (while replication of the lingering object was NOT prevented), which specifies:
  • The transport-specific network address of the "source DC" (the DC containing the lingering object)
    • Example: ed0c6501-28c1-47e9-b3db-5dcf281e9e31._msdcs.ADCORP.LAN)
  • The "distinguishedName" of the "lingering object"
    • Example: CN=JORGE DE ALMEIDA PINTO,OU=Users,OU=ORG,DC=ADCORP,DC=LAN
  • The "objectGUID"  of the "lingering object"
    • Example: a46ccb2a-2214-475e-b59a-c2aa3b357196
  • The "Directory partition" containing the lingering object(s)
    • Example: DC=ADCORP,DC=LAN
  • The "Destination highest property USN" (the highest commited USN on the DC NOT containing the lingering object)
    • Example: USN: 14221
• This event ID will only be logged on the DC when an attempt is made to inbound replicate the lingering object from the DC containing the lingering object. That attempt will be triggered when a changed is made on the lingering object on the DC containing the lingering object.

To DETECT lingering objects (but not remove):

• STRICT replication consistency MUST be enabled for the following to work.
• Will NOT work with LOOSE replication consistency as the data between the "DC with lingering objects" and "DC with correct data" is the same
• REPADMIN /REMOVELINGERINGOBJECTS <FQDN of DC with lingering objects> <objectGUID of DC with correct data> <DN of partition containing lingering objects> /ADVISORY_MODE
  • Example:
    • repadmin /removelingeringobjects BAD-DC.ADCORP.LAN ed0c6601-28c1-47e9-b3db-5dcf291d9e31 DC=ADCORP,DC=LAN /advisory_mode
• On the DC/GC containing the lingering objects the event IDs 1938 (starting detection summary), 1946 (for each lingering object detected) and 1942 (final detection summary) are registered in the Directory Service event log.

To DETECT and REMOVE lingering objects:

• STRICT replication consistency MUST be enabled for the following to work.
• Will NOT work with LOOSE replication consistency as the data between the "DC with lingering objects" and "DC with correct data" is the same
• REPADMIN /REMOVELINGERINGOBJECTS <FQDN of DC with lingering objects> <objectGUID of DC with correct data> <DN of partition containing lingering objects>
  • Example:
    • repadmin /removelingeringobjects BAD-DC.ADCORP.LAN ed0c6601-28c1-47e9-b3db-5dcf291d9e31 DC=ADCORP,DC=LAN
• On the DC containing the lingering objects the event IDs 1937 (starting removal summary), 1945 (for each lingering object detected and removed) and 1939 (final removal summary) are registered in the Directory Service event log.

So if STRICT replication is ENABLED, possible ways to detect lingering objects:

• REPADMIN /REMOVELINGERINGOBJECTS with the /ADVISORY_MODE option
• Scan DCs for event ID 1988 which contain information about the lingering objects (e.g. with EVENTCOMBMT)
• For global catalogs use can ALSO use GCCHK from joeware.net (http://www.joeware.net/win/free/tools/gcchk.htm)

So if LOOSE replication is ENABLED, possible ways to detect lingering objects:

• Scan DCs for event ID 1388 which contain information about the lingering objects (e.g. with EVENTCOMBMT)

 
MY €0.02:

• Don’t enable LOOSE replication (because of security issues with re-introduced objects that should have been deleted) as this really does not solve the real problem. It only extends it!
• Be carefull using a mixture of loose replication and strict replic
  ation as that may lead to inconsistencies and/or security issues
• Enable STRICT replication (=default on W2KSP4 and W2K3)
• Detect and cleanup lingering objects if applicable
• Make sure you MONITOR end-to-end AD and SYSVOL replication between DCs!!!
  • Should be a MUST on your priority list!!!
• Make sure you MONITOR DC/BH/GC/FSMO availability and detect disconnected DCs ASAP!!!
  • Should be a MUST on your priority list!!!
• Monitoring is a preventive measure that helps you NOT getting the headaches you may have now concerning lingering objects!

To make things a bit easier for those that experience lingering objects on W2K3 DCs:

I have created some scripts (see the zip file at the end of the post where it says ‘attachments’) that automate this a bit more. Use them as you wish and at your own risk!!! Get there scripts from HERE.

The logic works as follows:

• For EACH domain in the forest specify in ‘DC-LIST-FQDN-HEALTHY.TXT’ ONE DC that is considered HEALTHY
• Specify in ‘DC-LIST-FQDN-UNHEALTHY.TXT’ ALL DCs/GCs that are considered UNHEALTHY and may contain lingering objects
• Execute ‘GET-GUID-NCs-FROM-FQDN.CMD’ to generate:
  • ‘DC-LIST-GUID-FQDN-HEALTHY.TXT’
    • –> contains the GUID and FQDN of the HEALTHY DCs
  • ‘<FQDN UNHEALTHY DC>-NCs.TXT’
    • –> For each unhealthy DC contains ALL writable and read-only NCs (partitions)
• Execute ‘CLEAN-LINGERING-OBJECTS-ON-UNHEALTHY-DCs.CMD’
  • In detect mode (first argument = DETECT) (argument = case sensitive)
  • In clean mode (first argument = CLEAN) (argument = case sensitive)
  • ‘CLEAN-LINGERING-OBJECTS-ON-UNHEALTHY-DCs_DETECT.TXT’
    • –> contains ALL detected lingering objects
  • ‘CLEAN-LINGERING-OBJECTS-ON-UNHEALTHY-DCs_CLEAN.TXT’
    • –> contains ALL detected AND cleaned lingering objects

Were these scripts useful? Just say: Thanks!

Let me know what you think by leaving comments behind

Enjoy!

### DISCLAIMER ###

• These scripts are freeware. Use it as you wish at your own risk.
• I do not warrant these scripts to be fit for any purpose or use
• I do not guarantee that it will not damage or destroy your system
• Make sure you test these scripts FIRST in a test environment!
• If you don’t know how to use the scripts DON’T USE them!

 

More and additional information about lingering objects can be found in the following Microsoft articles:

• Fixing Replication Lingering Object Problems (Event IDs 1388, 1988, 2042)
  • http://technet2.microsoft.com/WindowsServer/en/Library/4a1f420d-25d6-417c-9d8b-6e22f472ef3c1033.mspx
• Event ID 1388 or 1988: A lingering object is detected
  • http://technet2.microsoft.com/WindowsServer/en/Library/77dbd146-f265-4d64-bdac-605ecbf1035f1033.mspx
• Event ID 2042: It has been too long since this machine replicated
  • http://technet2.microsoft.com/WindowsServer/en/Library/34c15446-b47f-4d51-8e4a-c14527060f901033.mspx
• A deleted account remains in the Address Book, e-mail is not received, or a duplicate account exists
  • http://technet2.microsoft.com/WindowsServer/en/Library/9b1c2595-4fe2-457a-8868-a9025a307c631033.mspx
• Outdated Active Directory objects generate event ID 1988 in W2K3
  • http://support.microsoft.com/?id=870695
• Lingering objects may remain after you bring an out-of-date global catalog server back online
  • http://support.microsoft.com/?id=314282
  • –> This procedure works for both W2K and W2K3 DCs!!!
• Information about lingering objects in a Windows 2000-based forest or in a Windows Server 2003-based forest
  • http://support.microsoft.com/?id=910205

The following has been copied from the URL below, for those who are interested to know how the "/removelingeringobjects" works.

http://technet2.microsoft.com/WindowsServer/en/Library/1465d773-b763-45ec-b971-c23cdc27400e1033.mspx

RemoveLingeringObjects Implementation

When you run repadmin /removelingeringobjects, the tool performs the following steps to compare the directories of the source and destination domain controllers and log (or remove) any found lingering objects:

1. Check to ensure that the directory partition and the source domain controller are valid.
2. Verify that the user has the DS-Replication-Manage-Topology extended right on the directory partition container object specified in <NC>. This extended right is required to verify object state between two domain controllers. Members of the Domain Admins group have this right by default.
3. Ensure that both source and destination use the same objects for comparison by merging the up-to-dateness vectors to filter out any objects that have not replicated from the source to the destination or from the destination to the source. This check rules out a lingering object on the destination if the destination has not received the tombstone from the source, and vice versa. Any such nonreplicated objects are removed from the comparison.
4. Create the list of object GUIDs for each domain controller to be compared. Examine the metadata of each object and use the merged up-to-dateness vector to determine whether the object should be present on both source and destination.
5. For each GUID that is in the list for the destination, determine if it is in the list of GUIDs for the source.
6. If a GUID is not found on the source, the object is identified to be outdated on the destination and is either displayed or deleted on the destination server. If advisory mode has been specified, the GUID is displayed only.

Cheers,
Jorge
———————————————————————————————
* This posting is provided "AS IS" with no warranties and confers no rights!
* Always evaluate/test yourself before using/implementing this!
* DISCLAIMER: https://jorgequestforknowledge.wordpress.com/disclaimer/
———————————————————————————————
############### Jorge’s Quest For Knowledge #############
######### http://JorgeQuestForKnowledge.wordpress.com/ ########
———————————————————————————————