Jorge's Quest For Knowledge!

All about Windows Server, ADDS, ADFS & FIM (It Is Just Like An Addiction, The More You Have, The More You Want To Have!)

(2006-05-08) Lingering Objects

Posted by Jorge on 2006-05-08


For those of you who have experienced lingering objects on a number of DCs/GCs will be able to tell that lingering objects are a real PITA. For those who don’t, trust me they are! Why? If you have them on a number of DCs/GCs, it takes quite a lot of manual work to clean them and besides that several issues can occur, like e-mail issues or replication issues or bogus entries in the GAL, that can make you and your users experience loss of functionality.

 

First let’s dive into some technical stuff of how AD works with deleted objects (tombstones)….

When some object is deleted on a DC it is changed into a tombstone. When an object is transformed into a tombstone ALL almost attributes values, except those mandatory and additionally manually configured, are stripped from the deleted object.

NOTE:

  • The values of the following attributes are preserved during a deletion:
    • objectGUID, objectSid, nTSecurityDescriptor and uSNChanged.
    • Additionally on W2K3 SP1 DCs: sIDHistory
  • Additional attributes can be configured to be preserved by:
    • Using ADSIEDIT.MSC and connecting to the schema partition
    • Configure the "searchFlags" property of the attribute which consists of bits, each with a certain meaning! Don’t just change this without knowing what you are doing!!!
    • Enabling the FOURTH bit on the "searchFlags" property preserves the attribute value in the tombstone of the deleted objects.
      • 1st bit: 2^0=1
      • 2nd bit: 2^1=2
      • 3rd bit: 2^2=4
      • 4th bit: 2^3=8 <——this one!

 

After that the new object form (the tombstone) replicates (at least… it should!) to all DCs/GCs. The lifetime of the tombstone within the directory service depends on the "tombstone lifetime period", which is a forest-wide configuration. The value for the "tombstone lifetime period" can be seen on:

  • Object: "CN=Directory Service,CN=Windows NT,CN=Services,CN=Configuration,DC=<domain>,DC=<tld>"
  • Attribute: "tombstoneLifetime"
  • Value:
    • Fresh install of AD with W2K DCs (all SPs): 60 days
    • Upgrading AD with W2K DCs to W2K3 DCs: 60 days
    • Upgrading AD with W2K DCs to W2K3 SP1 DCs: 60 days
    • Fresh install of AD with W2K3 DCs (all SPs): 60 days
    • Upgrading AD with W2K3 DCs to W2K3 SP1 DCs: 60 days
    • Fresh install of AD with W2K3 SP1 DCs (all SPs): 180 days
    • <not set> means: 60 days

 

A process on each DC called the "Garbage Collection Process" runs each 12 hours and checks, amongst other activities, if tombstones are older than the configured "tombstone lifetime period" they are removed from the directory forever! (nothing to worry as this is a normal process and it should work this ways!) That removal does not replicate to others DCs, but each DC on its own does this check.

 

So now let’s look at HOW lingering objects occur!

Lingering objects occur when a DC/GC is not available (e.g. due to network connectivity issues or DNS issues or it has been shutdown or whatever) for more than the “tombstone lifetime period”.

 

Because of that, that particular DC/GC is not able to replicate at least once within the “tombstone lifetime period” to receive the tombstones. If the unavailable/disconnected DC/GC becomes available again after the “tombstone lifetime period” has passed, inconsistencies (may… but probable will) exist between DCs/GCs, which might cause security related issues, replication issues and even loss of functionality.

Whether or not the lingering objects replicate back to the other DCs/GCs depends on how the two defense mechanisms (for W2K3 and later) against lingering objects are configured. W2K DCs only have one defense mechanism, being option 2.

  1. Replication time between DCs exceeds the “tombstone lifetime period”
  2. Strict and loose replication

 

(1): Replication time between DCs exceeds the “tombstone lifetime period”

When a DC/GC does not replicate at least once within the “tombstone lifetime period” with other DCs/GCs, replication is halted with that DC.

This defense is NOT in place when the following registry name is available and set to a value of 1 (one).

This defense is in place when the following registry name is set to a value of 0 (zero) or is not available:

  • Registry Key: HKLMSystemCurrentContro
    lSetServicesNTDSParameters
  • Registry Name: Allow Replication With Divergent and Corrupt Partner
  • Registry Type: REG_DWORD

 

(2): Strict and loose replication

When strict replication is enabled on a DC/GC the inbound replication of lingering objects is halted and reported (event ID 1988 in the Directory Service event log) for the partition that contains the lingering object. When loose replication is enabled on a DC/GC the inbound replication of lingering objects is NOT halted but it is reported (event ID 1388 in the Directory Service event log). Lingering objects can be detected on DCs by either looking for event ID 1388 or event ID 1988 on DCs/GCs using, amogst others, the EventCombMT tool (Resource Kit).

This defense is NOT in place when the following registry name is NOT available or it is set to a value of 0 (zero).

This defense is in place when the following registry name is available AND set to a value of 1 (one):

  • Registry Key: HKLMSystemCurrentControlSetServicesNTDSParameters
  • Registry Name: Strict Replication Consistency
  • Registry Type: REG_DWORD

 

When STRICT replication is ENABLED:

  • Event ID 1988 is registered in the Directory Service event log of the DC NOT containing the lingering (while replication of the lingering object was prevented), which specifies:
    • The transport-specific network address of the "source DC" (the DC containing the lingering object)
      • Example: ed0c6501-28c1-47e9-b3db-5dcf281e9e31._msdcs.ADCORP.LAN)
    • The "distinguishedName" of the "lingering object"
      • Example: CN=JORGE DE ALMEIDA PINTO,OU=Users,OU=ORG,DC=ADCORP,DC=LAN
    • The "objectGUID"  of the "lingering object"
      • Example: a46ccb2a-2214-475e-b59a-c2aa3b357196
  • This event ID will only be logged on the DC when an attempt is made to inbound replicate the lingering object from the DC containing the lingering object. That attempt will be triggered when a changed is made on the lingering object on the DC containing the lingering object.

 

When LOOSE replication is ENABLED:

  • Event ID 1388 is registered in the Directory Service event log of the DC which previoudly did NOT contain the lingering (while replication of the lingering object was NOT prevented), which specifies:
    • The transport-specific network address of the "source DC" (the DC containing the lingering object)
      • Example: ed0c6501-28c1-47e9-b3db-5dcf281e9e31._msdcs.ADCORP.LAN)
    • The "distinguishedName" of the "lingering object"
      • Example: CN=JORGE DE ALMEIDA PINTO,OU=Users,OU=ORG,DC=ADCORP,DC=LAN
    • The "objectGUID"  of the "lingering object"
      • Example: a46ccb2a-2214-475e-b59a-c2aa3b357196
    • The "Directory partition" containing the lingering object(s)
      • Example: DC=ADCORP,DC=LAN
    • The "Destination highest property USN" (the highest commited USN on the DC NOT containing the lingering object)
      • Example: USN: 14221
  • This event ID will only be logged on the DC when an attempt is made to inbound replicate the lingering object from the DC containing the lingering object. That attempt will be triggered when a changed is made on the lingering object on the DC containing the lingering object.

To DETECT lingering objects (but not remove):

  • STRICT replication consistency MUST be enabled for the following to work.
  • Will NOT work with LOOSE replication consistency as the data between the "DC with lingering objects" and "DC with correct data" is the same
  • REPADMIN /REMOVELINGERINGOBJECTS <FQDN of DC with lingering objects> <objectGUID of DC with correct data> <DN of partition containing lingering objects> /ADVISORY_MODE
    • Example:
      • repadmin /removelingeringobjects BAD-DC.ADCORP.LAN ed0c6601-28c1-47e9-b3db-5dcf291d9e31 DC=ADCORP,DC=LAN /advisory_mode
  • On the DC/GC containing the lingering objects the event IDs 1938 (starting detection summary), 1946 (for each lingering object detected) and 1942 (final detection summary) are registered in the Directory Service event log.

To DETECT and REMOVE lingering objects:

  • STRICT replication consistency MUST be enabled for the following to work.
  • Will NOT work with LOOSE replication consistency as the data between the "DC with lingering objects" and "DC with correct data" is the same
  • REPADMIN /REMOVELINGERINGOBJECTS <FQDN of DC with lingering objects> <objectGUID of DC with correct data> <DN of partition containing lingering objects>
    • Example:
      • repadmin /removelingeringobjects BAD-DC.ADCORP.LAN ed0c6601-28c1-47e9-b3db-5dcf291d9e31 DC=ADCORP,DC=LAN
  • On the DC containing the lingering objects the event IDs 1937 (starting removal summary), 1945 (for each lingering object detected and removed) and 1939 (final removal summary) are registered in the Directory Service event log.

So if STRICT replication is ENABLED, possible ways to detect lingering objects:

  • REPADMIN /REMOVELINGERINGOBJECTS with the /ADVISORY_MODE option
  • Scan DCs for event ID 1988 which contain information about the lingering objects (e.g. with EVENTCOMBMT)
  • For global catalogs use can ALSO use GCCHK from joeware.net (http://www.joeware.net/win/free/tools/gcchk.htm)

So if LOOSE replication is ENABLED, possible ways to detect lingering objects:

  • Scan DCs for event ID 1388 which contain information about the lingering objects (e.g. with EVENTCOMBMT)

 
MY €0.02:

  • Don’t enable LOOSE replication (because of security issues with re-introduced objects that should have been deleted) as this really does not solve the real problem. It only extends it!
  • Be carefull using a mixture of loose replication and strict replic
    ation as that may lead to inconsistencies and/or security issues
  • Enable STRICT replication (=default on W2KSP4 and W2K3)
  • Detect and cleanup lingering objects if applicable
  • Make sure you MONITOR end-to-end AD and SYSVOL replication between DCs!!!
    • Should be a MUST on your priority list!!!
  • Make sure you MONITOR DC/BH/GC/FSMO availability and detect disconnected DCs ASAP!!!
    • Should be a MUST on your priority list!!!
  • Monitoring is a preventive measure that helps you NOT getting the headaches you may have now concerning lingering objects!

To make things a bit easier for those that experience lingering objects on W2K3 DCs:

I have created some scripts (see the zip file at the end of the post where it says ‘attachments’) that automate this a bit more. Use them as you wish and at your own risk!!! Get there scripts from HERE.

The logic works as follows:

  • For EACH domain in the forest specify in ‘DC-LIST-FQDN-HEALTHY.TXT’ ONE DC that is considered HEALTHY
  • Specify in ‘DC-LIST-FQDN-UNHEALTHY.TXT’ ALL DCs/GCs that are considered UNHEALTHY and may contain lingering objects
  • Execute ‘GET-GUID-NCs-FROM-FQDN.CMD’ to generate:
    • ‘DC-LIST-GUID-FQDN-HEALTHY.TXT’
      • –> contains the GUID and FQDN of the HEALTHY DCs
    • ‘<FQDN UNHEALTHY DC>-NCs.TXT’
      • –> For each unhealthy DC contains ALL writable and read-only NCs (partitions)
  • Execute ‘CLEAN-LINGERING-OBJECTS-ON-UNHEALTHY-DCs.CMD’
    • In detect mode (first argument = DETECT) (argument = case sensitive)
    • In clean mode (first argument = CLEAN) (argument = case sensitive)
    • ‘CLEAN-LINGERING-OBJECTS-ON-UNHEALTHY-DCs_DETECT.TXT’
      • –> contains ALL detected lingering objects
    • ‘CLEAN-LINGERING-OBJECTS-ON-UNHEALTHY-DCs_CLEAN.TXT’
      • –> contains ALL detected AND cleaned lingering objects

Were these scripts useful? Just say: Thanks!

Let me know what you think by leaving comments behind

Enjoy!

### DISCLAIMER ###

  • These scripts are freeware. Use it as you wish at your own risk.
  • I do not warrant these scripts to be fit for any purpose or use
  • I do not guarantee that it will not damage or destroy your system
  • Make sure you test these scripts FIRST in a test environment!
  • If you don’t know how to use the scripts DON’T USE them!

 

More and additional information about lingering objects can be found in the following Microsoft articles:

The following has been copied from the URL below, for those who are interested to know how the "/removelingeringobjects" works.

http://technet2.microsoft.com/WindowsServer/en/Library/1465d773-b763-45ec-b971-c23cdc27400e1033.mspx


RemoveLingeringObjects Implementation

When you run repadmin /removelingeringobjects, the tool performs the following steps to compare the directories of the source and destination domain controllers and log (or remove) any found lingering objects:

  1. Check to ensure that the directory partition and the source domain controller are valid.
  2. Verify that the user has the DS-Replication-Manage-Topology extended right on the directory partition container object specified in <NC>. This extended right is required to verify object state between two domain controllers. Members of the Domain Admins group have this right by default.
  3. Ensure that both source and destination use the same objects for comparison by merging the up-to-dateness vectors to filter out any objects that have not replicated from the source to the destination or from the destination to the source. This check rules out a lingering object on the destination if the destination has not received the tombstone from the source, and vice versa. Any such nonreplicated objects are removed from the comparison.
  4. Create the list of object GUIDs for each domain controller to be compared. Examine the metadata of each object and use the merged up-to-dateness vector to determine whether the object should be present on both source and destination.
  5. For each GUID that is in the list for the destination, determine if it is in the list of GUIDs for the source.
  6. If a GUID is not found on the source, the object is identified to be outdated on the destination and is either displayed or deleted on the destination server. If advisory mode has been specified, the GUID is displayed only.

Cheers,
Jorge
———————————————————————————————
* This posting is provided "AS IS" with no warranties and confers no rights!
* Always evaluate/test yourself before using/implementing this!
* DISCLAIMER:
http://jorgequestforknowledge.wordpress.com/disclaimer/
———————————————————————————————
############### Jorge’s Quest For Knowledge #############
#########
http://JorgeQuestForKnowledge.wordpress.com/ ########
———————————————————————————————

About these ads

7 Responses to “(2006-05-08) Lingering Objects”

  1. tomek said

    Cool Jorge,
    Great post as always – I’ve read it with interest – nothing more to add :), except that in Windows 2000 removing lingeing objects may be even more painfull because lack of tools support.

    If lingering objects are present on one of few DCs sometimes it’s easier to get rid of these DCs and repromote them (dcpromo down and up) then clean up these objects, but it always useful to get to the cause which caused such DC to be out of replication for such long time.

    And last – people are often mixing sibiling objects (i.e. conflicting objects) with lilngering objects, I don’t know why but this is common mistake in the field.

    My .02 PLN (which is around .0052 EUR :) )

  2. Note: The scripts check for the presence of adfind.exe in the path. The scripts however fail to check if the file is in the current directory.

    So, either put the exe “adfind.exe” in a directory that is in your path, add the current directory to your windows path, or modify the CMD file to bypass this check (or to include a check in the current directory).

    Perhaps:

    :: Locate critical executables within the PATH variable
    IF NOT EXIST “adfind.exe” (
    FOR %%e IN (find.exe adfind.exe) DO (
    SET PATHLOC=”%%~$PATH:e”
    IF “!PATHLOC!”==”””” (
    ECHO.
    ECHO ERROR – Required executable, “%%e”, not located within the path!
    ECHO ERROR – Make sure the executable is located within one of the directories in the PATH variable!
    ECHO ERROR – PATH VARIABLE = %PATH%
    GOTO END
    )
    )
    )

    Although it might be just as easy to have it include the current directory as part of the path search. Your results may vary and I am in no way tryingto say this is the best solution, only one that seems to work. Remember, there are two CMD files that need to be updated should you choose this route (and this is per version Get-Guid ver 2 and Clean-Clingering ver 3).

    Jorge: If you consider this update of value, would you consider making an update to the scripts?

    Jean-Claude

  3. To finish/correct a prior post:

    Domain Controllers are perfect virtualization targets, but virtualizing a Domain Controller reintroduces

    As long as you follow the Microsoft documentation on not virtualizing machines with FSMO roles, like PDC emulator.

    Care should be taken here as DC’s tend to run low CPU utilization so that they have a quick response time.

  4. I goofed in my post earlier:

    the line should read:
    if not exist adfind.exe (
    not
    if not exist(adfind.exe) (

  5. I have a lingering object that is in the DELETED OBJECTS CN so I can’t seem to delete that, is there anyway to get rid of it? Will it disappear at the end of the Tombstone life? I can’t demote the DC as it won’t write the changes to the Primary GC, please help!

  6. Singh said

    Very informative article sir…Great publish!!

    Singh

  7. [...] http://jorgequestforknowledge.wordpress.com/2006/05/08/lingering-objects-2/ [...]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

 
%d bloggers like this: