Jorge's Quest For Knowledge!

All About Identity And Security On-Premises And In The Cloud – It's Just Like An Addiction, The More You Have, The More You Want To Have!

Archive for the ‘CSExport’ Category

(2019-10-06) Examining Pending Export Deletions In Azure AD Connect

Posted by Jorge on 2019-10-06


If you know FIM/MIM, you also know that Azure AD Connect is based upon that under the hood. As described in Azure AD Connect sync: Prevent accidental deletes, Azure AD Connect allows you to configure a specific threshold that represents a normal/accepted amount of deletions towards Azure AD. Now, if you have a low amount of objects that you need to investigate you can easily click through the Sync Service Manager. But what happens if you need to investigate hundreds or thousands of pending deletions? Try to do that in the Sync Service Manager and you’ll through a loooooot of pain! Are there easier ways to do that? Fortunately, YES! Therefore keep reading.

Now please aware that if the number of deletions is equal to or higher than the deletion threshold it will stop the complete export operation to Azure AD, meaning no adds, updates and deletes to Azure AD! This prevents unintended deletion due to mistakes, bad configurations, etc.

To analyze those deletions, use the next steps.

  1. Logon to the ACTIVE (NON-Staging!) AAD Connect server (To determine the ACTIVE AAD Connect Server, see below!)
  2. Open a PowerShell Command Prompt Window and export the pending exports from the connector space that needs further analysis (see below)
  3. Parse the CS Export file to make it readable (see below) (PowerShell GridView Is Opened AND A CSV File Generated!)
  4. Either use the PowerShell GridView or the CSV to analyze the data being exported!
  5. For objects being deleted check if those still exist in AD and what the state is (see below)

[ad.1] Determine The Active AAD Connect Server

Open a PowerShell Command Prompt Window, and execute:

Import-Module ADSYNC

Get-ADSyncGlobalSettingsParameter | ?{$_.Name -eq "Microsoft.Synchronize.StagingMode"} | Select Name,Value

REMARK: If the VALUE mentions TRUE, then it is the Passive (staging) Server, if the VALUE mentions FALSE or is empty, then it is the Active (Non-Staging) Server

[ad.2] Export The Pending Exports From The Connector Space That Needs Analysis

On The Active AAD Connect Server, open a PowerShell Command Prompt Window, and execute:

CD "C:\Program Files\Microsoft Azure AD Sync\Bin"
$connectorHT = New-Object system.collections.hashtable
Write-Host ""
Write-Host "+++ Available Connectors +++" -ForegroundColor Cyan
$connectorNr = 0
Get-ADSyncConnector | %{
    $connectorNr++
    $connectorName = $null
    $connectorName = $_.Name
    $connectorHT[$connectorNr.ToString()] = $connectorName
    Write-Host "[$connectorNr] – $connectorName" -ForegroundColor Magenta
    Write-Host ""
}
$chosenConnectorNr = $null
$chosenConnectorNr = Read-host "Please Choose The Connector By Typing Its Number"

$chosenConnectorName = $null
$chosenConnectorName = $connectorHT[$chosenConnectorNr]
$datetime = Get-Date -Format "yyyy-MM-dd_HH.mm.ss"
$csExportXMLFilepath = Join-Path "C:\TEMP" $($datetime + "_CS-" + $chosenConnectorName + "_PendingExports.xml")
$csExportCMD = ".\CSEXPORT.EXE `"$chosenConnectorname`" `"$csExportXMLFilepath`" /f:x"
Invoke-Expression $csExportCMD
Write-Host ""
Write-Host "Export File…….: $csExportXMLFilepath" -ForegroundColor Cyan
Write-Host ""

[ad.3] Parse The CS Export XML File

On The Active AAD Connect Server, open a PowerShell Command Prompt Window, and execute:

CD "<Folder With Script>"

$csExportCSVFilepath = $csExportXMLFilepath.TrimEnd(".xml")

.\Parse-CS-Export-XML-To-CSV.ps1 -outToAll -sourceXMLfilePaths $csExportXMLFilepath -targetFilePath $csExportCSVFilepath

REMARK: the GridView will be opened automatically!

image

Figure 1: Results After Parsing The XML File(s) To A CSV

In the GridView or Excel, any value added or deleted, will be specified as such. Unchanged values are not listed

image

Figure 2: GridView Sample Output

image

Figure 3: GridView Sample Output

image

Figure 4: GridView Sample Output

 image

Figure 5: GridView Sample Output

REMARK: To reopen the GridView using the CSV file use the following command:

Import-CSV $($csExportCSVFilepath + ".csv") | Out-Gridview

or

Import-CSV "<CSV File Path>" | Out-Gridview

[ad.5a] Check Deleted USERS Against AD

$csExportCSV = Import-CSV $($csExportCSVFilepath + ".csv")
$objectListUsers = @()
$csExportCSV | ?{$_."Object-Type" -eq "user" -And $_."Ops-Type" -eq "delete"} | %{
    $immutableID = $null
    $immutableID = $_."Source-ID"
     $userPrincipalName = $null
    $userPrincipalName = $_."AD-ID"

    $ldapFilter = $null
    $ldapFilter = "(|(raboADImmutableID=$immutableID)(userPrincipalName=$userPrincipalName))"

    $adObject = $null
    $adObject = Get-ADObject -LDAPFilter $ldapFilter -Server :3268 -Properties *

    $displayName = $null
    $status = $null
    $canonicalName = $null

    If ($adObject) {
        $displayName = $adObject.DisplayName
        $status = If (($adObject.userAccountControl -band 2) -eq "2") {"Disabled"} Else {"Enabled"}
        $canonicalName = $adObject.CanonicalName
    } Else {
        $displayName = "Unavailable"
        $status = "Unavailable"
        $canonicalName = "Unavailable"
    }

    $object = New-Object -TypeName System.Object
    $object | Add-Member -MemberType NoteProperty -Name "immutableID" -Value $immutableID
    $object | Add-Member -MemberType NoteProperty -Name "userPrincipalName" -Value $userPrincipalName
    $object | Add-Member -MemberType NoteProperty -Name "displayName" -Value $displayName
    $object | Add-Member -MemberType NoteProperty -Name "status" -Value $status
    $object | Add-Member -MemberType NoteProperty -Name "canonicalName" -Value $canonicalName
    $objectListUsers += $object
}
$objectListUsers | Out-GridView

REMARK: A Gridview will be opened automatically telling you the status of the object and if it exists in AD

[ad.5b] Check Deleted GROUPS Against AD

$objectListGroups = @()
$csExportCSV | ?{$_."Object-Type" -eq "group" -And $_."Ops-Type" -eq "delete"} | %{
    $immutableID = $null
    $immutableID = $_."Source-ID"
    $domain = $null
     $domain = $($_."AD-ID").SubString(0, $($_."AD-ID").IndexOf("\"))
    $sAMAccountName = $null
    $sAMAccountName = $($_."AD-ID").SubString($($_."AD-ID").IndexOf("\") + 1)
    $ldapFilter = $null
    $ldapFilter = "(|(raboADImmutableID=$immutableID)(sAMAccountName=$sAMAccountName))"
    $adObject = $null
    $adObject = Get-ADObject -LDAPFilter $ldapFilter -Server $domain`:389 -Properties *
    $displayName = $null
    $canonicalName = $null
    If ($adObject) {
        $displayName = $adObject.DisplayName
        $canonicalName = $adObject.CanonicalName
    } Else {
        $displayName = "Unavailable"
        $canonicalName = "Unavailable"
    }
    $object = New-Object -TypeName System.Object
    $object | Add-Member -MemberType NoteProperty -Name "immutableID" -Value $immutableID
    $object | Add-Member -MemberType NoteProperty -Name "sAMAccountName" -Value $sAMAccountName
    $object | Add-Member -MemberType NoteProperty -Name "displayName" -Value $displayName
    $object | Add-Member -MemberType NoteProperty -Name "canonicalName" -Value $canonicalName
    $objectListGroups += $object
}
$objectListGroups | Out-GridView

REMARK: A Gridview will be opened automatically telling you the status of the object and if it exists in AD

[ad.5c] Check Deleted CONTACTS Against AD

Function GuidToEscapedByte($guid) {
    $guidParts = $guid.Split("-")
     $reverse = $guidParts[0].ToCharArray()[($guidParts[0].Length – 1)..0] + $guidParts[1].ToCharArray()[($guidParts[1].Length – 1)..0] + $guidParts[2].ToCharArray()[($guidParts[2].Length – 1)..0]
    $rest = $guidParts[3].ToCharArray() + $guidParts[4].ToCharArray()
    for ($inc =0; $inc -lt $reverse.Length; $inc+=2) {
        $escapedGUID = $escapedGUID + "\" + $reverse[$inc+1] + $reverse[$inc]
    }
    for ($inc =0; $inc -lt $rest.Length; $inc+=2) {
        $escapedGUID = $escapedGUID + "\" + $rest[$inc] + $rest[$inc+1]
    }
    return $escapedGUID
}
$csExportCSV = Import-CSV $($csExportCSVFilepath + ".csv")
$objectListContacts = @()
$csExportCSV | ?{$_."Object-Type" -eq "contact" -And $_."Ops-Type" -eq "delete"} | %{
    $immutableID = $null
    $immutableID = $_."Source-ID"
    $objectGUID = $null
    $objectGUID = (New-Object -TypeName System.Guid -ArgumentList(,(([System.Convert]::FromBase64String($immutableID))))).Guid
    $objectGUIDEscaped = $null
    $objectGUIDEscaped = GuidToEscapedByte $objectGUID
    $mail = $null
    $mail = $_."AD-ID"
     $ldapFilter = $null
    $ldapFilter = "(|(objectGUID=$objectGUIDEscaped)(mail=$mail))"
    $adObject = $null
    $adObject = Get-ADObject -LDAPFilter $ldapFilter -Server :3268 -Properties *
    $displayName = $null
    $canonicalName = $null
    If ($adObject) {
        $displayName = $adObject.DisplayName
         $canonicalName = $adObject.CanonicalName
    } Else {
         $displayName = "Unavailable"
        $canonicalName = "Unavailable"
    }
    $object = New-Object -TypeName System.Object
     $object | Add-Member -MemberType NoteProperty -Name "immutableID" -Value $immutableID
    $object | Add-Member -MemberType NoteProperty -Name "mail" -Value $mail
    $object | Add-Member -MemberType NoteProperty -Name "displayName" -Value $displayName
    $object | Add-Member -MemberType NoteProperty -Name "canonicalName" -Value $canonicalName
    $objectListContacts += $object
}
$objectListContacts | Out-GridView

REMARK: A Gridview will be opened automatically telling you the status of the object and if it exists in AD

Now assuming you have confirmed all deletions are expected, you can lift the threshold or increase its value (temporarily) to allow the sync cycle to succeed! You need an Azure AD Admin Account with the Global Administrator role

  • If needed elevate your account through https://portal.azure.com/ → Privileged Identity Management \ Azure AD Roles \ Global Administrator – Activate
  • On the active AAD Connect server, open a PowerShell Command prompt Window and execute:

$aadAdminCreds=Get-Credential

Get-ADSyncExportDeletionThreshold -AADCredential $aadAdminCreds

Disable-ADSyncExportDeletionThreshold -AADCredential $aadAdminCreds

REMARK: The sync engine maybe synching as you do that and you may receive an error. Just wait until the sync engine finishes.

  • As soon as the sync engine is not executing a sync cycle, execute:

Start-ADSyncCycle -PolicyType Delta

  • As soon as that sync cycle has finished enable the threshold again using the previous value

Enable-ADSyncExportDeletionThreshold -DeletionThreshold <value> -AADCredential $aadAdminCreds

PS: this script also works for Pending Export Deletes in FIM/MIM and the script supports multiple source XML files (each for a different CS) as input files!

Ohhh, and I almost forgot! You can download the script from here! Smile

Cheers,

Jorge

————————————————————————————————————————————————————-
This posting is provided "AS IS" with no warranties and confers no rights!
Always evaluate/test everything yourself first before using/implementing this in production!
This is today’s opinion/technology, it might be different tomorrow and will definitely be different in 10 years!
DISCLAIMER:
https://jorgequestforknowledge.wordpress.com/disclaimer/
————————————————————————————————————————————————————-
########################### Jorge’s Quest For Knowledge ##########################
####################
http://JorgeQuestForKnowledge.wordpress.com/ ###################
————————————————————————————————————————————————————-

Posted in Azure AD Connect, Connector/MA, CSExport, Forefront Identity Manager (FIM) Sync, Microsoft Identity Manager (MIM), PowerShell, Tooling/Scripting, Tools, Windows Azure Active Directory | 1 Comment »

(2013-02-08) Parsing A CSExport Generated XML File Into A Scoped CSV File

Posted by Jorge on 2013-02-08


I needed to research the CSExport output. However, that’s an XML file and not really helpful in diagnosing the data in it. Converting it to a CSV, so that it can be loaded into Excel is much much better. So after some google magic, I found Carol Wapshere’s script. However, that script had two issues I needed to solve. At first it does not support multi-values and I needed to diagnose the proxyAddresses attribute. And secondly, because the Get-Content CMDlet is being used to read the XML file it chokes if the XML file size is too large. The error can be seen below.

SNAGHTML32e8542f

Figure 1: Error Reading An XML File File That’s Too Large With The Get-Content CMDlet

I added logic to the script to also process multi-valued attributes when applicable and instead of using the Get-Content CMDlet, I read the XML file as shown in the script below. I was not able to specify the “MaxCharactersInDcoument” property when using the Get-Content CMDlet. The XML file I was reading was about 500 MB and it contained about 1000000000 characters. The method I use below does not have a limit and allows the configuration “MaxCharactersInDcoument” property if needed.

# Original Script By Carol Wapsphere (http://www.wapshere.com/missmiis/using-powershell-to-parse-a-csexport-file) # Script Rewrite By Jorge de Almeida Pinto (https://jorgequestforknowledge.wordpress.com/) # # Takes an XML file created by CSEXPORT, and produces a CSV file more suitable for opening in Excel. # Supports both single-valued attributes and multi-valued attributes # # Before using the script: # * Define the attributes of interest # * Define the object types of interest. * means every object Param( [Parameter(Mandatory=$true)] [string] $sourceXML, [string] $targetCSV ) # List Of Attributes To Put In The CSV File. CHANGE THIS AS NEEDED! # The First 5 Attributes Are Available For All Connector Spaces $csvHeaderColumns = @("dn","connector-state","connector-type","mv-guid","object-type","givenName","sn","displayName","mail","mailNickname","legacyExchangeDN","proxyAddresses") # Object Types Of Interest $objectTypes = @("user") # Read The Source XML File [System.Xml.XmlDocument] $xmlCSExportDoc = New-Object System.Xml.XmlDocument $xmlCSExportDoc.load($sourceXML) # Check If CSV File Already Exists If (Test-Path $targetCSV) { Remove-Item -Path $targetCSV -Force } # Set The Starting Value For.. $csvHeader = $null # Write The CSV Header To The CSV File ForEach ($csvHeaderColumn In $csvHeaderColumns) { If ($csvHeader -eq $null) { $csvHeader = $csvHeaderColumn } Else { $csvHeader = $csvHeader + "," + $csvHeaderColumn } } Add-Content $targetCSV $csvHeader # Get The Information For The Scoped Objects ForEach ($csObject In $xmlCSExportDoc."cs-objects"."cs-object") { If ($objectTypes -Contains $csObject."object-type" -Or $objectTypes -Contains "*") { $csObjectHashTable = @{} $csObjectHashTable.Add("dn",$csObject."cs-dn") $csObjectHashTable.Add("connector-state",$csObject."connector-state") $csObjectHashTable.Add("object-type",$csObject."object-type") If ($csObject.connector -eq "0") { $csObjectHashTable.Add("connector-type","disconnector") $csObjectHashTable.Add("mv-guid","") ForEach ($csObjectAttribute In $csObject."unapplied-export-hologram".entry.attr) { If ($csObjectAttribute.multivalued -eq "false") { $csObjectHashTable.Add($csObjectAttribute.name,$csObjectAttribute.value) } If ($csObjectAttribute.multivalued -eq "true" -And $csObjectAttribute.type -ne "binary") { $multivaluedAttrValues = "" If ($csObjectAttribute.value -ne "" -And $csObjectAttribute.value -ne $null) { ForEach ($value in $csObjectAttribute.value) { If ($multivaluedAttrValues -eq "" -Or $multivaluedAttrValues -eq $null) { $multivaluedAttrValues = $value } Else { $multivaluedAttrValues += ";" + $value } } $csObjectHashTable.Add($csObjectAttribute.name,$multivaluedAttrValues) } Else { $csObjectHashTable.Add($csObjectAttribute.name,"") } } } } Else { $csObjectHashTable.Add("connector-type","connector") $csObjectHashTable.Add("mv-guid",$csObject."mv-link"."#text") ForEach ($csObjectAttribute In $csObject."synchronized-hologram".entry.attr) { If ($csObjectAttribute.multivalued -eq "false") { $csObjectHashTable.Add($csObjectAttribute.name,$csObjectAttribute.value) } If ($csObjectAttribute.multivalued -eq "true" -And $csObjectAttribute.type -ne "binary") { $multivaluedAttrValues = "" If ($csObjectAttribute.value -ne "" -And $csObjectAttribute.value -ne $null) { ForEach ($value in $csObjectAttribute.value) { If ($multivaluedAttrValues -eq "" -Or $multivaluedAttrValues -eq $null) { $multivaluedAttrValues = $value } Else { $multivaluedAttrValues += ";" + $value } } $csObjectHashTable.Add($csObjectAttribute.name,$multivaluedAttrValues) } Else { $csObjectHashTable.Add($csObjectAttribute.name,"") } } } } $csvLine = "" ForEach ($csvHeaderColumn in $csvHeaderColumns) { If ($csObjectHashTable.Contains($csvHeaderColumn)) { If ($csvLine -eq "") { $csvLine = "`"" + $csObjectHashTable.Item($csvHeaderColumn) + "`"" } Else { $csvLine += "," + "`"" + $csObjectHashTable.Item($csvHeaderColumn) + "`"" } } Else { If ($csvLine -eq "") { $csvLine = "," } Else { $csvLine += "," } } } Add-Content $targetCSV $csvline } }

When using the script you need to make adjustments to it before actually using it. This is shown below.

image

Figure 2: Configuring The Object Types And The Attributes To Be Parsed Into The CSV File

[1] you need to define the attribute list you want to parse in an array. The first 5 attributes are available for all object types in every connector.

[2] you need to define the object types you want to parse in an array.

To execute the script, see below.

image

Figure 3: Using The Script

.\FIM-Sync-Engine-Parse-CSExport.ps1 -sourceXML <Source XML> -targetCSV <Target CSV>

The CSV file in Excel should then look similar to

image

Figure 4: The CSV File In Excel

Happy diagnosing!

Cheers,

Jorge

———————————————————————————————

* This posting is provided "AS IS" with no warranties and confers no rights!

* Always evaluate/test yourself before using/implementing this!

* DISCLAIMER: https://jorgequestforknowledge.wordpress.com/disclaimer/

———————————————————————————————

############### Jorge’s Quest For Knowledge #############

######### http://JorgeQuestForKnowledge.wordpress.com/ ########

———————————————————————————————

Posted in CSExport, Forefront Identity Manager (FIM) Sync, PowerShell, Tooling/Scripting | 4 Comments »

 
%d bloggers like this: