Jorge's Quest For Knowledge!

All You Need To Know About Identity And Security On-Premises And In The Cloud. It's Just Like An Addiction, The More You Have, The More You Want To Have!

(2013-02-08) Parsing A Pending Exports Drop File And Viewing Changes Offline

Posted by Jorge on 2013-02-08


The following is quite a good post about parsing a pending exports drop file and viewing it in the PowerShell GridView.

SOURCE: FIM – CS Export XML Parser

<QUOTE SOURCE=”FIM – CS Export XML Parser”>

When implementing a new management agent or performing a mass change in FIM, I always want to be sure that any object exports are identified and intentional for all add / updates / deletes. The Sync engine does a great job at showing this for a small amount of objects. But what if you have hundreds, or even thousands object adds/updates/deletes? What if you wanted to validate that a specific attribute or object isn’t getting modified? How can we extract out that information into a sortable/manageable form? Looking around on the forums and online, I’ve seen tools to report the statistics, but not the actual data. The closest I’ve seen was from http://www.wapshere.com/missmiis/using-powershell-to-parse-a-csexport-file

Say, I want to be able to see the following below but in a more manageable form.

image

Figure 1: Pending Exports (Adds, Modifies And Deletes)

image

Figure 2: Pending Export (Add) Of A Specific Objects

(And, say I want to see what’s happening for the 102 updates, 1 adds, and 1 deletes)

image

Figure 3: Run Profile Statistics For The “Export (Drop File And Stop Run)” Run Profile

At first I looked around to see what FIM was capable of in reporting this information. Seems there were 2 main options. Either the Management Agent’s Drop File & Stop on Export or the CSExport.exe utility that comes with the Sync Engine. Both these options creates a XML file, however they are subtly different from each other. Below are the two described in detail.

CSEXPORT.EXE

This is the most powerful tool for exporting data out from the connector space. It comes with the FIM 2010 product and has a nice usage help to export only the information your after. This tool will export all the connector space objects into an XML file containing “tower” holograms. These tower holograms represent what is pending import/export/es-crowed/unapplied, etc..

An example of the command for exporting all pending export data from the FIM Service MA is shown below:

csexport.exe "FIM Service" "pendingExport.xml" /f😡 /o:h

clip_image001

Figure 4: Using CSEXPORT

You can find this utility in the “Bin” folder of where the sync engine service is installed. Depending on the number of objects pending export, this will give you a XML document filtered on the pending exports (similar to searching the connector-space on the MA for Pending Export). The hard part is making meaningful (and accurate) use out of the generated XML document.

This is where I started experimenting, creating modifications in Test and exporting the data to determine how the objects are represented in XML document. Unfortunately there is no documentation I can find on the exact behavior of these holograms (such as what happens when you have an export error, and pending modifications). I ended up writing a c# console XML parser, based off my testing in DEV. I plan on making it more customizable, but for now it seems to work ok. The utility I wrote will take any pending export file as generated from the CSExport.exe program and transform the XML into a CSV format (quote encased, pipe-delimited). The first three columns “ID, ObjectType, OperationType” exist for all objects and should be self explanatory. The rest of the columns show all add/update/delete attributes for any objects being added/updated/deleted. Blank means that it doesn’t apply to the object or it’s not being updated.

The utility can be ran to “-showmodifications” or just display what the values will be updated to. The difference is that the showmodifications will display the “Old Value” as seen in the Sync Engine whenever possible (see image below)

clip_image003

Figure 4: Viewing The CSV Produced By The CSExportParse Utility In ShowModifications Mode (Download Link available later)

*note, there are some side effects with viewing the “-showmodifications” output due to the way line breaks are outputted, I’ll blog more about that later*

When ran without the “-showmodifications” switch the output will be similar to the following image below:

clip_image005

Figure 5: Viewing The CSV Produced By The CSExportParse Utility In Normal Mode (Download Link available later)

As you can see in the images above, I imported the file into excel and can easily filter and group the data to show all changes queued up.

Download: CS Export Parser v1.0

*requires the .NET framework 3.5

Drop File and Stop Run (Export – Run Profile)

At first I decided to give parsing this information in PowerShell from the Log Export a try. I wanted to write a script that would transform the XML file into CSV format which can be sorted, queried, and manipulated in other programs, such as Excel. The result will be a CSV file with at least 3 headers:

  1. “ID” – The connector-space “anchor” or unique identifier in the connector space
  2. “ObjectType” – The type of object (Person,Group,Contact) of the CS object
  3. “OperationType” – Defines what is happening to the object [being created/being updated/being deleted]

All additional header columns listed in the file are the attributes names themselves being added/updated and pertain only to the OpertationType of add / updates. *Note, the max number of columns added is based off the max number of attributes being added/modified from a single object. Also, anytime an object has an attribute value of blank (and the operation is update/add) it means that that particular attribute isn’t going to be modified in target system. Remember this shows what’s being updated, if the value is blank for a field, it doesn’t mean it’s being removed, just that the attribute wasn’t touched for that object.

The XML XML information (in my opinion) is simpler to interpret then the csexport. However, one annoyingly thing I discovered is that objectTypes don’t seem to be defined for any updates or deletes (just adds). Therefore you would need to look back on the object’s ID to determine what type of object it is, this isn’t really the case for the CSExport as you can export almost any information (including the objectType) to build the report. Another limitation is that in my testing for custom management agents ECMA, it shows all update operations as “Replace” and doesn’t give any information as to what is being modified clip_image006

The script below can be used to generate the CSV file. *Note it doesn’t work with the CSExport utility.

Keep in mind that this is a work in progress…

NOTE FROM JORGE: the version of the code below can be used in a PowerShell script. I have been having issues with the script below when large amounts of data need to be read (e.g. 300000 lines) I have to split it in multiple files of for example 40000 lines to be able to read it. I still do not understand why this happens. The original script missed a declaration for $I = 0 in the update section

# http://blog.fractalengine.com/fim-log-export-xml-parser/ # File Name: FIM-Sync-Engine-Parse-Export-Run-Log.ps1 Param( [Parameter(Mandatory=$true)] [string] $filePath, [Switch] $showModifications ) function ProcessExportXML { $global:MainList = @() $global:Headers = @("ID","ObjectType","OperationType") if (!(Test-Path $filePath)) { throw "Invalid path: $filePath" } try { [System.Xml.XmlDocument] $xml = New-Object System.Xml.XmlDocument $xml.Load($filePath) #have to load the namespaces $namespaceURI = $xml.mmsml.xmlns $ns = @{"e"=$namespaceURI} getAdds -xml $xml -ns $ns getDeletes -xml $xml -ns $ns getUpdates -xml $xml -ns $ns -mods $showModifications $report = $global:MainList | select $global:Headers return $report } catch { } } function getUpdates($xml, $ns, $mods) { try { $updates = $xml | select-xml "//e:delta[@operation='update']" -Namespace $ns if ($updates -ne $null) { $totalUPD = $(if ($updates.count -gt 0) { $updates.count;} else { 1; }) $iUPD = 0 $updates | % { $node = $_.Node write-progress -Activity "Processing Updates" -Status $node.dn -PercentComplete ($iUPD / $totalUPD * 100) $obj = New-Object -TypeName System.Object $obj | Add-Member -MemberType NoteProperty -Name "ID" -Value $node.dn $obj | Add-Member -MemberType NoteProperty -Name "ObjectType" -Value $null $obj | Add-Member -MemberType NoteProperty -Name "OperationType" -Value $node.operation #get attributes $node.attr | % { $item = $_ if ($global:Headers -notcontains $item.name) { $global:Headers += $item.name } switch ($item.operation) { "update" { if ($mods) { #need to fix this, not sure how to handle multivalue attributes with modifications if ($item.multivalued -eq $true) { $stringBuilder = "" $valuesAdded = ($item.value | where { $_.operation -eq "add"}) $valuesRemoved = ($item.value | where { $_.operation -eq "delete"}) $valuesRemoved | % { $attributeValue = $_."#text" $stringBuilder += "Removed: [$attributeValue]`r`n" } $valuesAdded | % { $attributeValue = $_."#text" $stringBuilder += "Added: [$attributeValue]`r`n" } $obj | Add-Member -MemberType NoteProperty -Name $item.name -value $stringBuilder.trim() } else { $stringBuilder = "" $old = ($item.value | where { $_.operation -eq "delete"})."#text" $new = ($item.value | where { $_.operation -eq "add"})."#text" $stringBuilder = "Old: [$old]`r`nNew: [$new]" $obj | Add-Member -MemberType NoteProperty -Name $item.name -value $stringBuilder.trim() } } else { if ($item.multivalued -eq $true) { $stringBuilder = "" $valuesAdded = ($item.value | where { $_.operation -eq "add"}) $valuesAdded | % { $attributeValue = $_."#text" $stringBuilder += "$attributeValue;" } $obj | Add-Member -MemberType NoteProperty -Name $item.name -value $stringBuilder.trim() } else { $obj | Add-Member -MemberType NoteProperty -Name $item.name -value (($item.value | where { $_.operation -eq "add" })."#text") } } } "delete" { $obj | Add-Member -MemberType NoteProperty -Name $item.name -value "[deleted]" } "add" { if ($item.multivalued -eq $true) { $stringBuilder = "" $item.value | % { $stringBuilder += "$_;" } $obj | Add-Member -MemberType NoteProperty -Name $item.name -value $stringBuilder } else { $obj | Add-Member -MemberType NoteProperty -Name $item.name -value $item.value } } } } $global:MainList += $obj $iUPD++ } } } catch { write-host "Error! $_ " } } function getAdds($xml,$ns) { try { $adds = $xml | select-xml "//e:delta[@operation='add']" -Namespace $ns if ($adds -ne $null) { $totalADD = $(if ($adds.count -gt 0) { $adds.count;} else { 1; }) $iADD = 0 $adds | % { $node = $_.Node write-progress -Activity "Processing Adds" -Status $node.dn -PercentComplete ($iADD / $totalADD * 100) $obj = New-Object -TypeName System.Object $obj | Add-Member -MemberType NoteProperty -Name "ID" -Value $node.dn $obj | Add-Member -MemberType NoteProperty -Name "ObjectType" -Value $node."primary-objectclass" $obj | Add-Member -MemberType NoteProperty -Name "OperationType" -Value $node.operation #get attributes $node.attr | % { $item = $_ if ($global:Headers -notcontains $item.name) { $global:Headers += $item.name } if ($item.multivalued -eq $true) { $stringBuilder = "" $item.value | % { $stringBuilder += "$_;" } $obj | Add-Member -MemberType NoteProperty -Name $item.name -value $stringBuilder } else { $obj | Add-Member -MemberType NoteProperty -Name $item.name -value $item.value } } $global:MainList += $obj $iADD++ } } } catch { write-host "Error! $_" } } function getDeletes($xml, $ns) { try { $deletes = $xml | select-xml "//e:delta[@operation='delete']" -Namespace $ns if ($deletes -ne $null) { $totalDEL = $(if ($deletes.count -gt 0) { $deletes.count;} else { 1; }) $iDEL = 0 $deletes | % { $node = $_.Node write-progress -Activity "Processing Deletes" -Status $node.dn -PercentComplete ($iDEL / $totalDEL * 100) $obj = New-Object -TypeName System.Object $obj | Add-Member -MemberType NoteProperty -Name "ID" -value $node.dn $obj | Add-Member -MemberType NoteProperty -Name "ObjectType" -Value $null $obj | Add-Member -MemberType NoteProperty -Name "OperationType" -Value $node.operation $global:MainList += $obj $iDEL++ } } } catch { write-host "Error! $_" } } ProcessExportXML

To use this you would simple run the “Log Export” from your MA, then copy this code into a powershell console and specify the filePath to the XML document generated from the MA.

FIM-Sync-Engine-Parse-Export-Run-Log.ps1 -filePath "<Source XML File>" FIM-Sync-Engine-Parse-Export-Run-Log.ps1 -filePath "<Source XML File>" | Out-GridView OR FIM-Sync-Engine-Parse-Export-Run-Log.ps1 -filePath "<Source XML File>" -showModifications FIM-Sync-Engine-Parse-Export-Run-Log.ps1 -filePath "<Source XML File>" -showModifications| Out-GridView

The “ShowModifications” mode will show the “old & new” attribute values in each cell. Once done you can pipe the $report to an Export-csv cmdlet to write it’s contents out to a file, or just a out-gridview to display it within a powershell gridview. This is useful if you plan to use excel to filter and sort the objects.  With the first command you will see something like this

clip_image001[4]

Figure 6: Viewing The Output In PowerShell GridView When Running In Normal Mode

With the “-showModifications” switch it will look more like the following

clip_image002

Figure 7: Viewing The Output In PowerShell GridView When Running In ShowModifications Mode

</QUOTE SOURCE=”FIM – CS Export XML Parser”>

Cheers,

Jorge

———————————————————————————————

* This posting is provided "AS IS" with no warranties and confers no rights!

* Always evaluate/test yourself before using/implementing this!

* DISCLAIMER: https://jorgequestforknowledge.wordpress.com/disclaimer/

———————————————————————————————

############### Jorge’s Quest For Knowledge #############

######### http://JorgeQuestForKnowledge.wordpress.com/ ########

———————————————————————————————

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

 
%d bloggers like this: