The following is quite a good post about parsing a pending exports drop file and viewing it in the PowerShell GridView.
-
SOURCE: FIM – CS Export XML Parser
-
<QUOTE SOURCE=”FIM – CS Export XML Parser”>
When implementing a new management agent or performing a mass change in FIM, I always want to be sure that any object exports are identified and intentional for all add / updates / deletes. The Sync engine does a great job at showing this for a small amount of objects. But what if you have hundreds, or even thousands object adds/updates/deletes? What if you wanted to validate that a specific attribute or object isn’t getting modified? How can we extract out that information into a sortable/manageable form? Looking around on the forums and online, I’ve seen tools to report the statistics, but not the actual data. The closest I’ve seen was from http://www.wapshere.com/missmiis/using-powershell-to-parse-a-csexport-file
Say, I want to be able to see the following below but in a more manageable form.
Figure 1: Pending Exports (Adds, Modifies And Deletes)
-
Figure 2: Pending Export (Add) Of A Specific Objects
-
(And, say I want to see what’s happening for the 102 updates, 1 adds, and 1 deletes)
Figure 3: Run Profile Statistics For The “Export (Drop File And Stop Run)” Run Profile
-
At first I looked around to see what FIM was capable of in reporting this information. Seems there were 2 main options. Either the Management Agent’s Drop File & Stop on Export or the CSExport.exe utility that comes with the Sync Engine. Both these options creates a XML file, however they are subtly different from each other. Below are the two described in detail.
-
CSEXPORT.EXE
This is the most powerful tool for exporting data out from the connector space. It comes with the FIM 2010 product and has a nice usage help to export only the information your after. This tool will export all the connector space objects into an XML file containing “tower” holograms. These tower holograms represent what is pending import/export/es-crowed/unapplied, etc..
An example of the command for exporting all pending export data from the FIM Service MA is shown below:
csexport.exe "FIM Service" "pendingExport.xml" /f 
/o:h
-
Figure 4: Using CSEXPORT
-
You can find this utility in the “Bin” folder of where the sync engine service is installed. Depending on the number of objects pending export, this will give you a XML document filtered on the pending exports (similar to searching the connector-space on the MA for Pending Export). The hard part is making meaningful (and accurate) use out of the generated XML document.
This is where I started experimenting, creating modifications in Test and exporting the data to determine how the objects are represented in XML document. Unfortunately there is no documentation I can find on the exact behavior of these holograms (such as what happens when you have an export error, and pending modifications). I ended up writing a c# console XML parser, based off my testing in DEV. I plan on making it more customizable, but for now it seems to work ok. The utility I wrote will take any pending export file as generated from the CSExport.exe program and transform the XML into a CSV format (quote encased, pipe-delimited). The first three columns “ID, ObjectType, OperationType” exist for all objects and should be self explanatory. The rest of the columns show all add/update/delete attributes for any objects being added/updated/deleted. Blank means that it doesn’t apply to the object or it’s not being updated.
The utility can be ran to “-showmodifications” or just display what the values will be updated to. The difference is that the showmodifications will display the “Old Value” as seen in the Sync Engine whenever possible (see image below)
Figure 4: Viewing The CSV Produced By The CSExportParse Utility In ShowModifications Mode (Download Link available later)
-
*note, there are some side effects with viewing the “-showmodifications” output due to the way line breaks are outputted, I’ll blog more about that later*
When ran without the “-showmodifications” switch the output will be similar to the following image below:
Figure 5: Viewing The CSV Produced By The CSExportParse Utility In Normal Mode (Download Link available later)
-
As you can see in the images above, I imported the file into excel and can easily filter and group the data to show all changes queued up.
-
Download: CS Export Parser v1.0
*requires the .NET framework 3.5
-
Drop File and Stop Run (Export – Run Profile)
At first I decided to give parsing this information in PowerShell from the Log Export a try. I wanted to write a script that would transform the XML file into CSV format which can be sorted, queried, and manipulated in other programs, such as Excel. The result will be a CSV file with at least 3 headers:
- “ID” – The connector-space “anchor” or unique identifier in the connector space
- “ObjectType” – The type of object (Person,Group,Contact) of the CS object
- “OperationType” – Defines what is happening to the object [being created/being updated/being deleted]
All additional header columns listed in the file are the attributes names themselves being added/updated and pertain only to the OpertationType of add / updates. *Note, the max number of columns added is based off the max number of attributes being added/modified from a single object. Also, anytime an object has an attribute value of blank (and the operation is update/add) it means that that particular attribute isn’t going to be modified in target system. Remember this shows what’s being updated, if the value is blank for a field, it doesn’t mean it’s being removed, just that the attribute wasn’t touched for that object.
The XML XML information (in my opinion) is simpler to interpret then the csexport. However, one annoyingly thing I discovered is that objectTypes don’t seem to be defined for any updates or deletes (just adds). Therefore you would need to look back on the object’s ID to determine what type of object it is, this isn’t really the case for the CSExport as you can export almost any information (including the objectType) to build the report. Another limitation is that in my testing for custom management agents ECMA, it shows all update operations as “Replace” and doesn’t give any information as to what is being modified 
The script below can be used to generate the CSV file. *Note it doesn’t work with the CSExport utility.
Keep in mind that this is a work in progress…
NOTE FROM JORGE: the version of the code below can be used in a PowerShell script. I have been having issues with the script below when large amounts of data need to be read (e.g. 300000 lines) I have to split it in multiple files of for example 40000 lines to be able to read it. I still do not understand why this happens. The original script missed a declaration for $I = 0 in the update section
# http://blog.fractalengine.com/fim-log-export-xml-parser/
# File Name: FIM-Sync-Engine-Parse-Export-Run-Log.ps1
Param(
[Parameter(Mandatory=$true)]
[string] $filePath,
[Switch] $showModifications
)
function ProcessExportXML
{
$global:MainList = @()
$global:Headers = @("ID","ObjectType","OperationType")
if (!(Test-Path $filePath))
{
throw "Invalid path: $filePath"
}
try {
[System.Xml.XmlDocument] $xml = New-Object System.Xml.XmlDocument
$xml.Load($filePath)
#have to load the namespaces
$namespaceURI = $xml.mmsml.xmlns
$ns = @{"e"=$namespaceURI}
getAdds -xml $xml -ns $ns
getDeletes -xml $xml -ns $ns
getUpdates -xml $xml -ns $ns -mods $showModifications
$report = $global:MainList | select $global:Headers
return $report
}
catch {
}
}
function getUpdates($xml, $ns, $mods) {
try {
$updates = $xml | select-xml "//e:delta[@operation='update']" -Namespace $ns
if ($updates -ne $null)
{
$totalUPD = $(if ($updates.count -gt 0) { $updates.count;} else { 1; })
$iUPD = 0
$updates | % {
$node = $_.Node
write-progress -Activity "Processing Updates" -Status $node.dn -PercentComplete ($iUPD / $totalUPD * 100)
$obj = New-Object -TypeName System.Object
$obj | Add-Member -MemberType NoteProperty -Name "ID" -Value $node.dn
$obj | Add-Member -MemberType NoteProperty -Name "ObjectType" -Value $null
$obj | Add-Member -MemberType NoteProperty -Name "OperationType" -Value $node.operation
#get attributes
$node.attr | % {
$item = $_
if ($global:Headers -notcontains $item.name)
{
$global:Headers += $item.name
}
switch ($item.operation)
{
"update" {
if ($mods)
{
#need to fix this, not sure how to handle multivalue attributes with modifications
if ($item.multivalued -eq $true)
{
$stringBuilder = ""
$valuesAdded = ($item.value | where { $_.operation -eq "add"})
$valuesRemoved = ($item.value | where { $_.operation -eq "delete"})
$valuesRemoved | % {
$attributeValue = $_."#text"
$stringBuilder += "Removed: [$attributeValue]`r`n"
}
$valuesAdded | % {
$attributeValue = $_."#text"
$stringBuilder += "Added: [$attributeValue]`r`n"
}
$obj | Add-Member -MemberType NoteProperty -Name $item.name -value $stringBuilder.trim()
}
else {
$stringBuilder = ""
$old = ($item.value | where { $_.operation -eq "delete"})."#text"
$new = ($item.value | where { $_.operation -eq "add"})."#text"
$stringBuilder = "Old: [$old]`r`nNew: [$new]"
$obj | Add-Member -MemberType NoteProperty -Name $item.name -value $stringBuilder.trim()
}
}
else {
if ($item.multivalued -eq $true)
{
$stringBuilder = ""
$valuesAdded = ($item.value | where { $_.operation -eq "add"})
$valuesAdded | % {
$attributeValue = $_."#text"
$stringBuilder += "$attributeValue;"
}
$obj | Add-Member -MemberType NoteProperty -Name $item.name -value $stringBuilder.trim()
}
else {
$obj | Add-Member -MemberType NoteProperty -Name $item.name -value (($item.value | where { $_.operation -eq "add" })."#text")
}
}
}
"delete" {
$obj | Add-Member -MemberType NoteProperty -Name $item.name -value "[deleted]"
}
"add" {
if ($item.multivalued -eq $true)
{
$stringBuilder = ""
$item.value | % {
$stringBuilder += "$_;"
}
$obj | Add-Member -MemberType NoteProperty -Name $item.name -value $stringBuilder
}
else {
$obj | Add-Member -MemberType NoteProperty -Name $item.name -value $item.value
}
}
}
}
$global:MainList += $obj
$iUPD++
}
}
}
catch {
write-host "Error! $_ "
}
}
function getAdds($xml,$ns) {
try {
$adds = $xml | select-xml "//e:delta[@operation='add']" -Namespace $ns
if ($adds -ne $null)
{
$totalADD = $(if ($adds.count -gt 0) { $adds.count;} else { 1; })
$iADD = 0
$adds | % {
$node = $_.Node
write-progress -Activity "Processing Adds" -Status $node.dn -PercentComplete ($iADD / $totalADD * 100)
$obj = New-Object -TypeName System.Object
$obj | Add-Member -MemberType NoteProperty -Name "ID" -Value $node.dn
$obj | Add-Member -MemberType NoteProperty -Name "ObjectType" -Value $node."primary-objectclass"
$obj | Add-Member -MemberType NoteProperty -Name "OperationType" -Value $node.operation
#get attributes
$node.attr | % {
$item = $_
if ($global:Headers -notcontains $item.name)
{
$global:Headers += $item.name
}
if ($item.multivalued -eq $true)
{
$stringBuilder = ""
$item.value | % {
$stringBuilder += "$_;"
}
$obj | Add-Member -MemberType NoteProperty -Name $item.name -value $stringBuilder
}
else {
$obj | Add-Member -MemberType NoteProperty -Name $item.name -value $item.value
}
}
$global:MainList += $obj
$iADD++
}
}
}
catch {
write-host "Error! $_"
}
}
function getDeletes($xml, $ns) {
try {
$deletes = $xml | select-xml "//e:delta[@operation='delete']" -Namespace $ns
if ($deletes -ne $null)
{
$totalDEL = $(if ($deletes.count -gt 0) { $deletes.count;} else { 1; })
$iDEL = 0
$deletes | % {
$node = $_.Node
write-progress -Activity "Processing Deletes" -Status $node.dn -PercentComplete ($iDEL / $totalDEL * 100)
$obj = New-Object -TypeName System.Object
$obj | Add-Member -MemberType NoteProperty -Name "ID" -value $node.dn
$obj | Add-Member -MemberType NoteProperty -Name "ObjectType" -Value $null
$obj | Add-Member -MemberType NoteProperty -Name "OperationType" -Value $node.operation
$global:MainList += $obj
$iDEL++
}
}
}
catch {
write-host "Error! $_"
}
}
ProcessExportXML
-
To use this you would simple run the “Log Export” from your MA, then copy this code into a powershell console and specify the filePath to the XML document generated from the MA.
FIM-Sync-Engine-Parse-Export-Run-Log.ps1 -filePath "<Source XML File>"
FIM-Sync-Engine-Parse-Export-Run-Log.ps1 -filePath "<Source XML File>" | Out-GridView
OR
FIM-Sync-Engine-Parse-Export-Run-Log.ps1 -filePath "<Source XML File>" -showModifications
FIM-Sync-Engine-Parse-Export-Run-Log.ps1 -filePath "<Source XML File>" -showModifications| Out-GridView
-
The “ShowModifications” mode will show the “old & new” attribute values in each cell. Once done you can pipe the $report to an Export-csv cmdlet to write it’s contents out to a file, or just a out-gridview to display it within a powershell gridview. This is useful if you plan to use excel to filter and sort the objects. With the first command you will see something like this
![clip_image001[4]](http://jorgequestforknowledge.files.wordpress.com/2013/02/clip_image0014_thumb.jpg?w=842&h=343 "clip_image001[4]")
Figure 6: Viewing The Output In PowerShell GridView When Running In Normal Mode
-
With the “-showModifications” switch it will look more like the following
Figure 7: Viewing The Output In PowerShell GridView When Running In ShowModifications Mode
-
</QUOTE SOURCE=”FIM – CS Export XML Parser”>
-
Cheers,
Jorge
———————————————————————————————
* This posting is provided "AS IS" with no warranties and confers no rights!
* Always evaluate/test yourself before using/implementing this!
* DISCLAIMER: http://jorgequestforknowledge.wordpress.com/disclaimer/
———————————————————————————————
############### Jorge’s Quest For Knowledge #############
######### http://JorgeQuestForKnowledge.wordpress.com/ ########
———————————————————————————————
Like this:
Like Loading...
![clip_image001[4]](http://jorgequestforknowledge.files.wordpress.com/2013/02/clip_image0014.jpg "clip_image001[4]")
