Sync Engine Provider

Introduction

The Sync Engine contains functionality for monitoring and managing the Microsoft family of identity synchronization engines, including MIM, AADSync, the various versions of FIM 2010, and DirSync. The Sync Engine provider also supports data collection and time traveller features for the MIM Portal, and for password reset history, and includes a range of schedule steps and health probes for working with FIM and MIM.

The Sync Engine provider can be used to track data moving through the Connector Spaces and Metaverse, keep a record of operations history long past when the Operations Log is purged, and manage the health and scheduling of the sync environment.

Upgrading

If you migrate a sync engine configuration from 2.0 using the Upgrade tool, your provider settings will be populated with a new Sync Engine provider, and your data will be migrated over. After the upgrade tool runs, do the following:

  • Review your provider connection settings, and make sure that database names and connection strings, as well as settings for CSExport, and WMI timeouts are fully populated.
  • If you created a dummy provider as part of getting to the upgrade step, remove it now.
  • Use Panel Tools to run a full scan of Sync Engine (and Portal if configured).

Setup

Permissions

On the SoftwareIDM Panel Service account, grant the following permissions:

  • Add the service account to the FIMSyncOperators group
  • Add the service account to the FIMSyncBrowse group
  • Grant the service account the datareader role for the FIMSynchronizationService database
  • If using the Portal, grant the service account datareader role for the FIMService database

Connection Settings

After claiming a license that enables the Sync Engine Provider, navigate to provider settings. If you have upgraded a previous version of Sync Panel, these settings will partially be pre-populated. If this is a new deployment press the add icon to create a new environment connection.

Sync Settings

Each provider environment must be given a name. Short names are recommended, since the environment name will be prefixed to each Management Agent and Metaverse data silo. If necessary you can change the name at a later date, since Identity Panel supports renaming provider environments at any time.

Next enter connection settings for the database(s) and WMI. The Sync SQL Instance Name property should be the SQL instance of the FIMSynchronizationService database. Sync Server Name should be the host name to connect to the FIM WMI provider.

Note: for performance reasons the Sync Engine provider reads data directly from the sync engine database where possible. Only SELECT queries are performed, and all queries are made with a transaction level of ReadUncommitted to prevent interference with sync engine operation.

Although you may enter connection credentials for the SQL connection, it is recommended that you grant db datareader access to the service account used by SoftwareIDM Panel Service. All WMI provider interactions will use the service account credentials.

Rapid Scan

Sync Panel 2.0 introduced a feature called rapid scan. This addresses the issue of long running data scans on large systems, particularly with the initial full-scan.

Rapid Scan works by reading the binary CS hologram directly, rather than using the WMI to read attributes. Avoiding the latency of using the WMI gives a 30 to 60 fold performance boost. However, rapid scan is disabled by default because it must be supported by SoftwareIDM on specific versions of the Microsoft Sync Engine. When executing a rapid scan, the sync provider will periodically re-read the object from the WMI and compare the two. If the rapid scan holograms differ from the WMI hologram, the sync provider will fallback to using only the WMI scan.

Rapid scan is only supported on FIM 2010 R2, and is not compatible with configurations that synchronize multi-value binary attributes.

Because the Sync provider only scans objects that have changed since the last scan, the performance boost of CS Rapid Scan is usually only needed on the initial full scan.

When choosing whether to enable rapid scan, it is recommended that you first verify compatibility with your version of FIM by contacting SoftwareIDM, or by performing a rapid scan in a lower region test environment.

WMI Timeout and CS Export

There are known issues with the Microsoft WMI interface for FIM and AADSync which can result in timeouts and errors. To compensate, the Sync provider supports fallback using CSExport.exe. To configure this, simply entered a desired WMI Timeout duration (five to ten seconds is typical), and provide the full local file-system path of the CSExport executable. Note: When using the CSExport fallback feature, you should configure your schedules to use a panel tools instance that is installed locally on the FIM server for steps that run MAs or perform full scans.

Full Scan

After saving connection settings, you should use PanelTool to perform a Full-Scan. This will do the initial data population, and create the MA configuration records that will allow you to configure charts and schedules.

  • Open a command prompt on the server where you installed the scheduler service
  • Run PanelTool.exe
  • After it loads, select the menu option to run "Full Scan: <environment>".
    Full Scan

After completing the first full scan you should restart the IdentityPanel virtual directory in IIS to clear cached values.

The Sync engine provider includes a full array of history counters that may be used in charts and history filtering.

Chart Counters

Import Add Import Update Import Rename Import Delete Import Delete-Add Import Failure Filtered Disconnector Join no flow Join Join - MV Delete Projection no flow Projection Disconnector Unknown Filtered - MV Delete Filtered Connector Import Flow Flow - MV Delete Connector Connector - MV Delete Connector Delete Connector Delete - Add Flow Failure Export Add Export Update Export Rename Export Delete Export Delete - Add Export Failure Provision Flow Provision no flow Prov Add no flow Provision Add Prov Rename no flow Prov Rename Provision Delete Prov Delete - Add no flow Prov Delete - Add Provision Reset Discovery Error Export Error Retry Error Sync Error

Password Source Password Target Password WMI Reset Password Error

Schedule Steps

Truncate Ops Log

The Truncate Ops Log step removes old data from the MIM operations log, retaining the specified number of days worth of history.

History

Returns a History Record with RecordOf set to "Truncate Ops Log to (n) Days", and Argument set to the Id of the Environment. Result is set to either "success" or "WMI Error".

Concurrency

The Truncate Ops Log step obtains an exclusive lock on the sync service environment. This prevents simultaneous execution of MA Run steps, Full Scan, and Log truncate steps against the same sync service.

Settings

  • Environment: Required, name of the MIM/AADSync/FIM/DirSync environment configured in "MIM and AADSync" settings.
  • Days to Retain: Required, number of days of history to retain after purging runs.

Truncate Password Log

The Truncate Password Log step removes old data from the MIM Password sync history, retaining the specified number of days worth of history.

History

Returns a History Record with RecordOf set to "Truncate Password Log to (n) Days", and Argument set to the Id of the Environment. Result is set to either "success" or "WMI Error".

Concurrency

The Truncate Password Log step obtains an exclusive lock on the sync service environment. This prevents simultaneous execution of MA Run steps, Full Scan, and Log truncate steps against the same sync service.

Settings

  • Environment: Required, name of the MIM/AADSync/FIM/DirSync environment configured in "MIM and AADSync" settings.
  • Days to Retain: Required, number of days of history to retain after purging password history.

Password History Scan

The Password History Scan step reads password change and password sync data from the Sync Engine. It reads the mms_connectorspace.password_history xml column to retrieve data about WMI Password SET invocations. This captures self-service password resets from the FIM Portal. It uses the WMI MIIS_PasswordChangeHistorySource and MIIS_PasswordChangeHistoryTarget classes to retrieve password synchronization data.

Each time a password history scan executes, it captures all password changes that have occurred since the last history scan. If no previous scans have occurred it captures all resets/syncs since the password history log was truncated.

History

Returns a PasswordScanRecord history entry. This record has RecordOf set to "Password History", and Argument set to the Id of the Environment. The Result is either "success" or an error message.

Counters are incremented for:

  • Password Source: 55 — Password synced from Source MA
  • Password Target: 56 — Password successfully exported to Target MA
  • Password WMI Reset: 57 — Password SET or CHANGE was invoked using WMI API (typically FIM Portal self-service password reset). Note: the FIM Portal will invoke WMI resets on the AD MA, so object details are linked to the AD MA in the password history record.
  • Password Error: 58 — Some error occurred setting the password to the Target system. The error counter is incremented even if the reset/sync was ultimately successful, if multiple attempts were required.

Concurrency

The Password History scan step may be executed simultaneously with any other step except another Password History scan of the same Sync Service environment.

Settings

  • Environment: Required, name of the MIM/AADSync/FIM/DirSync environment configured in "MIM and AADSync" settings.

Full Scan Sync

The Full Scan Sync step reads each CS and MV object in the Sync Engine and compares to see if it is different from what is stored in Sync Panel. The Full Scan also reads MA and MV configuration data, including run profile settings for creating Run MA steps. The Full Scan Sync step uses high speed hashing to improve performance.

History

Returns a FullScanRecord history entry. This record has RecordOf set to "Full scan", and Argument set tot he Id of the Environment. The Result is either "success" or an error message.

Counters are incremented for CS changes and MV changes. All CS changes are tallied as if they were Imports, and MV changes are tallied as synchronizations.

Counters are incremented for:

  • Import Add: 2 — New CS Object
  • Import Update: 3 — Updated CS Object
  • Import Delete: 5 — Deleted CS Object
  • Projection: 13 — New MV Object
  • Import Flow: 18 — Updated MV Object
  • Connector - MV Delete: 21 — Deleted MV Object

Concurrency

The Full Scan step obtains an exclusive lock on the sync service environment. This prevents simultaneous execution of MA Run steps, Full Scan, and Log truncate steps against the same sync service.

Settings

  • Environment: Required, name of the MIM/AADSync/FIM/DirSync environment configured in "MIM and AADSync" settings.

Run MA

The Run MA step executes a selected management agent run profile and scans the result. Run MA steps cannot be created until after a Full Scan has been executed against the Sync Service environment, because the full scan handles MA and Run Profile discovery.

History Returns a RunRecord history entry. If the Run Profile has multiple steps, only the history scan from the last step will be returned. The Others will be POSTed directly to the SyncPanel History collection.

Counters are incremented for all statistics tracked by the Sync Service Operations Log:

0: "No statistics",
1: "Import Unchanged",
2: "Import Add",
3: "Import Update",
4: "Import Rename",
5: "Import Delete",
6: "Import Delete-Add",
7: "Import Failure",
8: "Filtered Disconnector",
9: "Join no flow",
10: "Join",
11: "Join - MV Delete",
12: "Projection no flow",
13: "Projection",
14: "Disconnector",
15: "Unknown",
16: "Filtered - MV Delete",
17: "Filtered Connector",
18: "Import Flow",
19: "Flow - MV Delete",
20: "Connector",
21: "Connector - MV Delete",
22: "Connector Delete",
23: "Connector Delete - Add",
24: "Flow Failure",
25: "Export Add",
26: "Export Update",
27: "Export Rename",
28: "Export Delete",
29: "Export Delete - Add",
30: "Export Failure",
31: "Provision Flow",
32: "Provision no flow",
33: "Prov Add no flow",
34: "Provision Add",
35: "Prov Rename no flow",
36: "Prov Rename",
37: "Provision Delete",
38: "Prov Delete - Add no flow",
39: "Prov Delete - Add",
40: "Provision Reset",
41: "Discovery Error",
42: "Export Error",
43: "Retry Error",
44: "Sync Error"

Concurrency

If the selected Run Profile has only Import and Export steps, Run MA obtains a shared lock on the Sync Service environment. If there is a synchronization step, then an exclusive lock is obtained. This allows Imports and Exports to run in parallel, but prevents deadlocking the Sync Service with parallel sync profiles.

Settings

  • MA: Required, the MA that should be executed. By default, MA Names in the drop-down list are prefixed with the name of the Environment. This behavior is overridden if an MA Name is re-mapped in the Attribute Naming settings.
  • Run Profile: Required, the Run Profile to execute.

Health Probes

The Sync Engine provider includes several health probes for validating the status of the FIM sync engine and portal.

FIM Password Queue

Queries the FIM Sync Engine WMI to determine the length of the Password Queue. If the length of the password queue grows, this can indicate that FIM is unable to connect to targets of password synchronization.

Probe Result

  • Value: integer length of queue
  • StringValue: Value.ToString()

Settings

  • FIM Sync Server: WMI host to connect to
  • Fail Rule — Default rule is context.Value > 5 which returns true if more than 5 passwords are pending export synchronization.

Workflow Queue

Queries the FIM Services SOAP interface to determine how many workflows are pending in the queue. This is done by Enumerating the filter /WorkflowInstance[WorkflowStatus='Created' or WorkflowStatus='Pending'] . Because the workflow queue probe must enumerate the result set, a large number of pending workflows may reduce the frequency of health checks, due to the latency of executing this probe.

Probe Result

  • Value: integer length of queue
  • StringValue: Value.ToString()

Settings

  • Fail Rule — Default rule is context.Value > 10 which returns true if more than 50 workflow instances are pending.
  • Url — FIM Service url, including port. e.g. http://localhost:5725/
  • Service Identity – UPN of service account for FIM Service
  • User (opt)
  • Domain (opt)
  • Password (opt — may be overridden by Sync:Password appSetting)

FIM Service Query

Enumerates and counts an arbitrary FIM Service XPath query. This can be used e.g. to determine how many Approval objects are in a Pending state. Because the filter probe must enumerate the result set, a large number of results may reduce the frequency of health checks, due to the latency of executing this probe.

Probe Result

  • Value: integer size of result set
  • StringValue: Value.ToString()

Settings

  • Filter — XPath filter to execute against the FIM Service (e.g. /Approval[ApprovalStatus='Pending'] )
  • Fail Rule — No default rule is set
  • Url — FIM Service url, including port. e.g. http://localhost:5725/
  • Service Identity – UPN of service account for FIM Service
  • User (opt)
  • Domain (opt)
  • Password (opt — may be overridden by Sync:Password appSetting)

FIM Portal Object

Looks up a single object in the FIM Service portal, converts it to an ObjectRecord, and executes a rule against the result. This probe enumerates the provided XPath filter and converts the first result to an ObjectRecord, adding all properties to the Attributes collection. Multi-Value attributes are also added to the Attributes dictionary, and separated by '|' character.

Probe Result

  • Value: Result of applying the Object Rule to the filtered object
  • StringValue: Value.ToString()

Settings

  • Filter — XPath filter to execute against the FIM Service
  • Object Rule - Rule Engine rule to evaluate the ObjectRecord constructed from the FIM Service result
  • Fail Rule — No default rule is set
  • Url — FIM Service url, including port. e.g. http://localhost:5725/
  • Service Identity – UPN of service account for FIM Service
  • User (opt)
  • Domain (opt)
  • Password (opt — may be overridden by Sync:Password appSetting)

Copyright © SoftwareIDM

Table of Contents