Filtering Synchronization Data

Article
09/29/2009

A filter is used to restrict the data that is synchronized. Typically, the source provider applies the filter during change enumeration to restrict the changes that are sent. Sync Framework supports the following types of filters.

Item filters restrict synchronization data to a subset of items, such as to only synchronize .txt files between two file folders, ignoring files of other types. Items do not change in a way that causes an existing item to move into or out of the filter. Item filters are simple to use, but the metadata used for synchronization grows in proportion to the number of items that are in the synchronization scope.
Change unit filters restrict synchronization data to a subset of change units, such as to only synchronize the name and phone number fields of a contact, ignoring the remaining change units.
Custom filters restrict synchronization data in a way that is unknown to Sync Framework. Items can move into and out of custom filters over time. Sync Framework defines additional metadata, interfaces, and classes that efficiently support custom filters while keeping the overall size of the metadata small.

The filter can be defined by the synchronization application or by the source or destination provider, or it can be negotiated between the source and destination providers.

The filter is used by the source provider when it detects changes. Change batches built during change detection include only the synchronization data that passes the filter.

The filter can also be used by the destination provider when it applies changes. Changes that are applied to the destination replica include only the synchronization data that passes the filter.

Controlling which Items are Synchronized

An item filter defines which item changes are sent by the source provider during change enumeration. Because the way that items are filtered depends on the type of data in the synchronization scope, Sync Framework allows the source provider to define the filtering mechanism. If communication about the filter is required between the provider and the synchronization application, this can be accomplished by using whatever mechanism is appropriate.

When the source provider filters item changes, Sync Framework requires that the source provider attach information about the filter to all change batch objects sent during change enumeration. The presence of item filter information in the change batch object informs Sync Framework that a filtered synchronization is occurring, so that Sync Framework can correctly handle the knowledge for the filtered change batch. If the source provider filters the items included in a change batch but does not specify item filter information to the change batch object, Sync Framework might not correctly handle the knowledge.

When the source provider uses a filter to restrict the items that are contained in a change batch, the provider must attach information about the filter to the ChangeBatch (for managed code) or ISyncChangeBatch (for unmanaged code) object. The filter information is represented by an ItemListFilterInfo object (for managed code) or an ISyncFilterInfo object that has the SYNC_FILTER_INFO_FLAG_ITEM_LIST flag set (for unmanaged code). In unmanaged code, the provider creates an ISyncFilterInfo object by using IProviderFilteredSyncServices::CreateFilterInfo. The filter information is attached to the change batch by using ChangeBatch (for managed code) or IProviderFilteredSyncServices::CreateFilteredEnumerationChangeBatch (for unmanaged code) to create the change batch object.

For more information on how to use item filtering, see How to: Filter Enumerated Items.

Controlling which Change Units are Synchronized

A change unit filter defines which change unit changes are included for each item change that is sent by the source provider during change enumeration. A change unit filter is useful when a replica only stores a subset of the change units defined for the items in the scope.

For example, a synchronization community exchanges contact information, and defines change units for name, phone number, and address. One replica in the community is a mobile device that can only store name and phone number. This replica uses a change unit filter to specify that it only tracks knowledge for these two change units. When a source replica synchronizes data to the device replica, it uses the change unit filter to send information only about the name and phone number change units. The learned knowledge from the source provider is projected onto the change unit filter so the knowledge on the device replica contains knowledge of only the name and phone number change units. Similarly, when a change is made locally on the device and the device subsequently acts as the source replica in a synchronization session, the replica that receives the changes updates its knowledge only for the specified set of change units.

When a change unit filter is used, Sync Framework projects the learned knowledge for the change batch onto the set of change units specified by the filter so that the learned knowledge for the change batch contains information only about the specified set of change units.

To filter change unit changes, the source provider creates a ChangeUnitListFilterInfo object (for managed code) or IChangeUnitListFilterInfo (for unmanaged code) and specifies the set of change units that are synchronized. In unmanaged code, the IChangeUnitListFilterInfo object is created by specifying SYNC_FILTER_INFO_FLAG_CHANGE_UNIT_LIST to the IProviderFilteredSyncServices::CreateFilterInfo method. The source provider attaches the filter information to the change batch by using ChangeBatch (for managed code) or IProviderFilteredSyncServices::CreateFilteredEnumerationChangeBatch (for unmanaged code) to create the change batch object.

For more information on how to use a change unit filter, see How to: Filter Enumerated Change Units.

Controlling what is Synchronized by Using a Custom Filter

A custom filter is defined outside of Sync Framework, typically by a provider developer, though it may also be defined by a third party. Sync Framework has no knowledge of how the filter determines what to include in the synchronization scope. A custom filter can be defined in such a way that items move into and out of the filter over time. For example, on a replica that stores media files, a filter can be defined that includes all files that are rated as three stars or better. When the user changes the rating on a file, it can move into or out of the filter.

Be aware that custom filters cannot be used by simple providers, or by providers that report constraint conflicts or use the change application service, or unexpected results may occur.

To efficiently support custom filters, Sync Framework defines several new concepts.

A filter-tracking replica is a replica that maintains metadata that defines which items and change units are currently in a tracked filter, which items and change units have moved into or out of the tracked filter recently, and filter forgotten knowledge that covers all items and change units that are not in the tracked filter and have been forgotten. Tracking a filter does not affect which items are stored in a replica. A replica can track multiple filters while storing all items in the synchronization scope. Typically, a filter-tracking replica can provide a filtered change batch for any of its tracked filters.
A filtered replica is a replica that stores item and change unit data only for the items and change units that are in the filter, as well as metadata for items and change units that were in the filter but have moved out. A filtered replica always tracks the filter that it uses, and may track additional filters as well. A filtered destination provider can request a filtered change enumeration from the source provider, or it can request a full change enumeration and filter the changes itself during change application.
Ghosts are items or change units in a filtered replica that were in the filter and have moved out. The filtered replica stores metadata for ghosts, but does not store item or change unit data.
Filter forgotten knowledge defines a starting point for filter tracking. A filter tracking replica can save storage space by removing ghosts and advancing the filter forgotten knowledge to contain the highest version of the ghosts that have been removed. Filter forgotten knowledge is also used when a replica starts tracking a filter after the filer has been in use by other replicas in the synchronization community.

Tracking Filters

It is recommended that all replicas in a synchronization community track the filters that are used in the community. When a filtered replica receives a filtered change enumeration from a filter-tracking replica, the knowledge of the filtered replica is kept small. When a filtered replica receives a filtered change enumeration from a replica that does not track the filter, the size of the knowledge grows in proportion to the number of changes sent.

For more information on tracking filters, see How to: Track Filters and Enumerate Filtered Changes.

Filter Metadata

To track a filter, a replica stores filter metadata for each item or change unit that in the replica. The filter metadata contains the elements in the following table:

Is in filter	Move version
Indicates whether the item or change unit is in the filter.	The change version that caused the item to move in relation to the filter. When an item is created and the item is in the filter, this version is set to the creation version of the item. When an item has never been in the filter, this version is set to (0,0).

When a replica starts tracking a filter, all items or change units must be evaluated against the filter. If an item or change unit is in the filter, its filter metadata must indicate that it is in the filter and its move version must be set to the latest change version of the item or change unit. If an item is not in the filter, its filter metadata must indicate that it is not in the filter, and its move version must be set to (0,0).

A filter-tracking replica can also store filter forgotten knowledge for each filter that it tracks. When a filter-tracking replica tracks a filter from the origination of the filter, it does not store filter forgotten knowledge unless it receives synchronization data from a replica that does not track the filter. When a filter-tracking replica starts tracking a filter sometime after the filter has originated, it must initialize the filter forgotten knowledge to the current knowledge of the replica. When tombstones and filter change metadata are cleaned up on a replica, the filter forgotten knowledge can be discarded and the forgotten knowledge used instead.

Negotiating Tracked Filters

A filter-tracking provider must implement IFilterTrackingProvider (for managed code) or IFilterTrackingProvider (for unmanaged code). This interface is used to communicate the filters that are tracked on both the source replica and the destination replica. The source replica sends filter metadata for the filters that are tracked on both the source replica and destination replica.

When both providers in a synchronization session are filter-tracking providers, Sync Framework mediates the negotiation of which filters are tracked by both providers. Tracked filters are negotiated between two filter-tracking providers by using the following steps:

After calling BeginSession (for managed code) or IKnowledgeSyncProvider::BeginSession (for unmanaged code), Sync Framework calls SpecifyTrackedFilters (for managed code) or IFilterTrackingProvider::SpecifyTrackedFilters (for unmanaged code) on the destination provider.
The destination provider enumerates its list of tracked filters and passes each one to the RequestTrackedFilterCallback delegate (for managed code) or to the IFilterRequestCallback::RequestFilter method of the IFilterRequestCallback object (for unmanaged code) that is specified in the call to SpecifyTrackedFilters.
For each filter specified by the destination provider, Sync Framework calls TryAddTrackedFilter (for managed code) or IFilterTrackingProvider::AddTrackedFilter (for unmanaged code) on the source provider.
The source provider maintains a list of filters tracked by the destination provider and returns a value that indicates whether it tracks the specified filter.
Sync Framework passes this value to the destination provider.

Sending Filter Metadata for Tracked Filters

When the destination provider tracks filters that the source provider also tracks, the source provider must send filter metadata for the mutually tracked filters. To do this, add the following changes to the source provider implementation of GetChangeBatch (for managed code) or IKnowledgeSyncProvider::GetChangeBatch (for unmanaged code):

Add a FilterKeyMap object to the ChangeBatch object by setting the FilterKeyMap property (for managed code) or add an IFilterKeyMap object to the ISyncChangeBatchWithFilterKeyMap object by calling ISyncChangeBatchWithFilterKeyMap::SetFilterKeyMap (for unmanaged code). The filter key map object contains the filters that are tracked by both the source provider and the destination provider. The FilterKeyMap property must be set (for managed code) or the SetFilterKeyMap method called (for unmanaged code) before any groups are started in the change batch.
For each group in the change batch, add filter forgotten knowledge for tracked filters by calling SetFilterForgottenKnowledge (for managed code) or ISyncChangeBatchWithFilterKeyMap::SetFilterForgottenKnowledge (for unmanaged code).
Send filter change metadata for each item or change unit that has moved into or out of a tracked filter. Filter change metadata is added to a change by calling AddFilterChange (for managed code) or IFilterTrackingSyncChangeBuilder::AddFilterChange (for unmanaged code). The FilterChange properties (for managed code) or SYNC_FILTER_CHANGE values (for unmanaged code) indicate whether the item has moved into or out of the filter and the version of the change that caused the move. The filter change metadata is only added when the move version is not contained in the destination knowledge of the item. When change units are used, filter change metadata must be added if the move version is not contained in the destination knowledge of any of the change units. Be aware that move versions are treated as item versions even when change units are used, so containment must be checked by using a form of the Contains method (for managed code) or the ISyncKnowledge::ContainsChange method (for unmanaged code) that takes an item version and not one that uses a change unit version.
When change units are used and at least one change unit change is not contained in the destination knowledge, call ContainsMarker (for managed code) or IKnowledgeWithMarkers::ContainsAllChangeUnitsRequiredMarker (for unmanaged code) on the destination knowledge, specifying AllChangeUnitsRequired and the item ID of the owning item (for managed code) or the item ID of the owning item (for unmanaged code). If this method indicates that all change units are required, send all change units for the item and call SetAllChangeUnitsPresent on the ItemChange object (for managed code) or IFilterTrackingSyncChangeBuilder::SetAllChangeUnitsPresentFlag (for unmanaged code).

Sending Filtered Changes

Typically, a source provider that represents a filter-tracking replica can enumerate filtered change batches for any of the tracked filters. The filter can either be identified by the application, by using a custom mechanism between the application and the source provider, or it can be negotiated between the source provider and the destination provider. Filter negotiation is described later in this document.

To send a filtered change batch, add the following elements to the GetChangeBatch method of the source provider:

Create a CustomFilterInfo (for managed code) or ICustomFilterInfo (for unmanaged code) object that contains the filter used for synchronization. Create a filtered change batch object by passing the filter information object to the appropriate change batch constructor ChangeBatch (for managed code) or by calling IProviderFilteredSyncServices::CreateFilteredEnumerationChangeBatch (for unmanaged code). Also, pass the filter forgotten knowledge of the filter when creating the change batch object, instead of the forgotten knowledge of the replica.
When an item or change unit was previously in the filter, but is not currently in the filter, specify the ChangeKind property as Ghost (for managed code) or pass the SYNC_CHANGE_FLAG_GHOST flag to ISyncChangeBatchBase::AddItemMetadataToGroup (for unmanaged code).
When the filtering type of the destination provider is CurrentItemsOnly (for managed code) or FT_CURRENT_ITEMS_ONLY (for unmanaged code), include an item in the change batch only if it is in the filter. That is, do not send ghosts.
When the filtering type of the destination provider is CurrentItemsAndVersionsForMovedOutItems (for managed code) or FT_CURRENT_ITEMS_AND_VERSIONS_FOR_MOVED_OUT_ITEMS (for unmanaged code), include items that are in the filter and items that have moved out. Move-out information must be sent when an item or change unit was previously in the filter, the item or change unit is not currently in the filter, and the version of the change that moved the item or change unit out of the filter is not contained by the destination knowledge.

Applying Filter Metadata

When a destination provider represents a filter-tracking provider and uses a change applier to apply changes to the destination replica, the destination provider must implement IFilterTrackingNotifyingChangeApplierTarget (for managed code) or IFilterTrackingNotifyingChangeApplierTarget (for unmanaged code). This interface contains methods that Sync Framework uses to obtain the filter key map and filter forgotten knowledge of tracked filters. It also contains the SaveKnowledgeWithFilterForgottenKnowledge (for managed code) or IFilterTrackingNotifyingChangeApplierTarget::SaveKnowledgeWithFilterForgottenKnowledges (for unmanaged code) method, which Sync Framework calls instead of StoreKnowledgeForScope (for managed code) or ISynchronousNotifyingChangeApplierTarget::SaveKnowledge (for unmanaged code) for filter-tracking providers. Sync Framework uses this method to send updated knowledge, forgotten knowledge, and filter forgotten knowledge to the provider after a change batch is applied.

In addition to implementing IFilterTrackingNotifyingChangeApplierTarget, the SaveItemChange or SaveChangeWithChangeUnits method (for managed code), or ISynchronousNotifyingChangeApplierTarget::SaveChange or ISynchronousNotifyingChangeApplierTarget::SaveChangeWithChangeUnits method (for unmanaged code) of the destination provider must be updated to perform the following steps:

Get the filter change metadata for each tracked filter by calling GetFilterChange (for managed code) or ISyncChangeWithFilterKeyMap::GetFilterChange (for unmanaged code) on the change object.
When filter change metadata is present, check to ensure it is not obsolete. A filter change is obsolete when its move version is contained by the destination knowledge of the item. When change units are used, a filter change is obsolete only when the move version is contained in the destination knowledge of all of the change units. If the filter change is obsolete, do not apply it to the destination replica.
When filter change metadata is present and not obsolete, check to ensure it is not in conflict with the filter change information already present in the destination replica. To check for conflicts, perform the following steps:
1. Get the move version that is currently stored for the item or change unit in the destination replica.
2. Check whether the move version is contained in the made-with knowledge of the item, or contained in the made-with knowledge of every change unit change associated with the item.
3. If the move version is not contained in the appropriate made-with knowledge, the filter change is in conflict. The destination provider must resolve it as appropriate and assign a new move version for the change.
4. If no conflicts are detected with the move version, check the move-in flag of the source filter change against the move-in flag of the destination item or change unit. If the value of the flag is inconsistent, the destination provider must evaluate the item or change unit against the filter, and assign the correct move-in flag value along with a new move version.
5. If no conflicts of any kind are detected, store the filter change metadata along with the metadata for the item.
When filter change metadata is not present for a tracked filter, or for a filter that is tracked by the destination replica but not by the source replica, evaluate the change against the filter on the destination. Update the filter metadata to include whether the item is in the filter, and update the move version to the version of the change if the change caused the item to move in relation to the filter. When there are multiple change units associated with a filter and a change to any of them causes the item to move in relation to the filter, create a new local version and assign the new version as the move version of the item and as the change version for all of the change units associated with the filter.

Filtered Replicas

A filtered replica stores item and change unit data only for the items and change units that are in its filter, as well as ghosts, which are metadata for items and change units that were in the filter but have moved out. A filtered replica also tracks its filter, and may track other filters as well. A filtered replica can negotiate a filter with the source provider, in which case the source provider produces a filtered change batch. If the source provider cannot produce a filtered change batch, the filtered provider can filter the changes itself and apply only those that are in its filter.

For more information on implementing a filtered replica, see How to: Filter a Replica.

Enumerating Items Moved into the Filter

A filtered replica implements the IFilteredReplicaNotifyingChangeApplierTarget (for managed code) or IFilteredReplicaNotifyingChangeApplierTarget (for unmanaged code) interface. This interface contains the GetNewMoveInItems (for managed code) or IFilteredReplicaNotifyingChangeApplierTarget::GetNewMoveins (for unmanaged code), which Sync Framework uses to obtain items that have moved into the filter after a certain point in time. An item is added to the list that is returned from this method when the item is active, the item is in the filter, and the move version of the item is not contained in the knowledge specified to the method.

Sending Unfiltered Changes

When the filtered replica is the source replica and the destination replica does not request a filtered change enumeration, or a filter is not successfully negotiated between the two replicas, the source replica enumerates changes as if it is not filtered. This differs from the process of sending filtered changes in the following ways:

An unfiltered change batch is created.
The forgotten knowledge of the replica is passed when creating the change batch and not the filter forgotten knowledge as when filtered changes are sent.

Applying Filtered Changes

When the destination provider uses a change applier to apply changes, it must indicate whether the destination item or change unit is a ghost when it sends destination versions to the change applier. To do this, specify Ghost for the ChangeKind property (for managed code) or pass the SYNC_CHANGE_FLAG_GHOST flag to IDestinationChangeVersionsBuilder::AddItemMetadata (for unmanaged code) when the item is a ghost in the destination replica. An item is a ghost when metadata exists for the item in the destination metadata store, the item has not been deleted, and no data exists for the item in the destination store.

When the changes from the source provider are not filtered, the destination provider evaluates all changes sent by the source provider against the filter of the destination replica. The destination provider applies changes that pass the filter and updates any affected ghosts. The destination provider excludes any skipped changes from the knowledge of the change batch.

The destination provider must also make the following changes to its SaveItemChange or SaveChangeWithChangeUnits (for managed code), or ISynchronousNotifyingChangeApplierTarget::SaveChange or ISynchronousNotifyingChangeApplierTarget::SaveChangeWithChangeUnits (for unmanaged code) method.

Handle CreateGhost and UpdateGhost (for managed code) or SSA_CREATE_GHOST or SSA_UPDATE_GHOST (for unmanaged code) actions by creating or updating the metadata for a ghost. Filter change metadata must also be updated, if applicable.
Handle the MarkItemAsGhost (for managed code) or SSA_GHOST_ITEM (for unmanaged code) action by removing the data for the item or change unit and updating the metadata for the item or change unit to reflect the change.
Handle the UnmarkItemAsGhost (for managed code) or SSA_UNGHOST_ITEM (for unmanaged code) action by retrieving the data for the item or change unit, storing the data in the destination replica, and updating the metadata to reflect the change.
Handle the DeleteGhostAndStoreTombstone (for managed code) or SSA_DELETE_GHOST_AND_STORE_TOMBSTONE (for unmanaged code) action by marking the metadata for the ghost as deleted.

Cleaning Up Ghosts

To free storage space on a filtered replica, ghosts can be removed. When a ghost is removed, the filter forgotten knowledge for the filter must be updated by passing the move version of the ghost to the ForgetTo (for managed code) or IForgottenKnowledge::ForgetToVersion (for unmanaged code) method of the filter forgotten knowledge.

Negotiating the Filter

The destination provider can specify the filter that is used by the source provider during change enumeration. The source provider can accept or deny the filter that the destination provider requests. The destination provider can continue to request filters until one is found that the source provider accepts. This negotiation is mediated by Sync Framework. The filter can be of whatever type is most appropriate for the providers.

To participate in filter negotiation, the source provider must implement ISupportFilteredSync (for managed code) or ISupportFilteredSync (for unmanaged code) and the destination provider must implement IRequestFilteredSync (for managed code) or IRequestFilteredSync (for unmanaged code).

Filter negotiation is achieved by using the following steps:

Before the source provider begins enumerating changes and after calling BeginSession (for managed code) or IKnowledgeSyncProvider::BeginSession (for unmanaged code), Sync Framework starts filter negotiation by calling the SpecifyFilter method (for managed code) or IRequestFilteredSync::SpecifyFilter (for unmanaged code) on the destination provider.
During processing of SpecifyFilter, the destination provider passes filters to the FilterRequestCallback delegate (for managed code) or the IFilterRequestCallback::RequestFilter method (for unmanaged code) that is specified by Sync Framework. When the destination provider does not track the requested filter, it specifies a filtering type of CurrentItemsOnly (for managed code) or FT_CURRENT_ITEMS_ONLY (for unmanaged code). When the destination provider tracks the requested filter, it specifies a filtering type of CurrentItemsAndVersionsForMovedOutItems (for managed code) or FT_CURRENT_ITEMS_AND_VERSIONS_FOR_MOVED_OUT_ITEMS (for unmanaged code).
During processing of FilterRequestCallback (for managed code) or RequestFilter (for unmanaged code), Sync Framework calls the TryAddFilter (for managed code) or ISupportFilteredSync::AddFilter (for unmanaged code) method of the source provider. If the source provider does not support the requested filter, the destination provider can continue to request filters until it finds one that is supported.
If the destination provider cannot find a filter that is accepted by the source provider, it can terminate synchronization by throwing SyncInvalidOperationException (for managed code) or by returning an error code, such as SYNC_E_FILTER_NOT_SUPPORTED (for unmanaged code).

When a filter has been successfully negotiated, the source provider uses it to determine which items to include during change enumeration.

For more information on how to negotiate filters, see How to: Negotiate a Filter.

Share via