Detect duplicate data

Article
11/21/2017

Applies To: Dynamics 365 (online), Dynamics 365 (on-premises), Dynamics CRM 2016, Dynamics CRM Online

Duplicate detection lets organizations set duplicate detection policies and create duplicate detection rules for business and custom entities. These rules can be applied across different record types in Microsoft Dynamics 365. For example, an organization may define that a lead is a duplicate of a contact, if they have the same name and phone number. Based on the duplicate detection rules set by the administrator, the system alerts the user about potential duplicates when the user tries to create new records or update existing records. To maintain data quality, you can schedule a duplicate detection job to check for duplicates for all records that match a certain criteria. You can clean the data by deleting, deactivating, or merging the duplicates reported by a duplicate detection job.

To detect duplicates in the system, create a duplicate detection rule for a specific entity type. A duplicate detection rule is represented by the duplicate rule (DuplicateRule) entity. You can create multiple detection rules for the same entity type. However, you can publish a maximum of five duplicate detection rules per entity type at one time.

A rule can have one or more duplicate detection rule conditions that are represented by the duplicate rule condition (DuplicateRuleCondition) entity. The conditions are combined by the system as in logical AND operation. A duplicate detection rule specifies a base entity type and a matching entity type. A duplicate rule condition specifies the name of a base attribute and the name of a matching attribute. For example, specify an account as a base entity and a contact as a matching entity to compare last names and addresses. The matching criteria consist of operators such as exactly match, first n-number of characters, or last n-number of characters.

Duplicate detection works by comparing generated match codes of existing records with each new record being created. These match codes are created as each new record is created. Therefore, there is potential for one or more duplicate records to be created if they are processed at the exact same moment. In addition to detecting duplicates as they are created, you should schedule duplicate detection jobs to check for other potential duplicate records.

To create duplicate detection rules and duplicate detection rule conditions in the Microsoft Dynamics 365 database, use the IOrganizationService.Create method or the CreateRequest message.

Important

Take special precautions when you create duplicate detection rules for appointments. The recurring appointment master and the recurring appointment records share some of the same attribute values, such as subject, location, and regarding object. Using any of these attributes as a duplicate detection rule condition (DuplicateRuleCondition), may result in incorrectly marking recurring appointment records as duplicates. To avoid this, also include a start time attribute as a duplicate detection rule condition. For more information about working with recurring appointments, see Schedule and appointment entities.

The duplicate detection rules are system-wide. You must publish them before running a duplicate detection job to detect duplicates for bulk data or retrieve duplicates for a particular entity record. To publish a duplicate detection rule, use the PublishDuplicateRuleRequest message. Duplicate rule publishing is an asynchronous operation that runs in the background.