Edit

Considerations for initial synchronization

  • Article

Before you start dual-write on a table, you can run an initial synchronization to handle existing data on both sides: finance and operations apps and customer engagement apps. You can skip the initial synchronization if you don't have to sync data between the two environments.

The initial synchronization lets you copy existing data from one app to another bidirectionally, and there are several considerations when you run it. For example, you might have to migrate data before your go-live. In this case, data can be loaded into one side through data migration and then synced to the other side through the initial synchronization.

We recommend that you use the following approach for the initial synchronization:

  • Single-threaded tables: First migrate data into the finance and operations app, and then trigger the initial synchronization to move the data over to Dataverse. Based on lab testing that Microsoft has done, this sequence has better performance than synchronization from Dataverse to finance and operations apps.
  • Multi-threaded tables: First migrate data into Dataverse, and then trigger the initial synchronization to move the data over to the finance and operations app.

Constraints

Data migration slow-down with enabled dual-write

If you first activate the map in dual-write and then start to import data, migration performance will be poor. We recommend that you not activate running maps in dual-write until the data migration is completed.

Limit of 500,000 rows per run

The maximum number of rows that is allowed through initial synchronization is 500,000 per run. The limit of 500,000 rows applies to each legal table, because each legal entity runs separately. For more information, see Integrate data into Dataverse. In particular, pay attention to the note that states, "To optimize performance and not overload the apps, we currently limit project executions to 500k rows per execution per project."

If there must be more than 500,000 rows in a run when using initial synchronization, we recommend that you migrate data into the finance and operations app and Dataverse separately, and skip the initial synchronization.

Twenty-four-hour limit

If you're running the initial synchronization from Dataverse to the finance and operations app, the import result must be received back from the finance and operations app within 24 hours. Otherwise, a time-out occurs. Therefore, if you're syncing lots of data, and the single run takes more than 24 hours, the initial synchronization might fail because of a time-out. For example, an initial synchronization from Dataverse to a finance and operations app for the Customer/Account table involves 70,000 rows. Therefore, the run might take more than 24 hours and time out.

Don't run the initial synchronization from Dataverse to a finance and operations app for single-threaded tables if the data volume is more than 70,000 rows. Because these tables don't support multi-threading during import, a time-out might occur if the volume is more 70,000 rows. In this situation, you should migrate data into the finance and operations app and Dataverse separately, and skip the initial synchronization.

Currently, there's a limit of 40 legal entities while the environments are being linked. If you try to enable maps where more than 40 legal entities are linked between the environments, you receive the following error message:

Dual-write failure - Plugin registration failed: [(Unable to get partition map for project DWM-1ae35e60-4bc2-4905-88ea-XXXXX. Error Exceeds the maximum partitions allowed for mapping DWM-1ae35e60-4bc2-4905-88ea- XXXXX)], One or more errors occurred.

Initial synchronization isn't currently supported for table maps that have 10 or more lookups

This limitation applies only to the initial synchronization from Dataverse for table maps that have 10 or more lookups. If you run the initial synchronization against a table map that has 10 or more lookups, you might receive the following error message:

5 Attempts to get data from https://XXXX.azure-apim.net/apim... Failed

As a workaround you can split the initial sync into these steps:

  1. Remove some of the lookup columns that aren't mandatory from the dual-write table map and bring the number of lookups to 10.
  2. After the lookup columns are removed, save the map, and do the initial sync.
  3. After the initial sync for the first step is successful, add the remaining lookup columns and remove the lookup columns that were synced in first step. Once again make sure the number of lookup columns is 10. Save the map and run the initial sync. Repeat these steps to make sure all the lookup columns are synced.
  4. Add all the lookup columns back to the map, save the map and run the map with skip initial sync. This enables the map for live sync mode.

Five-minute limit for finance and operations data export

If you're running the initial synchronization from the finance and operations app to Dataverse and the finance and operations data export takes more than five minutes, then the initial sync might time out. The time-out can happen if the data table has virtual columns with the postLoad method, or the export query isn't optimized (for example, if it has missing indexes).

This type of synchronization is supported in Platform update 37 (PU37) and later. Therefore, you should update your finance and operations app to PU37 or later.

Security role for write access

Every user in a customer engagement organization with dual-write must be added to the Dual-Write Runtime User role. Without this role, users are unable to create any rows in tables in the customer engagement organization.

Company and Currency Exchange Tables Required Security Role

Company and currency exchange tables are global in nature and all dual-write users require read access to these two tables. To provide access, all dual-write users need to be added to the Dual-Write App User security role. If a user doesn't have this security role assigned to them, they're unable to read tables that contain Company and Currency values.

Error handling capabilities

Initial synchronization is a full push or incremental push

If an individual row fails to be synchronized, you can't resync only that individual row. The first run of initial synchronization pushes the whole data set. This behavior is known as a full push. When change tracking is enabled in finance and operations apps, the subsequent runs are an incremental push that is based on the last run map version. When change tracking is disabled, subsequent runs cause a full push. During the incremental push, the error records won't be considered for resubmission.

Only the top five errors can be viewed

You can view only the top five errors from the initial synchronization error log.

Double quotes limitation

Because some source fields are processed during the initial sync process, double quotes in your data can cause the initial sync to fail with an error message similar to following:

Initial writes failed while parsing the data for leg:From Dynamics 365 for Finance and Operations Entity to Dynamics 365 for Sales Entity - Legal Entity. Message: Type=Microsoft.VisualBasic.FileIO.MalformedLineException, Msg=Line # cannot be parsed using the current Delimiters.

Addressing this error requires you to locate and remove the double quotes from your data.

Known issues

For information about known issues, see Troubleshoot issues during initial synchronization.

Guidance matrix

finance and operations app instance Dataverse instance Has data to run the initial synchronization Description Maximum volume in a table Single-threaded or multi-threaded Approach
New New No A new finance and operations app instance and a new customer engagement app instance, where neither app has initial data Not applicable Any
  • Activate dual-write, and skip the initial synchronization.
New New Yes A new finance and operations app instance and a new customer engagement app instance, where one of the apps has migrated data < 500,000 Single-threaded
  1. Migrate data to the finance and operations app.
  2. Run the initial synchronization.
< 500,000 Multi-threaded
  1. Migrate data to Dataverse.
  2. Run the initial synchronization.
> 500,000 Any
  1. Migrate data to each app outside the initial synchronization.
  2. Activate dual-write, and skip the initial synchronization.
New Existing Yes A new finance and operations app instance and an existing customer engagement app instance < 70,000 Single-threaded
  1. Create a new company in the finance and operations app.
  2. Bootstrap Dataverse for the company code.
  3. Run the initial synchronization.
> 70,000 Single-threaded
  1. Create a new company in the finance and operations app.
  2. Bootstrap Dataverse for the company code.
  3. Migrate data to each app outside the initial synchronization.
  4. Activate dual-write, and skip the initial synchronization.
< 500,000 Multi-threaded
  1. Create a new company in the finance and operations app.
  2. Bootstrap Dataverse for the company code.
  3. Run the initial synchronization.
> 500,000 Any
  1. Create a new company in the finance and operations app.
  2. Bootstrap Dataverse for the company code.
  3. Migrate data to each app outside the initial synchronization.
  4. Activate dual-write, and skip the initial synchronization.
Existing New Yes An existing finance and operations app instance and a new customer engagement app instance < 500,000 Any
  • Run the initial synchronization.
> 500,000 Any
  1. Migrate data to each app.
  2. Activate dual-write, and skip the initial synchronization.
Existing Existing Yes An existing finance and operations app instance and an existing customer engagement app instance < 70,000 Single-threaded
  1. Bootstrap Dataverse for the company code.
  2. Run the initial synchronization.
> 70,000 Single-threaded
  1. Bootstrap Dataverse for the company code.
  2. Migrate data to each app outside the initial synchronization.
  3. Activate dual-write, and skip the initial synchronization.
< 500,000 Multi-threaded
  1. Bootstrap Dataverse for the company code.
  2. Run the initial synchronization.
> 500,000 Any
  1. Bootstrap Dataverse for the company code.
  2. Migrate data to each app outside the initial synchronization.
  3. Activate dual-write, and skip the initial synchronization.

Single-threaded tables

  • Sales tax codes (msdyn_taxcodes)
  • Customers V3 (accounts)
  • Vendors V2 (msdyn_vendors)
  • Warehouses (msdyn_warehouses)
  • Product categories (msdyn_productcategories)
  • Employment (cdm_employments)
  • Position worker assignments (cdm_positionworkerassignmentmaps)
  • Warehouse locations (msdyn_inventorylocations)
  • Modes of delivery (msdyn_shipvias)