Patterns
This unit presents common patterns (best practices) that have been seen among clients and the use of dual-write to provide a better understanding of the real-life implications and uses of integration. From these real-life scenarios are the following patterns (best practices) and anti-patterns (pitfalls) for dual-write, initial sync, live sync, and other processes.
Initial sync
Dual-write lets you sync initial data between finance and operations apps and Dataverse. The initial sync is best used to create the initial data of low volumes; it’s not intended to handle large loads of data for data migration purposes. In fact, you shouldn’t enable the dual-write table mapping or do an initial sync until all data migration processes are complete. The initial sync provides data from your destination application to the newer implementation.
For example, if you have a complete finance and operations implementation, and you’ve added a Microsoft Dynamics 365 Customer Engagement implementation, then the initial sync can be useful in helping you transfer existing data from finance and operations apps to Customer Engagement to get started on the Customer Engagement implementation without waiting for a data migration scenario. Additionally, dual-write initial sync will not overwrite existing data in either environment.
If you need to remap or re-create data in either system, we recommend that you clean up the environment before doing the initial sync. Make sure that you check your master data for the initial sync and then ensure that all fields, filtered fields, and legal entities are planned for the initial sync. A limit is in place of 500,000 rows for each run and 70,000 for a single-threaded table. If the initial sync is above these limits, then you should migrate data into each environment (finance and operations apps or Dataverse) separately and then skip the initial synchronization. Another option is to synchronize several legal entities, one at a time, rather than all at once.
For more information, see Considerations for initial synchronization.
Piggybacking, or async implementation
Dual-write sync is triggered as the last step before committing Database transactions at the source application. The same transaction content for all relevant records is sent to the target app as a single batch. Only when the entire batch is accepted at the target application will the source database transaction be committed. Fields that are set in the target application as part of the create or update transaction won’t sync back to the source. This scenario is referred to as piggybacking and it’s disallowed to prevent an infinite, circular integration.
For example, a Sales Quote line is created in Microsoft Dynamics 365 Sales. That Sales Quote line record syncs to finance and operations apps and is created in that application. As part of the creation in finance and operations apps, some business logic runs and defaults several fields, such as Site and Warehouse. These fields aren’t set back in Dynamics 365 Sales. In this case, we recommend that you replicate the default logic in Dynamics 365 Sales to match the default records in finance and operations apps.
Occasionally, another option is to move the default logic to a separate, asynchronous job that can update the record in a separate transaction and safely trigger a sync back to the source application. This process is considered an anti-pattern (pitfall) because performance could be affected, so you should evaluate it on a case-by-case basis.
Custom triggers on lookup columns in customer engagement apps
This scenario relates specifically to master and detail or header and lines in dual-write.
Example
You’re on the company’s sales floor, and your only task is to add a quote into the app for Dynamics 365 Sales. You want to complete this task quickly, with no extras, and no requirements from the system’s perspective. You talk to a customer, enter their information, and create a quote. Later, you’ll fill in the rest of the information, when it will be ready to sync between Dynamics 365 Sales and Dynamics 365 Supply Chain Management.
In this case, you want to enter a quote and sync the quote only when it is marked as ready for integration. To do so, you can create a custom column on the quote header to use as a trigger when it’s ready for integration. Next, you can create a custom column on the quote line table as a calculated field. Then, you’ll set the filter on the dual-write Quote header table map and Quote line map for those fields for the correct value in the calculated columns. As the custom column in the header is updated, the dual-write trigger will work and will trigger the dual-write integration on the related rows. This process creates an effective on-demand integration based on the update of the custom columns.
finance and operations apps transactions – Header and lines in the same transaction
For any transaction, finance and operations apps creates data in a batch and then sends it as a batch to Dataverse. If two records are being created as a part of the same transaction, and they reference each other, an error will occur, such as “Unable to write data to entity X. Unable to look up entity with values XXXX. Unable to look up entity X with values XXXX.” In this case, the mitigation for the issue is to create entity relationships within finance and operations apps to indicate to the dual-write platform that the two entities are linked and that relationships exist between the two records that, in this case, are processing in the same transaction.
Virtual tables and dual-write
The Get started integrating finance and operations apps with Microsoft Power Platform and the Use key concepts and setup to integrate finance and operations apps with Microsoft Power Platform modules described virtual entities as an option for accessing data between finance and operations apps and Dataverse without physically copying the information between the two applications. The data will not exist in both databases for the applications and will only reside in finance and operations apps. To decide on the better-fitting option for your data, you should explore a few design considerations.
The data residency of virtual tables is an important consideration. If the business process requires that you physically copy the data, then virtual tables and entities might not be the correct solution for you. In this case, you’ll want to use the dual-write integration to ensure that both applications have a copy of the data.
Consider asking yourself the following data analytics questions:
- Do the business needs dictate that the analytics need to rely on source data?
- Are latency requirements in place?
- Is a requirement for low latency established for the data that needs to be copied from system to system?
- Are the data transactional and data replication costs high?
Virtual tables don’t copy physical data into the Dataverse applications. Virtual tables provide a low latency option for data that needs to be copied or replicated and can be a helpful solution for data replication. Because virtual tables don’t physically copy data into Dataverse, you’ll want to consider this solution by using virtual tables to house and process data for transactional data, such as retail transactions or on-hand inventory.
However, dual-write addresses scenarios where data needs to be physically copied between applications.