Query and analyze the incremental updates

Microsoft Dataverse data can continuously change through create, update, and delete transactions. Synapse Link for Dataverse provides incremental folders to help you query and analyze the incremental updates made to Dataverse data during a user-specified time interval. Organizations with large datasets can analyze the incremental update data and:

• Drop stale and stagnant data to save data storage costs. 
• Track data changes during a user-specified time period.

When creating an Azure Synapse Link for Dataverse, you can enable the incremental update feature to create a series of timestamped folders containing only the changes to the Dataverse data that occurred during the user-specified time interval. In each timestamp folder, each exported table is stored under a separate DataverseTableName folder.

Important

Timestamp and table folders are created only when there is a data update during the user-specified time interval.

This feature will apply to all selected tables within Azure Synapse Link for Dataverse and, by default, all the tables selected will be assigned append-only mode with incremental updates.

This feature can't be enabled with the option: Connect to your Azure Synapse workspace. For customers who require Azure Synapse analytics access, follow this instruction to setup the link: Create an Azure Synapse Link for Dataverse with your Azure Synapse Workspace

This feature is designed to work with Azure Data Factory or Synapse Pipeline to copy data from Azure Data Lake Storage Gen2 to an Azure SQL Database. More information:Copy Dataverse data into Azure SQL

Prerequisites

Azure Synapse Link for Dataverse. This guide assumes that you have already met the prerequisites to create an Azure Synapse Link. More information: Create an Azure Synapse Link for Dataverse with Azure Data Lake

Connect Dataverse to Synapse workspace with incremental folder enabled

1. Sign in to Power Apps and select your environment.

2. On the left navigation pane, select Azure Synapse Link. If the item isn’t in the side panel pane, select …More and then select the item you want.

3. On the command bar, select + New link.

4. Select the Subscription, Resource group, and Storage account. Select Next. 

5. Add the tables you want to export, and then select Advanced. 

6. Turn on Show advanced configuration settings and Enable Incremental Update Folder Structure.

7. Enter the time interval (in minutes) for how often the incremental updates should be captured, and then select Save. 

   

Note

The minimum time interval is 15 minutes. That means the incremental update folder will be created every 15 minutes and contain the changes that occurred within the time interval. This setting is also configurable after the link creation via Manage tables

Ensure Connect to your Azure Synapse workspace Azure Synapse workspace is not checked in the first page of setup.

View incremental folder at Microsoft Azure Storage

1. Select the desired Azure Synapse Link, and then select Go to Azure data lake on the command bar.

2. Select the Containers under Data Storage.

3. Select dataverse-environmentName-organizationUniqueName. Incremental updates folders are named by creation timestamp ("yyyy-MM-dd'T'HH:mm:ss.SSSz") in UTC.

   

Note

Due to the retry mechanism features, an extra empty timestamp folder might be created within the user-specified time interval.

See also

What is Azure Synapse Link for Dataverse?