Query and analyze the incremental updates
Microsoft Dataverse data can continuously change through create, update, and delete transactions. Synapse Link for Dataverse provides incremental folders to help you query and analyze the incremental updates made to Dataverse data during a user-specified time interval. Organizations with large datasets can analyze the incremental update data and:
- Drop stale and stagnant data to save data storage costs.
- Track data changes during a user-specified time period.
When creating an Azure Synapse Link for Dataverse, you can enable the incremental update feature to create a series of timestamped folders containing only the changes to the Dataverse data that occurred during the user-specified time interval. In each timestamp folder, each exported table is stored under a separate DataverseTableName folder.
Timestamp and table folders are created only when there is a data update during the user-specified time interval.
This feature will apply to all selected tables within Azure Synapse Link for Dataverse and, by default, all the tables selected will be assigned append-only mode with incremental updates.
This feature can't be enabled with the option: Connect to your Azure Synapse workspace. For customers who require Azure Synapse analytics access, follow this instruction to setup the link: Create an Azure Synapse Link for Dataverse with your Azure Synapse Workspace
This feature is designed to work with Azure Data Factory or Synapse Pipeline to copy data from Azure Data Lake Storage Gen2 to an Azure SQL Database. More information:Copy Dataverse data into Azure SQL
Azure Synapse Link for Dataverse. This guide assumes that you have already met the prerequisites to create an Azure Synapse Link with a Synapse workspace. More information: Create an Azure Synapse Link for Dataverse with your Azure Synapse Workspace
Connect Dataverse to Synapse workspace with incremental folder enabled
Sign in to Power Apps and select your environment.
On the left navigation pane, select Azure Synapse Link. If the item isn’t in the left navigation pane, select …More and then select the item you want.
On the command bar, select + New link.
Select the Subscription, Resource group, and Storage account. Select Next.
Add the tables you want to export, and then select Advanced.
Turn on Show advanced configuration settings and Enable Incremental Update Folder Structure.
Enter the time interval (in minutes) for how often the incremental updates should be captured, and then select Save.
Please be advised that time interval configuration is considered final once the setup process is completed and will be grayed out to indicate that it can't be changed.
The minimum time interval is 15 minutes. That means the incremental update folder will be created every 15 minutes and contain the changes that occurred within the time interval.
Ensure Connect to your Azure Synapse workspace Azure Synapse workspace is not checked in the first page of setup.
View incremental folder at Microsoft Azure Storage
Select the desired Azure Synapse Link, and then select Go to Azure data lake on the command bar.
Select the Containers under Data Storage.
Select dataverse-environmentName-organizationUniqueName. Incremental updates folders are named by creation timestamp ("yyyy-MM-dd'T'HH:mm:ss.SSSz") in UTC.
Due to the retry mechanism features, an extra empty timestamp folder might be created within the user-specified time interval.
Submit and view feedback for