Fetch data from dataverse table to Blob Storage(ORC format) using adf

Question

Hi,

I have a requirement to create a pipeline that should perform the incremental load from dataverse table to blob.

It goes like this.

1. Truncate staging tables
2. Set new refresh DateTime variable to current date time
3. Lookup last refresh date
4. load data from dataverse modified/created after last refresh date

Answer 1

Hello @Akshatha ,

Welcome to the Microsoft Q&A platform and thanks for your query.

The incremental copy using from Dataverse using ADF connector is a bit challenging and complicated. Instead, you may explore Azure Synapse Link for Dataverse which contains a SinkModifiedOn column which can be used to get the incremental updates.

After that you can follow Read the Incremental Updates of your Dataverse data tutorial to implement an incremental solution for your Dataverse source using Synapse notebooks or Synapse Data pipeline (same pipelines to that of ADF) by pointing the source to the exported tables in Synapse workspace and then copy to your desired destination store.

Please note that you will have to keep track (for example log TIMESTAMP details in a SQL) of the previous pipelines TIMESTAMP_START and TIMESTAMP_END so that you can query the latest incremental data accordingly.

Hope this will help. Please let us know if any further queries.

------------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators