Auto incremental load (or run adf pipeline) when there is a change in data using CDC

Question

Hi,

We wanted to add CDC to one of our projects so that we can process data (by loading incrementally). when ever there is a change after. We have followed this pattern to incrementally load the data: https://learn.microsoft.com/en-us/azure/data-factory/tutorial-incremental-copy-change-data-capture-feature-portal

The problem is, that this pattern is a pull mechanism, we have to provide start and end times to check if there was a change in data during the time. But we want it to automatically load data whenever there is any change in the source (Push mechanism). Is there a way to do so?

Answer 1

You can use an Azure Logic App to trigger a pipeline run when data has been changed.

Answer 2

Hi @Obaid UrRehman ,

Thankyou for your question.

As suggested by @Alberto Morillo , using Logic app is efficient way of achieving your requirement as the other way i.e Event based trigger in ADF only works for blob creation and deletion. It doesn't support SQL insertion or modification.

You can set logic app frequency to every minute so that it automatically triggers every minute to check if there is any new insertion in the SQL table.

For more details, kindly check out the following video: Executing a Pipeline from Azure Logic Apps

Hope this will help. Please let us know if any further queries.

------------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you.
  Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators