Check incremental data in db

Question

Check incremental data in db

Vineet S 1,390

Hi how to use event base trigger to check if if any incremental data loaded in sql server which is source database the pipeline should loading that data.. Pls share screen shot

Nandan Hegde 36,146 Reputation points MVP Volunteer Moderator

2024-08-02T03:15:11.6133333+00:00

Can you please mention whether it is Azure SQL or On Prem SQL server?
And in case of on prem SQL server, what is the version of the SQL server?
Vineet S 1,390 Reputation points

2024-08-03T04:42:55.82+00:00

Azure sql server
NIKHILA NETHIKUNTA 4,600 Reputation points Microsoft External Staff

2024-08-05T09:07:07.3333333+00:00

Hi @Vineet S
Thank you for your question and using the Microsoft Q&A platform.
To set up a pipeline in Azure Data Factory that checks for incremental data in a SQL Server source database and loads that data you can refer to the below link:

https://learn.microsoft.com/en-us/azure/data-factory/tutorial-incremental-copy-change-data-capture-feature-portal

Hope this helps. Please let us know if you have any further questions.
NIKHILA NETHIKUNTA 4,600 Reputation points Microsoft External Staff

2024-08-06T16:11:35.9233333+00:00

Hi @Vineet S
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
NIKHILA NETHIKUNTA 4,600 Reputation points Microsoft External Staff

2024-08-07T16:38:59.7133333+00:00

Hi @Vineet S
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

1 answer

Your answer

Nandan Hegde 36,146 Reputation points MVP Volunteer Moderator

2024-08-02T03:15:11.6133333+00:00

Can you please mention whether it is Azure SQL or On Prem SQL server?
And in case of on prem SQL server, what is the version of the SQL server?
Vineet S 1,390 Reputation points

2024-08-03T04:42:55.82+00:00

Azure sql server
NIKHILA NETHIKUNTA 4,600 Reputation points Microsoft External Staff

2024-08-05T09:07:07.3333333+00:00

Hi @Vineet S
Thank you for your question and using the Microsoft Q&A platform.
To set up a pipeline in Azure Data Factory that checks for incremental data in a SQL Server source database and loads that data you can refer to the below link:

https://learn.microsoft.com/en-us/azure/data-factory/tutorial-incremental-copy-change-data-capture-feature-portal

Hope this helps. Please let us know if you have any further questions.
NIKHILA NETHIKUNTA 4,600 Reputation points Microsoft External Staff

2024-08-06T16:11:35.9233333+00:00

Hi @Vineet S
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
NIKHILA NETHIKUNTA 4,600 Reputation points Microsoft External Staff

2024-08-07T16:38:59.7133333+00:00

Hi @Vineet S
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

Answer 1

To have an event based trigger, you can enable triggers within Azure SQL database to trigger stored procedure on update/insert/delete of any data from a table.

And via Stored procedure, you can use the External REST API endpoint to trigger an ADF pipeline via REST API :

https://learn.microsoft.com/en-us/shows/azure-friday/calling-external-rest-endpoints-from-azure-sql-database

The below blog provides the details for event trigger :https://datasharkx.wordpress.com/2022/12/02/event-trigger-azure-data-factory-synapse-pipeline-via-azure-sql-database/

And then use the watermark way or CDC way for incremental load :

https://learn.microsoft.com/en-us/azure/data-factory/tutorial-incremental-copy-change-data-capture-feature-portal

Share via

Check incremental data in db

1 answer

Your answer