Azure data factory

Question

1.The job should only extract new records from the source database

2.The job should be able to identify these new records without having to record job start times/record numbers etc.in a database

HOW TO ACHIEVE IT

Answer 1

Hello @Anandazure ,
Thanks for the ask and using Microsoft Q&A platform .
As I understand at this time the intend to copy records from SQL , but unfortunately you have not mentioned what is the sink , I am assuming ist SQL .

You have also not mentioned the record volume , if the record volume is less , may be you can use

Option 1

1. Use a Lookup , read the source
2. Use a foreach(FE) loop
3. Inside the FE loop call a Store procedure activity and call a SP on the sink side and INSERT the record .

Option 2

1. You can use copy activity and dump all the records from the source .
2. Call a Store procedure activity and call a SP on the sink side and INSERT the record.

Option 3

1. You can also use the mapping data flow with Alter row transformation to achieve this .

Please do let me know how it goes .
Thanks
Himanshu

-------------------------------------------------------------------------------------------------------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators