Azure Data Factory key column watermakr workflow?

Question

1. How to Incrementally copy data using Identity column or incrementing key column instead of time-stamp. We checked many blogs but didn't find how to implement this.
2. This mechanisms is going to be used for ‘N’ # of tables. How can we implement “For-each loop” of multiple table in this case?
3. How can we implement, Exception handling and logging in ADF pipeline:
   • Way to recover and retry attempt
   • Abort the operation and some action to occur to store those record separately
4. If a table throws an error, instead of terminating the pipeline ,the error part should be stored in logs and version of that table shouldn't get updated.

Answer 1

Hello @Anmol Ganju ,

Thanks for the ask and using the forum .

Since you mentioned Identity column I am assuming that your source is a DB and let me assume that the sink is either a DB or a blob . If its a DB , please read tables(s) ( before you start the copy activity ) and get the max identity column value , you can use this value to create a dynamic query with the "Where clause" in the copy activity and only get the latest records . if you have sink as blob , you will have a create a watermark table and update the table when you write the records to the blob .

In the copy activity you do have a option of fault tolerance , i am not sure if that gone through that . On the ask on exception handling , you can use the retry option . For logging errors may be you can see if you can see something like this .

Let me know if you have any further queries .

Thanks
Himanshu