Azure Data Factory Copy Data, Azure table storage to table storage only update if Timestamp of destination row is smaller

Gavin Armitage 1 Reputation point

Would like some help with the following scenario, I have setup a simple copy from one Azure table to another Azure table. What I want to do is only update the destination row if the timestamp of the source row is greater then the destination row. On the Sink tab i use the Insert Type, can see the Merge/Insert option and one that says dynamic content. Is there a way to achieve this?

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
7,174 questions
0 comments No comments
{count} votes

1 answer

Sort by: Most helpful
  1. HimanshuSinha-msft 601 Reputation points

    Hello @Gavin Armitage

    You can acheive this using copy acivity , but i think we will have to make some adjustments . Since the idea to get all the new records from the source which are not in destination . In SQL world this can be something get max(timestamp) from the destination table and then use that on the source query . So in the context of the issue in hand

    we are looking for a query like

    Timestamp gt datetime'2020-06-04T19:34:07.534Z'

    To make it dynamic , I am using a variable activity and contruct the date clause

    @concat('Timestamp gt datetime',''' , addMinutes(utcNow(), -10) ,''')   

    , this clause will give me the all the records in the last 10 mins . You can run the trigger at every 10 mins and you should have the records in the destination .

    I am adding the image for clarity , please so let me know as to how it goes .

    Unfortunately tablke storage are not good for aggregation .