question

GavinArmitage-1615 avatar image
0 Votes"
GavinArmitage-1615 asked HimanshuSinha-MSFT commented

Azure Data Factory Copy Data, Azure table storage to table storage only update if Timestamp of destination row is smaller

Would like some help with the following scenario, I have setup a simple copy from one Azure table to another Azure table. What I want to do is only update the destination row if the timestamp of the source row is greater then the destination row. On the Sink tab i use the Insert Type, can see the Merge/Insert option and one that says dynamic content. Is there a way to achieve this?

azure-data-factory
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

HimanshuSinhamfst-5269 avatar image
0 Votes"
HimanshuSinhamfst-5269 answered HimanshuSinha-MSFT commented

Hello @GavinArmitage-1615

You can acheive this using copy acivity , but i think we will have to make some adjustments . Since the idea to get all the new records from the source which are not in destination . In SQL world this can be something get max(timestamp) from the destination table and then use that on the source query . So in the context of the issue in hand

we are looking for a query like

Timestamp gt datetime'2020-06-04T19:34:07.534Z'

To make it dynamic , I am using a variable activity and contruct the date clause

 @concat('Timestamp gt datetime',''' , addMinutes(utcNow(), -10) ,''') 

, this clause will give me the all the records in the last 10 mins . You can run the trigger at every 10 mins and you should have the records in the destination .

I am adding the image for clarity , please so let me know as to how it goes .







Unfortunately tablke storage are not good for aggregation .




9134-adf-ui.gif




adf-ui.gif (2.7 MiB)
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

thanks for this, I am currently already doing this. The question is around how can I check the destination that I want to update. If you click on the Sink tab there are 3 options, Insert/Merge/Dynamic content. I want to check the destination row before the update and compare on timestamp. Is this possible with Dynamic content?

0 Votes 0 ·

Hello @GavinArmitage-1615


My sincere apoloziges here for the delay response .


As i understand you are trying to check at the runtime . I don;t think its possible in copy activity .


Thanks & stay safe


Himanshu




Please do consider to click on "Accept Answer" and "Up-vote" on the post that helps you, as it can be beneficial to other community members


0 Votes 0 ·