Azure Data Factory Data Copy from Cosmos

Tahmid Eshayat 286 Reputation points
2020-06-26T03:57:27.08+00:00

We have one scenario, where we have to transfer/copy newly added/updated data from cosmos dB to a SQL Server every 2 hours.
Currently, data copy copies all the files, can I configure to only copy the newly added/updated ?

Thanks

Azure Functions
Azure Functions
An Azure service that provides an event-driven serverless compute platform.
4,264 questions
Azure Cosmos DB
Azure Cosmos DB
An Azure NoSQL database service for app development.
1,442 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,545 questions
0 comments
Accepted answer
  1. Jaryd 181 Reputation points
    2020-06-26T04:05:08.113+00:00

    I’m not sure if mine is the best way to do this, maybe there is a built in incremental refresh but...

    I use a lookup activity to select the max date time from the sink. Save that to a variable.

    Then use a select query with the the variable in the where clause to only pull new ones From the source.

0 additional answers

Sort by: Most helpful
Most helpful Newest Oldest