Share via

How to automatically backfill if the upstream data is updated after it is initially posted?

Feiyue Jiang 41 Reputation points Microsoft Employee
2020-09-22T02:32:11.767+00:00

I want to capture the recently updated data.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,046 questions
Accepted answer
  1. MartinJaffer-MSFT 26,206 Reputation points
    2020-09-29T22:15:01.617+00:00

    Hello @Feiyue Jiang . I will be assisting you in Himanshu's place.

    If I understand correctly, you are referring to the change feed stream.

    If you want Data Factory pipeline to run whenever a new change has been made, that is a push model. Currently, the only "push" Data Factory accepts, is the Blob Event trigger. Fortunately, this is relatively simple to implement.

    1. Cosmos feed pushes event to processor
    2. Processor writes to Blob
    3. ADF Blob Event trigger subscribes to blob creation events. Pipeline runs.

    Depending upon which cosmos API you are using, there is good support with Azure Functions to act as the processor.

0 additional answers

Sort by: Most helpful
Most helpful Newest Oldest

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.