T-N record pull from API

Karthik Malaiyappan 0 Reputation points
2023-07-17T15:39:41.7966667+00:00

I am using rest linked service to copy the data from web using API and inserting the data into database. every time I run the pipeline it will get the whole data and now I want to restrict the pipeline base on the data like one day before inserted/modified data?

I believe that there should be a way to do that using parameter or some other way? since I am new to ADF I am not pretty aware of it the functionality. Any help would be greatly appreciated.

Thanks.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,211 questions
{count} votes

1 answer

Sort by: Most helpful
  1. Subashri Vasudevan 11,216 Reputation points
    2023-07-18T01:42:20.5366667+00:00

    Hi Karthik

    As you have mentioned, we can pass value of date as a parameter to url. Here is a thread in SO with similar use case, with only change, in the source data set you can use the below expression in relative url. (No need to use parameters in data set)

    @concat('?CreatedAt=',addDays(utcNow(),-1))

    Based on the format of the date, we could use the below expression.

    @concat('?CreatedAt=',formatDateTime(addDays(utcNow(),-1),'MM/dd/yyyy')


Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.