filter data from json

Abedin, Masood 20 Reputation points
2023-03-15T19:57:14.14+00:00

I am new to the Azure Data Factory. I have an UpdatedDate column in this format,

[
  638137843666973190,
  60
]

In the copy data, filter property, I have this code

{
UpdatedDate: {
$gte: ISODate('@{formatDateTime(pipeline().parameters.windowStart)}'),
$lt: ISODate('@{formatDateTime(pipeline().parameters.windowEnd)}')}
}

what formula should I assign as a Default value for windowStart and End parameters.

Thank you.

Azure Cosmos DB
Azure Cosmos DB
An Azure NoSQL database service for app development.
1,434 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,464 questions
{count} votes

Accepted answer
  1. GeethaThatipatri-MSFT 27,007 Reputation points Microsoft Employee
    2023-03-16T20:27:39.13+00:00

    @Abedin, Masood Thanks for providing additional information

    Are you using a tumbling window? if yes you can follow the steps from this link

    from the pipeline go to Edit trigger and from there go to trigger run parameters as shown below

    User's image

    User's image

    User's image

    This splits a trigger and assigns values to the pipeline parameter corresponding to the triggers window. Now you do not need to assign a default value to the pipeline parameter.

    The format looks like "2023-03-16T19:46:00Z"

    If you are not using a tumbling window do not try to evaluate expression in the pipeline parameter. insted either use a veriable or evaluate directly filter like shown below.

    User's image

    use trigger time insted of utcnow for greater consistency, refer to this link for more information.

    Regards

    Geetha

    1 person found this answer helpful.

0 additional answers

Sort by: Most helpful