How to use/pass comparison query operators (like Greater Than $gt/Less Than or Equal To $lte, etc) in Filter condition in Source Dataset of Azure COSMOS for Mongo DB in the COPY activity?

Question

How to use/pass comparison query operators (like Greater Than $gt/Less Than or Equal To $lte, etc) in Filter condition in Source Dataset of Azure COSMOS for Mongo DB in the COPY activity?

Astha Chaturvedi 0

Hello Team,

I am sourcing data from Azure COSMOS DB for Mongo DB and load the same to Azure SQL Server DB. The data has to be read/loaded incrementally from source in every run based on a Date column.

Could you please help me on how I can pass/use the comparison query operators (like Greater Than $gt/Less Than or Equal To $lte, etc) in Filter condition in Source Dataset of Azure COSMOS for Mongo DB in the COPY activity in order to read the incremental data in every run based on aCOSMOS_SourceDS_FilterCondition.jpg date column?

Thank you

Regards,

Astha Chaturvedi

COSMOS_SourceDS_FilterCondition.jpg

Smaran Thoomu 24,110 Reputation points Microsoft External Staff Moderator

2024-07-03T10:51:06.76+00:00
Hi @Astha Chaturvedi

Thanks for the question and using MS Q&A platform.

To use comparison query operators in the filter condition of the source dataset for Azure Cosmos DB for Mongo DB in the COPY activity, you can use the MongoDB query syntax in the filter expression.

For example, if you want to filter the data based on a date column named myDateColumn and retrieve only the documents where the value of myDateColumn is greater than or equal to a specific date, you can use the following filter expression:

{ "myDateColumn": { "$gte": ISODate("2022-01-01T00:00:00.000Z") } }

In this example, the $gte operator is used to specify that the value of myDateColumn should be greater than or equal to the date 2022-01-01T00:00:00.000Z. You can replace this date with the specific date that you want to use for filtering.

Similarly, you can use other comparison query operators such as $lt, $lte, $gt, $ne, etc. in the filter expression as per your requirement.

Once you have defined the filter expression, you can use it in the filter condition of the source dataset in the COPY activity to retrieve only the documents that match the filter criteria.

I hope this helps! Let me know if you have any further questions.
Smaran Thoomu 24,110 Reputation points Microsoft External Staff Moderator

2024-07-04T09:48:44.3366667+00:00

@Astha Chaturvedi We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Smaran Thoomu 24,110 Reputation points Microsoft External Staff Moderator

2024-07-17T12:12:04.46+00:00

@Astha Chaturvedi We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

Your answer

Smaran Thoomu 24,110 Reputation points Microsoft External Staff Moderator

2024-07-03T10:51:06.76+00:00

Hi @Astha Chaturvedi

Thanks for the question and using MS Q&A platform.

To use comparison query operators in the filter condition of the source dataset for Azure Cosmos DB for Mongo DB in the COPY activity, you can use the MongoDB query syntax in the filter expression.

For example, if you want to filter the data based on a date column named myDateColumn and retrieve only the documents where the value of myDateColumn is greater than or equal to a specific date, you can use the following filter expression:

{ "myDateColumn": { "$gte": ISODate("2022-01-01T00:00:00.000Z") } }

In this example, the $gte operator is used to specify that the value of myDateColumn should be greater than or equal to the date 2022-01-01T00:00:00.000Z. You can replace this date with the specific date that you want to use for filtering.

Similarly, you can use other comparison query operators such as $lt, $lte, $gt, $ne, etc. in the filter expression as per your requirement.

Once you have defined the filter expression, you can use it in the filter condition of the source dataset in the COPY activity to retrieve only the documents that match the filter criteria.

I hope this helps! Let me know if you have any further questions.
Smaran Thoomu 24,110 Reputation points Microsoft External Staff Moderator

2024-07-04T09:48:44.3366667+00:00

@Astha Chaturvedi We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Smaran Thoomu 24,110 Reputation points Microsoft External Staff Moderator

2024-07-17T12:12:04.46+00:00

@Astha Chaturvedi We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

Share via

How to use/pass comparison query operators (like Greater Than $gt/Less Than or Equal To $lte, etc) in Filter condition in Source Dataset of Azure COSMOS for Mongo DB in the COPY activity?

Your answer