How to Use Filter Activity Output as a Source in Copy Activity in Azure Data Factory Pipeline

Question

How to Use Filter Activity Output as a Source in Copy Activity in Azure Data Factory Pipeline

Tarun Manoj Kumar Sahu 20

I'm fairly new to Azure Data Factory and need help with a pipeline I'm building. My goal is to read data from a CSV file stored in an Amazon S3 bucket, filter out records where the Status column is 'active', and then copy the filtered data into a SQL Server sink. I’m using a Lookup activity to fetch the data and passing its output (@activity('Lookup1').output.value) to a Filter activity. The filter condition I’ve set is @equals(toLower(item().Status), 'active'). This gives me the filtered records, but I’m unsure how to pass this filtered output into the Copy activity so that only the active records are written to the SQL Server. Any guidance on how to achieve this would be greatly appreciated.

Nandan Hegde 36,146 Reputation points MVP Volunteer Moderator

2025-06-05T10:12:33.4533333+00:00
Any specific reason why you are using a lookup activity to read the data from the S3? There are some limitations w.r.t lookup activity.

The best way would be 2 of the below aspects :

Load the entire data from file into staging table in sink and then use the stored procedure activity to filter the data from stage and load it into destination

Dataflow can be used to read the file and then use filter activity within dataflow and then load that data into destination
Tarun Manoj Kumar Sahu 20 Reputation points

2025-06-05T10:26:14.5233333+00:00

@Nandan Hegde Since I was working with a small sized dataset, I thought using lookup will work. I will implement the solution that you suggested. Thank you for your answer.
J N S S Kasyap 3,625 Reputation points Microsoft External Staff Moderator

2025-06-05T11:01:36.79+00:00

@Tarun Manoj Kumar Sahu
Just checking in to see if the below answer provided by @Nandan Hegde  helped.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Accepted answer

0 additional answers

Your answer

Nandan Hegde 36,146 Reputation points MVP Volunteer Moderator

2025-06-05T10:12:33.4533333+00:00

Any specific reason why you are using a lookup activity to read the data from the S3? There are some limitations w.r.t lookup activity.

The best way would be 2 of the below aspects :

Load the entire data from file into staging table in sink and then use the stored procedure activity to filter the data from stage and load it into destination

Dataflow can be used to read the file and then use filter activity within dataflow and then load that data into destination
Tarun Manoj Kumar Sahu 20 Reputation points

2025-06-05T10:26:14.5233333+00:00

@Nandan Hegde Since I was working with a small sized dataset, I thought using lookup will work. I will implement the solution that you suggested. Thank you for your answer.
J N S S Kasyap 3,625 Reputation points Microsoft External Staff Moderator

2025-06-05T11:01:36.79+00:00

@Tarun Manoj Kumar Sahu
Just checking in to see if the below answer provided by @Nandan Hegde  helped.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Answer 1

Nandan Hegde 36,146 MVP Volunteer Moderator

Dataflow :https://www.youtube.com/watch?v=kHzLDyUHB3E

https://learn.microsoft.com/en-us/azure/data-factory/data-flow-filter

Share via

How to Use Filter Activity Output as a Source in Copy Activity in Azure Data Factory Pipeline

0 additional answers

Your answer