Share via

copy activity - select * from .csv dataset

arkiboys 9,696 Reputation points
2021-09-15T08:32:55.077+00:00

Hello,
At present I read all the data inside a .csv file which sits in the blobstorage container.
On each run of the data factory, based on the parameters based I want to select only certain rows, i.e.
select * from (dataset of the csv) where field1 = @param1 and field3 = @param2

Is this possible?
I can do the exact same thing in copy activity pointing to sql server but not sure how this is achieved reading from csv.

Thank you

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,007 questions
0 comments
Accepted answer
  1. Saurabh Sharma 23,821 Reputation points Microsoft Employee
    2021-09-15T19:07:18.853+00:00

    Hi @arkiboys ,

    Thanks for using Microsoft Q&A !!
    This is not possible in Copy Activity and you need to use other alternatives like Data Flow to filter out unwanted records. OR you can try using lookup activity to get the records and then add logic to exclude the records if it works for you.

    Thanks
    Saurabh

1 additional answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. arkiboys 9,696 Reputation points
    2021-09-15T19:42:24.59+00:00

    I see.
    In that case:
    1-
    In lookup activity, I read the .csv file
    2-
    create a dataflow to filter the rows being read from .csv
    i.e. Field3 = @param1
    How is this achieved?

    Thank you

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.