Counting rows of a CSV file using Get Metadata in Azure Data Factory

Question

How can I use Get Metadata activity in Azure Data Factory to count the number of rows of a CSV file? I need to perform this task as part of my pipeline but cannot find a suitable solution. Any suggestions or examples would be greatly appreciated.

Answer 1

Hey,

Not sure whether 1 can use get meta data activity to get the row count.

You can use the below options:

1. Lookup activity: but there is restriction of 5000 rows
2. Dataflow : [Source (csvfile) -> DerivedColumn (rowcount) -> Aggregate (count(rowcount)) -> sink (cache the output of aggregate)]
3. use copy activity and copy the data from csv into a SQL table and use :

      @activity('Copy data1').output.rowsRead
   

Answer 2

Hi Aditya Singh ,

Thankyou for posting your query on Microsoft Q&A platform. As per my understanding you want to get the number of rowcount in ADF .

Other approach apart from the one shared by Nandan , would be to use a single copy activity to load the file from one folder to another within your ADLS and capture the rowsread property from the copy activity output.

You can use the below expression for achieving the requirement:

@activity('copyActivity').output.rowsRead

Here is how the output of copy activity would look like:

Hope it helps. Kindly accept the answer by clicking on Accept answer button. Thankyou