Capture Records from a ZIP file in a Copy Data Activity - ADF

Michael Mimbela Arenaza 41 Reputation points
2022-03-09T13:45:02.553+00:00

How can I capture the records decompressed by a Copy Data Activity in Azure DataFactory? I have used this function "activity('NameofActivity').output.RowsCopied" but it throws the following attached error: Operation on target Stp_Ins_BITACORA_FLUJO_PROCESO_03_TCPZ failed: The expression 'activity('Cpy_Unzip_TCPZ145').output.RowsCopied' cannot be evaluated because property 'RowsCopied' doesn't exist, available properties are 'dataRead, dataWritten, filesRead, filesWritten, sourcePeakConnections, sinkPeakConnections, copyDuration, throughput, errors, effectiveIntegrationRuntime, usedDataIntegrationUnits, billingReference, usedParallelCopies, executionDetails, dataConsistencyVerification, durationInQueue'.
When i use output.dataWritten or output.dataRead donde give me the records in the file.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,196 questions
0 comments No comments
{count} votes

Accepted answer
  1. HimanshuSinha-msft 19,386 Reputation points Microsoft Employee
    2022-03-10T23:00:49.237+00:00

    Thanks @Nasreen Akter & @Pratik Somaiya .
    Hello @Michael Mimbela Arenaza ,
    I concure with @nasreen , when you are decompressing a file , you cannot get the row count .
    You never mentioned what kind of uncompressed files are there , lets assume if you have csv files , You can do

    1. Add one more copy activity ,
    2. Copy the records( not file ) from source to a sink ( may be a container)
    3. Once the copy is done , you will be see the row data as called out by @Nasreen Akter .
    4. Delete the sink data as we do not need that anyways .

    Please do let me if you have any queries.
    Thanks
    Himanshu


    • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification
      • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators
    0 comments No comments

2 additional answers

Sort by: Most helpful
  1. Nasreen Akter 10,791 Reputation points
    2022-03-09T14:02:48.593+00:00

    Hi @Michael Mimbela Arenaza ,

    Thank you for the ask. rowsCopied attribute does not always present in the CopyActivity-->output. Please see the documentation below.
    "rowsCopied: Number of rows copied to sink. This metric does not apply when copying files as-is without parsing them, for example, when source and sink datasets are binary format type, or other format type with identical settings."

    https://learn.microsoft.com/en-us/azure/data-factory/copy-activity-monitoring?tabs=data-factory#monitor-programmatically

    You can see the attributes after running the CopyActivity --> then go to the output (please see the screenshot below). Hope this helps, thanks!

    181486-image.png

    181505-image.png

    0 comments No comments

  2. Pratik Somaiya 4,206 Reputation points
    2022-03-10T06:40:20.083+00:00

    Hello @Michael Mimbela Arenaza

    Are you selecting a Compression Type parameter in the Copy Activity

    https://learn.microsoft.com/en-us/azure/data-factory/supported-file-formats-and-compression-codecs-legacy#compression-support

    As correctly mention by @Nasreen Akter you can get the rowsCopied from activity output

    But if you need to store it then you will have to perform another logic to count number of rows from the decompressed file and then store it

    0 comments No comments