Debugging ADF Copy Data Activity

Keyur Patel 20 Reputation points
2024-06-05T08:14:24.01+00:00

Hello Techies,

 

Is there any option in ADF to check the incoming data from Source and processed data to sink in the monitor tab just like a Web API call?

 

 

Any help will be much appreciated.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,877 questions
0 comments No comments
{count} votes

Accepted answer
  1. Vlad Costa 615 Reputation points
    2024-06-05T08:30:45.1+00:00

    Hi Keyur,

    You can monitor the data movement from source to sink in many ways, but it might not be exactly like a Web API call (sorry!).

    One of the ways is to use the Copy Activity output. After a data copy activity, you can get the count of records in source and sink. You can store this information in a variable and pass that variable to a stored procedure to insert that value in a table.
    https://stackoverflow.com/questions/62178764/get-count-of-records-in-source-and-sink-in-azure-data-factory

    Another approach is to use Data Flows. In the Mapping Data Flow, you can enable the 'Allow schema drift' option if the source columns change often. This setting allows all incoming source fields to flow through the transformations to the sink.

    https://stackoverflow.com/questions/71559802/perform-data-checks-in-azure-data-factory

    However, if you want to perform data checks, such as checking for null values in some features or if a feature is all nulls, you might need to use a combination of 'Lookup Activity', 'Validation Activity', and 'Delete Activity' with your criteria and rules defined. You can also use the Assert task in Mapping Data Flows.

    https://stackoverflow.com/questions/71559802/perform-data-checks-in-azure-data-factory

    If you find this response helpful and it resolves your issue, please consider marking it as “Accepted” or giving it an upvote. This will help others in the community find the solution more easily.


1 additional answer

Sort by: Most helpful
  1. Amira Bedhiafi 17,786 Reputation points
    2024-06-05T08:28:01.79+00:00

    While there isn't a direct way to view the incoming data from the source and the processed data to the sink within the monitor tab like you would with a Web API call, you can :

    • When designing your pipeline, you can use the Debug mode to run the pipeline and check the intermediate data. This allows you to run the pipeline without triggering a full pipeline execution.
    • Within the Copy Data activity, you can preview the data for both source and sink. This is available in the Source and Sink tabs respectively, where you can click on the "Preview data" button.
    • After running a pipeline, you can check the output of each activity. Go to the Monitor tab, select the pipeline run, and click on the specific activity to see its output. The output includes details like the number of rows copied and any errors that occurred.
    • Enable logging for your Copy Data activity to capture detailed logs. These logs can be written to Azure Blob Storage, Azure Data Lake Storage, or Azure Log Analytics.