Debugging ADF Copy Data Activity

Question

Hello Techies,

 

Is there any option in ADF to check the incoming data from Source and processed data to sink in the monitor tab just like a Web API call?

 

 

Any help will be much appreciated.

Answer 1

Hi Keyur,

You can monitor the data movement from source to sink in many ways, but it might not be exactly like a Web API call (sorry!).

One of the ways is to use the Copy Activity output. After a data copy activity, you can get the count of records in source and sink. You can store this information in a variable and pass that variable to a stored procedure to insert that value in a table.
https://stackoverflow.com/questions/62178764/get-count-of-records-in-source-and-sink-in-azure-data-factory

Another approach is to use Data Flows. In the Mapping Data Flow, you can enable the 'Allow schema drift' option if the source columns change often. This setting allows all incoming source fields to flow through the transformations to the sink.

https://stackoverflow.com/questions/71559802/perform-data-checks-in-azure-data-factory

However, if you want to perform data checks, such as checking for null values in some features or if a feature is all nulls, you might need to use a combination of 'Lookup Activity', 'Validation Activity', and 'Delete Activity' with your criteria and rules defined. You can also use the Assert task in Mapping Data Flows.

https://stackoverflow.com/questions/71559802/perform-data-checks-in-azure-data-factory

If you find this response helpful and it resolves your issue, please consider marking it as “Accepted” or giving it an upvote. This will help others in the community find the solution more easily.

Answer 2

While there isn't a direct way to view the incoming data from the source and the processed data to the sink within the monitor tab like you would with a Web API call, you can :

• When designing your pipeline, you can use the Debug mode to run the pipeline and check the intermediate data. This allows you to run the pipeline without triggering a full pipeline execution.
• Within the Copy Data activity, you can preview the data for both source and sink. This is available in the Source and Sink tabs respectively, where you can click on the "Preview data" button.
• After running a pipeline, you can check the output of each activity. Go to the Monitor tab, select the pipeline run, and click on the specific activity to see its output. The output includes details like the number of rows copied and any errors that occurred.
• Enable logging for your Copy Data activity to capture detailed logs. These logs can be written to Azure Blob Storage, Azure Data Lake Storage, or Azure Log Analytics.