Azure Data Factory Custom Logging

Question

Azure Data Factory Custom Logging

Smitha Krishna Murthy 1

I have an Azure Data Pipeline which has a data flow as below. I need to get the count of records processed in each of these steps so that I can write it to a custom table in Snowflake. How do I achieve this?

KranthiPakala-MSFT 46,642 Reputation points Microsoft Employee Moderator

2022-10-04T01:46:15.847+00:00

Hello @Smitha Krishna Murthy ,

We still have not heard back from you. Just wanted to check if the below suggestion was helpful? If it answers your query, please do click Accept Answer and/or Up-Vote, as it might be beneficial to other community members reading this thread. And, if you have any further query do let us know.

1 answer

Your answer

KranthiPakala-MSFT 46,642 Reputation points Microsoft Employee Moderator

2022-10-04T01:46:15.847+00:00

Hello @Smitha Krishna Murthy ,

We still have not heard back from you. Just wanted to check if the below suggestion was helpful? If it answers your query, please do click Accept Answer and/or Up-Vote, as it might be beneficial to other community members reading this thread. And, if you have any further query do let us know.

Answer 1

Hello @Smitha Krishna Murthy ,

Thanks for the question and using MS Q&A platform.

As per my understanding you would like to get the count of records for desired transformations(stream) and log them to your desired sink. In order to get the count of a transformation output you will have to create a new branch (nothing but a new stream) and then add an Aggregate transformation and leave group by property as empty as it is optional and then in the aggregate configuration, give a column name as rowCountOfX (just an example) and use count() in the expression box.

Please see below example for reference:

Once you have the row count details, then have a sink transformation with your desired data store and map the column to store the row count data.

To explore more about aggregate transformation please refer to this video: Aggregate Transformation in Mapping Data Flow in Azure Data Factory

Hope this will help. Please let us know if any further queries.

------------------------------

Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how

KranthiPakala-MSFT 46,642 Reputation points Microsoft Employee Moderator

2022-09-29T23:08:02.367+00:00

Hello @Smitha Krishna Murthy ,

Just checking in to see if the above answer helped. If this answers your query, do click Accept Answer and Up-Vote for the same. And, if you have any further query do let us know.

Share via

Azure Data Factory Custom Logging

1 answer

Your answer