Logging parameters of ForEach Activity

Question

Hi, I have a very simple pipeline, like this:

I have a diagnostic setting for this ADF pointing to a Log Analytics Workspace. I need to log and query via KQL the Copy Activity (located inside the ForEach) with its corresponding iteration paramenters. However, when querying Log Analytics, there is nothing mentioning the parameters a specific iteration ran with. Can you give a hand on this? Regards

Answer 1

Thanks @Jona for sharing your resolution here. Reiterating this information here to ensure it reaches a broader audience.

It simply involves delving further into the input field of the logs, where you can find the iteration parameters for your activity. Another workaround is to use user properties which means to replicate the iteration parameters into user properties.

Answer 2

Actually, it's simpler. It was just matter of getting deeper into the input field. There you can see the iteration parameters of your activity.

Another workaround was to use user properties. I mean to replicate the iteration parametros into user properties.

Regards

Answer 3

Since ADF does not automatically log the iteration parameters of a ForEach activity, you need to implement custom logging within your activities. You can create an Azure Function or Logic App to log the necessary information, including any parameters and values you want to track, to your Log Analytics Workspace. Within your ForEach activity, before or after the Copy activity, add a Web activity that calls your Azure Function or Logic App. Pass the iteration parameters to this Web activity. This way, each iteration of the ForEach loop will log its parameters via the Azure Function or Logic App. Within your Azure Function or Logic App, use the Azure Monitor Data Collector API to log the details you're interested in. This involves making an HTTP POST request to the API endpoint with the log data, including your ForEach iteration parameters. Once your custom logs are in place, you can use KQL to query them :

CustomActivityLogs_CL
| where ActivityName_s == "CopyActivityName"
| project ForEachParameter_s, ActivityRunId_s, TimeGenerated, OtherDetails_s
| order by TimeGenerated desc