Logging parameters of ForEach Activity

Jona 265 Reputation points
2024-02-07T14:54:08.7166667+00:00

Hi, I have a very simple pipeline, like this: User's image

I have a diagnostic setting for this ADF pointing to a Log Analytics Workspace. I need to log and query via KQL the Copy Activity (located inside the ForEach) with its corresponding iteration paramenters. However, when querying Log Analytics, there is nothing mentioning the parameters a specific iteration ran with. Can you give a hand on this? Regards

Azure Monitor
Azure Monitor
An Azure service that is used to collect, analyze, and act on telemetry data from Azure and on-premises environments.
2,667 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,063 questions
0 comments No comments
{count} votes

Accepted answer
  1. SwathiDhanwada-MSFT 16,571 Reputation points
    2024-02-16T05:33:31.55+00:00

    Thanks @Jona for sharing your resolution here. Reiterating this information here to ensure it reaches a broader audience.

    It simply involves delving further into the input field of the logs, where you can find the iteration parameters for your activity. User's image Another workaround is to use user properties which means to replicate the iteration parameters into user properties. User's image

    1 person found this answer helpful.
    0 comments No comments

2 additional answers

Sort by: Most helpful
  1. Jona 265 Reputation points
    2024-02-13T13:40:39.59+00:00

    Actually, it's simpler. It was just matter of getting deeper into the input field. There you can see the iteration parameters of your activity. User's image

    Another workaround was to use user properties. I mean to replicate the iteration parametros into user properties. User's image

    Regards

    1 person found this answer helpful.

  2. Amira Bedhiafi 12,136 Reputation points
    2024-02-10T15:11:56.3533333+00:00

    Since ADF does not automatically log the iteration parameters of a ForEach activity, you need to implement custom logging within your activities. You can create an Azure Function or Logic App to log the necessary information, including any parameters and values you want to track, to your Log Analytics Workspace. Within your ForEach activity, before or after the Copy activity, add a Web activity that calls your Azure Function or Logic App. Pass the iteration parameters to this Web activity. This way, each iteration of the ForEach loop will log its parameters via the Azure Function or Logic App. Within your Azure Function or Logic App, use the Azure Monitor Data Collector API to log the details you're interested in. This involves making an HTTP POST request to the API endpoint with the log data, including your ForEach iteration parameters. Once your custom logs are in place, you can use KQL to query them :

    CustomActivityLogs_CL
    | where ActivityName_s == "CopyActivityName"
    | project ForEachParameter_s, ActivityRunId_s, TimeGenerated, OtherDetails_s
    | order by TimeGenerated desc
    
    
    0 comments No comments