Getting Details for Previously executed ADF Pipeline

Question

Hello All, Context: We are using ADF to move data into Cosmos. For certain tables, we are recreating the container (more like full load of data). While doing so, we are setting the throughput for the container.

Now we are looking for the throughput configured for a given run in the past (as the throughput value has gone through some changes).

Here is what we are looking for and haven't had any luck with google search:

1. For a given run, where can I check the configurations of the Run (e.g. Throughput, Partition Set etc.)
2. Is this information logged somewhere and can we use Log Analytics to query this information. If it not logged already, is there a way that we can log this information?

Answer 1

Hello @Vignesh Iyer and welcome to Microsoft Q&A! I hope this will be a good first experience.

So detail like parallel copies, DIU used, and time taken, can be found in the input/output of an activity run. As a proof of concept, go to the monitoring tab, and select a pipeline run for a pipeline which used a copy activity. Hover mouse between "Activity Name" and "Activity Type", and some options will appear as in below picture.

The arrow-leaving-box button is for "Output" and contains the info I think you are looking for. In my particular example, I have a failed copy from SQL, but the details are still there. See below image.

Once you have confirmed these are the sort of things you are looking for, I can elaborate on other ways to fetch the same details. These include powershell or rest api. I think (80% certain) these are also available in Log Analytics.