For desired Synapse pipelines which are executed , I want to check the performance of the Activity . For that I am manually going to each Activity Run > and selecting >Output icon and then Detail icon and getting the information manually . Is there a way to download that information easily. Steps : Azure Synapse Studio > Monitor > Select the required Pipeline > Activity runs > Select required Activity Type > Click Output Icon and Details Icon

Hi @Gaikwad, Shubhangi (Contractor) You can add a diagnostic setting on your synapse workspace and configure the "IntegrationActivityRuns" to be sent to one of the below: log analytics workspace and then analyze your activity outputs using Kusto Query Language Archive it to storage account and then use databricks or synapse serverless to analyze the logs written to the storage account. Hope it helps! Thanks.

For Synapse pipelines how to easily dowload Output and Details

Accepted answer

skumarrana 321 Reputation points

2022-07-04T16:18:29.177+00:00
Hi @Gaikwad, Shubhangi (Contractor)

You can add a diagnostic setting on your synapse workspace and configure the "IntegrationActivityRuns" to be sent to one of the below:

log analytics workspace and then analyze your activity outputs using Kusto Query Language

Archive it to storage account and then use databricks or synapse serverless to analyze the logs written to the storage account.

Hope it helps!

Thanks.
Please sign in to rate this answer.

2 people found this answer helpful.
Gaikwad, Shubhangi (Contractor) 21 Reputation points

2022-07-05T06:37:20.07+00:00

Are there any additional charges after configuring of diagnostic setting .Please share the cost details.

I had missed in the above question , looking for Data written,Copy duration,Throughput,Reading from source etc like details.

skumarrana 321 Reputation points

2022-07-05T14:09:34.037+00:00

Enabling diagnostic settings will not by itself incur any costs but it depends on the platform you choose to ingest, store and analyze the logs generated by the diagnostic settings. Here we are looking at two platforms:

Log analytics workspace

The diagnostic settings for synapse will export the logs to log analytics workspace - there is cost to export (please see the link: https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/resource-logs-categories#microsoftsynapseworkspaces)

There will be additional charges for data storage, retention and analysis of the data. Please see the link: https://azure.microsoft.com/en-us/pricing/details/monitor/

Storage Account (this is a cheaper option and you may want to try this one and explore)

Storage account will also incur charges based on the amount of data stored, the type of replication configured etc. https://azure.microsoft.com/en-us/pricing/details/storage/blobs/

Yes, you will be able to see all the attributes of the ADF pipeline activity run and the numbers captured for each of those attributes. Here is a snippet of a sample activity run from the log file exported through diagnostic settings (JSON format)

Nathan 1 Reputation point

2022-09-05T11:08:08.483+00:00

Hi,

In the log data, is it possible to calculate the pipeline's runtime cost?
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

For Synapse pipelines how to easily dowload Output and Details

0 additional answers

Your answer