How to create log file for pipeline execution in Azure Synapse Analytics?

Question

What are the steps to create a log file for the execution of the pipeline in Azure Synapse Analytics? I am looking to include specific columns, such as pipeline_run_id, pipeline status, pipeline start time, pipeline end time, pipeline duration, rows read, rows written, error, error message, etc.

Answer 1

Below blog explains the same :

https://datasharkx.wordpress.com/2023/04/20/copying-variable-parameter-into-sink-azure-blob-storage-azure-database-etc-via-data-factory-synapse-pipeline/

You can use ADF system variables to get your values

Answer 2

Hi @ADITYA SINGH

To create a log file for the execution of a pipeline in Azure Synapse Analytics, you can use the built-in logging functionality provided by Synapse Studio. This allows you to monitor the latest status, issues, and progress of your pipelines.

Here are the steps to view the pipeline runs list and filter your pipeline runs:

1. Open Synapse Studio and select your workspace.
2. Select the Monitor section on the left.
3. Select Pipeline runs to view the list of pipeline runs.
4. You can filter the list of pipeline runs to the ones you're interested in by specifying a field on which you'd like to filter. For example, you can filter the view to see only the pipeline runs for the pipeline named "holiday".
5. To view details about your pipeline run, select the pipeline run. Then view the activity runs associated with the pipeline run. If the pipeline is still running, you can monitor the progress.

Regarding the specific columns you mentioned, Synapse Studio provides a set of system variables that you can use to include in your pipeline logs. However, not all the columns you mentioned are available as system variables.

Here are some of the system variables that you can use to include in your pipeline logs:

• pipeline_run_id
• pipeline_name
• activity_name
• activity_run_id
• activity_type
• activity_status
• activity_duration
• rows_read
• rows_written
• error
• error_message

To include these system variables in your pipeline logs, you can use the Set Variable activity in your pipeline and set the value to the system variable you want to include. Then, you can use the Append Variable activity to append the value of the system variable to a log file.

For more information on how to monitor pipeline runs using Synapse Studio, please refer to the official Azure documentation: Monitor pipeline runs using Synapse Studio.

I hope this helps! Let me know if you have any more questions.