Diagnostic Logs for Synapse Analytics

Question

Hi Team - I have activated diagnostic logging for my Synapse workspace to log all the operational details on the pipeline/activity runs. When I explore SynapseIntegrationActivityRuns table in Log analytics workspace, I don’t see the error messages, error codes and details being logged in it.(( activity errors such as Notebook,Copy, etc)

 

Meanwhile, ADF counterpart to this log event (ADFActivityRun) does include these and they are extremely valuable for monitoring the system and debugging. Are there any plans to add these fields as part of the logging? 

Please suggest if they are any alternative solutions.

Thanks,

Abhiram

Answer 1

Hello Abhiram,

Hope you are doing well today. It's possible that the error messages, error codes, and details you're looking for are not included in the SynapseIntegrationActivityRuns table by default. However, you can customize the diagnostic logging to include the information you need.

To do this, you can create a custom logging schema for your Synapse workspace. This will allow you to include additional fields in the logs, such as error messages and codes. Here are the high-level steps you can follow to create a custom logging schema:

1. Create a new logging schema JSON file that includes the fields you want to log. You can include fields such as error message, error code, and any other details you need.
2. Upload the logging schema file to your Azure Storage account.
3. In the Synapse workspace, go to the "Workspace settings" tab and select "Diagnostic settings".
4. Create a new diagnostic setting and select the data types you want to log (e.g., pipeline runs, activity runs).
5. Under "Destination details", select "Log Analytics" and choose the Log Analytics workspace where you want to store the logs.
6. Under "Additional logging settings", select "Custom logging" and provide the path to the logging schema file you uploaded to Azure Storage.

Once you've completed these steps, the logs generated by your Synapse workspace should include the additional fields you specified in the custom logging schema. You can then query the logs in Log Analytics to view error messages, error codes, and other details for your pipeline and activity runs.

Regards,

Umang Middha

umangmiddha2@gmail.com