Collect synapse pipeline, trigger, run status into Azure log analytics and ingest data into Azure database.

Question

Hi,
I am trying to make a monitoring system that collect information from Azure synapse into log analytics and then try to ingest data into Azure database.
I created a log analytics workspace and turn on diagnostic in azure synapse and connect to log analytics workspace but can not find and example on how to run the query
to get the job information pipeline info in log analytics and ingest info to database.

How can I do that? Do you have any example? And how can I save this information into azure sql

Answer 1

Hello @NC ,

Thanks for the question and using MS Q&A platform.

Azure Log Analytics Workspace:

Once the Diagnostics settings configured on the Synapse workspace, you can query directly on the job information as shown below:

Note: In Log Analytics workspace, you can query the job information pipeline info and export to local file and then you can bulk insert into the Azure SQL Database.

For more details, refer to Use Azure Monitor with your Azure Synapse Analytics workspace and

Azure Data Explorer:

For seamless integration, you can create a Azure Data Explorer.

The Azure Data Explorer supports cross-service queries between Azure Data Explorer, Application Insights (AI), and Log Analytics (LA). You can query your Log Analytics or Application Insights workspace using Azure Data Explorer query tools and in a cross-service query. The article shows you how to create a cross-service query and add the Log Analytics or Application Insights workspace to the Azure Data Explorer Web UI.

Once the Log Analytics workspace connected to the Azure Data Explorer, you can directly export data to SQL.

Export data to SQL allows you to run a query and have its results sent to a table in a SQL database, such as a SQL database hosted by the Azure SQL Database service.

For more details, refer to Query data in Azure Monitor using Azure Data Explorer and Export data to SQL.

Hope this will help. Please let us know if any further queries.

------------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators