Diagnostic Settings not functioning as expected, not all CL tables registering events

Question

Diagnostic Settings not functioning as expected, not all CL tables registering events

Mike Rees 0 Microsoft Employee

I wanted to tag this with Azure Log Analytics but couldn't find that tag. Here goes
We have two spark pools that have the same Spark Version (3.4), same Spark Configuration (which is used to write events to three tables (SparkListenerEvent_CL, SparkLoggingEvent_CL and SparkMetrics_CL).

Pool A has no recent events registered in SparkListenerEvent_CL, but does in the other two tables. The comparable pool, Pool B, has events in all three tables.

I have been trying to troubleshoot this for a couple of days over the past two weeks or so. I have cleared the spark configs, and re assigned them (on Pool A). I have deleted the Diagnostic Settings on this pool (Pool A). At no time did these changes register any records in the SparkListenerEvent_CL table, but events did register in the other two tables.

I run the following query after running a test notebook (and waiting for events to propagate):

SparkListenerEvent_CL
| where workspaceName_s == '<workspacename>' and TimeGenerated >= ago(8h)
| summarize count(), min(TimeGenerated), max(TimeGenerated) by clusterName_s
| order by clusterName_s 
;
SparkLoggingEvent_CL
| where workspaceName_s == '<workspacename>' 
and TimeGenerated >= ago(8h)
| summarize count(), min(TimeGenerated), max(TimeGenerated) by clusterName_s
| order by clusterName_s 
;
SparkMetrics_CL
| where workspaceName_s == '<workspacename>' 
and TimeGenerated >= ago(8h)
| summarize count(), min(TimeGenerated), max(TimeGenerated) by clusterName_s
| order by clusterName_s

In the first query results (SparkListenerEvent_CL), one record is produced for Pool B.

In the other two query results, there are 18 records (for 18 different spark pools, including both Pool A and Pool B).

I've done this several times now. Again, clearing/reassigning Spark Configuration, and deleting and adding back the appropriate Diagnostic Settings record. In doing so, I did not check the all boxes, and instead checked the only Category available ("Big Data Pool Applications Ended").

I cannot see if there are ingestion errors in this log analytics workspace due to query error.
I'm using :

 .show ingestion failures

And the error message is a little interesting ""error":{"code":"MgmtCslFailedParsing","message":"ApiContractViolationError: management command parsing failed with error: SyntaxError: Expected "schema" but "f" found., Location: line 1, column 25"......"
Each time the column number changes. For example, I ran it again, and
{"error":{"code":"MgmtCslFailedParsing","message":"ApiContractViolationError: management command parsing failed with error: SyntaxError: Expected "capacity", "cluster", "database", "databases", "entity", "function", "functions", "schema", "tables", or "version" but "i" found., Location: line 1, column 7"

Both of these pools (one working (Pool B) one not (Pool A) have the same permissions in the same synapse workspace. Or at least they seem to. Both pools have the same Diagnostic Settings, writing to the same Log Analytics workspace.

I don't see how permissions to write to Log Analytics workspace could differ. We have pipelines using each of these pools (though the one is more frequently used). Records should register at least once a day for Pool A, with many records per day for Pool B.
Fact is, there are several spark pools that are not registering events in SparkListenerEvent_CL, and are in the other two tables.

Siva shunmugam Nadessin 5,370 Reputation points Microsoft External Staff Moderator

2026-01-27T00:54:53.35+00:00
Hello Mike Rees,

Thank you for reaching out to the Microsoft Q&A forum.

It sounds like you're dealing with a frustrating issue where SparkListenerEvent_CL isn't receiving records for one of your pools (Pool A), even though the other tables (SparkLoggingEvent_CL and SparkMetrics_CL) are receiving data just fine. You’ve already done quite a bit of troubleshooting, but let's go through some potential solutions and areas to investigate to try to get to the root cause.

Key Points:

Pool A: No events in SparkListenerEvent_CL, but events in SparkLoggingEvent_CL and SparkMetrics_CL.

Pool B: Events in all three tables (SparkListenerEvent_CL, SparkLoggingEvent_CL, SparkMetrics_CL).

Recent Attempts: Cleared and reassigned Spark configurations, deleted and re-added diagnostic settings (with only "Big Data Pool Applications Ended" checked).

Troubleshooting Steps

Verify Diagnostic Settings Configuration:

Ensure that all relevant diagnostic settings are properly configured for Pool A. It seems you only checked the “Big Data Pool Applications Ended” category. Double-check that all necessary categories are selected in the diagnostic settings for both Spark event logging and metrics. For example:

SparkJobs (for job-level events),

SparkStages (for stage-level events),

SparkExecutor (for executor-level events),

SparkDriver (for driver-level events),

Big Data Pool Applications Started/Ended (for application-level events).

If you're missing one or more categories, events may not be logged to SparkListenerEvent_CL, even though they're being logged in the other tables.

Check for Event Filtering or Exclusions:

Verify if there’s any filtering or exclusion mechanism that might be causing certain events to be dropped for Pool A. Ensure there are no filters set in the diagnostic settings, Spark configurations, or Log Analytics workspace that could be inadvertently blocking specific event types.

Check Permissions:

Even though you mentioned both pools have the same permissions, it’s worth double-checking that there are no specific role-based access control (RBAC) or workspace permissions that might be limiting Pool A’s ability to send data to Log Analytics. For example, ensure that:

The Managed Identity of the Spark Pool (or the service principal) has proper permissions to send logs and metrics to Log Analytics.

The Diagnostic settings have the necessary write permissions to Log Analytics (ensure the workspace is correctly specified in the diagnostic settings).

Examine Ingestion Failures:

The ingestion failures you’re seeing in the error logs indicate a potential issue with the format or schema of the data being sent to Log Analytics. The error message you provided:

"ApiContractViolationError: management command parsing failed with error: SyntaxError: Expected "capacity", "cluster", "database", "databases", "entity", "function", "functions", "schema", "tables", or "version" but "i" found., Location: line 1, column 7"

suggests that the logs from Pool A may be failing to match the expected schema.

Action:

Investigate the logs and format of the data being sent to Log Analytics for Pool A.

Use the .show ingestion failures command in Log Analytics to check if there are any ingestion issues specifically related to SparkListenerEvent_CL for Pool A.

.show ingestion failures | where tableName == "SparkListenerEvent_CL" | summarize count() by message, errorCode

Investigate Spark Job Configuration:

Check if there are any configuration issues on Pool A's Spark jobs that could prevent event logging, particularly for listener events.

Are the jobs in Pool A configured to emit SparkListenerEvent? Sometimes, custom job configurations can inadvertently disable event emission or only emit certain types of events.

Compare Configurations of Pool A and Pool B:

Even though both pools have the same Spark version and configuration, there may still be some subtle differences. Compare all configurations between Pool A and Pool B:

Check if the cluster size, memory, or other runtime configurations might differ.

Look at the Spark history server logs for any errors or warnings related to event logging for Pool A.

Try Recreating Diagnostic Settings:

You’ve deleted and re-added the diagnostic settings, but it might be worthwhile to recreate them from scratch. Try deleting the current configuration, then recreate the diagnostic settings with all categories enabled for both pools and ensure they are pointing to the same Log Analytics workspace.

Check for Updates in Spark Pools or Diagnostic Tools:

Occasionally, there are updates to the diagnostic settings or Spark pools themselves that might introduce new bugs or changes. Ensure that your Spark Pools are up to date with the latest patches or updates from Azure, and check the Azure documentation for any changes in how logs are handled.

Try a Minimal Configuration:

As a test, try reducing the Spark job’s complexity or test with a minimal notebook that writes very basic logs to SparkListenerEvent_CL to check if the problem persists. This could help determine if the issue is related to the job’s complexity or the event system itself.

Conclusion

It seems that the issue might be related to either a misconfiguration in the diagnostic settings, potential permission issues, or ingestion failures due to formatting issues. Ensuring that all the necessary diagnostic categories are enabled, verifying permissions, and checking for ingestion errors should help resolve the issue

Let me know if you need further guidance!
Bharath Y P 4,310 Reputation points Microsoft External Staff Moderator

2026-01-27T13:38:44.72+00:00

Hello Mike Rees, We just wanted to kindly follow up to check if you had a chance to review above response. hope the information shared was helpful. Please feel free to reach out if you have any additional questions or need further clarification. Thanks
Siva shunmugam Nadessin 5,370 Reputation points Microsoft External Staff Moderator

2026-01-28T01:08:23.7166667+00:00

Hello Mike Rees,

For your query There is only ONE category, your response indicates there are multiple categories that should be available

It looks like you've uploaded a screenshot of your diagnostic settings in Azure. From your description, you're concerned about the configuration for Pool A and ensuring that all necessary logging categories for Spark jobs, stages, executors, and drivers are enabled.

In the screenshot you provided, only the “Big Data Pool Applications Ended” category is checked, while other key categories such as SparkJobs, SparkStages, SparkExecutor, SparkDriver, and Big Data Pool Applications Started/Ended may not be visible or selected. These categories are crucial for logging at different levels of Spark events.

If you do not see them in the interface, there may be an issue with the setup or a limitation in the available options. Here are a few steps you can take to resolve this:

Check All Available Categories: In the diagnostic settings, verify if other categories are listed under "Categories." They should be accessible, but if not, try refreshing or reviewing your workspace settings.

Consult Documentation: Ensure you're following the correct guidelines for enabling comprehensive logging, especially for Spark events. Check the latest Azure documentation for diagnostic settings.

Reference Document:

Diagnostic settings in Azure Monitor - https://learn.microsoft.com/en-us/azure/azure-monitor/platform/diagnostic-settings?tabs=portal

Apache Spark diagnostic logs in Azure Synapse Analytics - https://learn.microsoft.com/en-us/troubleshoot/azure/synapse-analytics/spark/perf/synapse-analytics-apache-spark-pools-diagnostic-logs

Monitor Apache Spark applications with Azure Log Analytics - https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/apache-spark-azure-log-analytics

Supported logs for bigDataPools - https://learn.microsoft.com/en-us/azure/azure-monitor/reference/supported-logs/microsoft-synapse-workspaces-bigdatapools-logs

Azure Portal Navigation: Sometimes, additional categories may only show up based on specific configurations or services. If you're still unable to find those categories, double-check if you have the necessary permissions or if there's a service limitation.

Check for Updates: Sometimes Azure updates or changes its logging options. Ensure that your Azure environment is up to date.

Let me know if you need help with anything else related to this setup!
Siva shunmugam Nadessin 5,370 Reputation points Microsoft External Staff Moderator

2026-02-02T12:50:38.4366667+00:00
Hello Mike Rees,

We understand that your Azure Synapse Spark pools with Diagnostic Settings enabled send events to three primary tables in Log Analytics:

SparkLoggingEvent_CL this for continuous log4j messages (always populated if jobs/notebooks run)

SparkMetrics_CL this is for continuous driver/executor metrics (always populated if jobs/notebooks run)

SparkListenerEvent_CL this is the table for lifecycle events: ApplicationStarted, ApplicationEnded, JobStarted/Ended (batch jobs)

But in you case You observed that Pools like Pool B have events in all three tables but Pools like Pool A have events in SparkLoggingEvent_CL and SparkMetrics_CL , but no events in SparkListenerEvent_CL

The root cause that you are not seeing the events in SparkListenerEvent_CL might be because the SparkListenerEvent_CL is not a continuous telemetry table. it only logs Spark application lifecycle events. The table remains empty might be because there is of few below resion

No new ApplicationEnded event due to Long-running interactive session / warm pool driver.

Reused Spark application for multiple notebooks/jobs cause No new lifecycle events

When batch jobs run, or when a Spark pool restarts or wakes up from being paused, new application start and end events are created, and those events show up in the SparkListenerEvent table.

So the absence of records in SparkListenerEvent_CL is expected behavior in long-running or warm pools.

Overview of Log Analytics in Azure Monitor - Azure Monitor | Microsoft Learn

Meanwhile, the other two tables (SparkLoggingEvent_CL and SparkMetrics_CL) are continuous, which explains why they still receive events.

To generate or verify SparkListenerEvent_CL events, you can try with the below possible scenarios:

Restart the Spark pool or submit a new batch Spark job

Ensures a new Spark application is created and terminated

Will trigger ApplicationStarted and ApplicationEnded events

Enable auto-pause for idle pools: Causes idle Spark applications to stop > triggers ApplicationEnded events automatically

Use SparkLoggingEvent_CL and SparkMetrics_CL for monitoring day-to-day activity

These tables are continuous and reliable for job-level insights

Do not rely on SparkListenerEvent_CL for continuous pipeline monitoring.

deleting/re-adding diagnostic settings or clearing configs will not replay past lifecycle events; only new ones are captured.We understand that your Azure Synapse Spark pools with Diagnostic Settings enabled send events to three primary tables in Log Analytics:

SparkLoggingEvent_CL this for continuous log4j messages (always populated if jobs/notebooks run)

SparkMetrics_CL this is for continuous driver/executor metrics (always populated if jobs/notebooks run)

SparkListenerEvent_CL this is the table for lifecycle events: ApplicationStarted, ApplicationEnded, JobStarted/Ended (batch jobs)

But in you case You observed that Pools like Pool B have events in all three tables but Pools like Pool A have events in SparkLoggingEvent_CL and SparkMetrics_CL , but no events in SparkListenerEvent_CL The root cause that you are not seeing the events in SparkListenerEvent_CL might be because the SparkListenerEvent_CL is not a continuous telemetry table. it only logs Spark application lifecycle events. The table remains empty might be because there is of few below resion

No new ApplicationEnded event due to Long-running interactive session / warm pool driver.

Reused Spark application for multiple notebooks/jobs cause No new lifecycle events

When batch jobs run, or when a Spark pool restarts or wakes up from being paused, new application start and end events are created, and those events show up in the SparkListenerEvent table. So the absence of records in SparkListenerEvent_CL is expected behavior in long-running or warm pools. Overview of Log Analytics in Azure Monitor - Azure Monitor | Microsoft Learn Meanwhile, the other two tables (SparkLoggingEvent_CL and SparkMetrics_CL) are continuous, which explains why they still receive events. To generate or verify SparkListenerEvent_CL events, you can try with the below possible scenarios:

Restart the Spark pool or submit a new batch Spark job

Ensures a new Spark application is created and terminated

Will trigger ApplicationStarted and ApplicationEnded events

Enable auto-pause for idle pools: Causes idle Spark applications to stop > triggers ApplicationEnded events automatically

Use SparkLoggingEvent_CL and SparkMetrics_CL for monitoring day-to-day activity

These tables are continuous and reliable for job-level insights

Do not rely on SparkListenerEvent_CL for continuous pipeline monitoring.

deleting/re-adding diagnostic settings or clearing configs will not replay past lifecycle events; only new ones are captured.

1 answer

Your answer

Bharath Y P 4,310 Reputation points Microsoft External Staff Moderator

2026-01-27T13:38:44.72+00:00

Hello Mike Rees, We just wanted to kindly follow up to check if you had a chance to review above response. hope the information shared was helpful. Please feel free to reach out if you have any additional questions or need further clarification. Thanks

Answer 1

It seems you are experiencing issues with the SparkListenerEvent_CL table not registering events for Pool A, despite having the same configurations as Pool B, which is functioning correctly. Here are some troubleshooting steps and considerations based on the context provided:

Configuration Verification: Ensure that the Spark configuration settings for Pool A are indeed identical to those of Pool B. Sometimes, subtle differences can lead to discrepancies in event logging.
Diagnostic Settings: Since you mentioned deleting and re-adding the Diagnostic Settings, ensure that the settings are correctly configured to capture all necessary event categories. You may want to double-check that the correct categories are selected, especially if you are only checking "Big Data Pool Applications Ended."
Event Filtering: Review the filters applied to the diagnostic emitters. If there are specific event names or logger names being filtered out, it could prevent events from being logged in SparkListenerEvent_CL.
Permissions: Although you indicated that both pools have the same permissions, it may be worth verifying the permissions on the Log Analytics workspace itself to ensure there are no discrepancies that could affect data ingestion.
Ingestion Errors: The errors you are encountering when using .show ingestion failures suggest there may be issues with the queries being run. Ensure that the syntax of your queries is correct and that you are using valid commands. The error messages indicate parsing issues, which may be due to incorrect command structure or unsupported syntax.
Logs and Metrics: Since you are receiving logs in the other two tables, it may be helpful to compare the logs from both pools to identify any differences in behavior or errors that could provide clues about why events are not being captured in SparkListenerEvent_CL.
Support: If the issue persists after trying these steps, consider reaching out to Microsoft Support for more in-depth troubleshooting, as they can access detailed diagnostic logs that may not be visible to you.

By following these steps, you should be able to narrow down the cause of the issue with the SparkListenerEvent_CL table not registering events for Pool A.

References:

Share via

Diagnostic Settings not functioning as expected, not all CL tables registering events

1 answer

Your answer