Send Synapse pyspark notebook logs to log analytics workspace

Question

I want to send spark application logs to log analytics workspace and I am following the document here: https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/apache-spark-azure-log-analytics#write-custom-application-logs Is there a way to achieve this with managed identity and RBAC roles? I have made the following settings in my spark pool configurations and have ensure necessary role access in Azure Log analytics workspace as well. I am still getting an error that secret is invalid.

I am seeing an error "ERROR LogAnalyticsConfigurationLoader [LogAnalyticsConfigurationLoader-0]: Azure Log Analytics secret is invalid. Please check the configuration." . However, in my spark configuration, I have

spark.synapse.logAnalytics.enabled true

spark.synapse.logAnalytics.useManagedIdentity true

spark.synapse.logAnalytics.workspaceId <workspace id>

Answer 1

Hi Suryanarayanan Swetha,
Azure Synapse Spark does not support Managed Identity–only authentication for sending Spark application logs to Azure Log Analytics. This behavior is by design.

Although the Spark configuration supports

spark.synapse.logAnalytics.useManagedIdentity=true,

The Log Analytics integration still requires a Log Analytics workspace key to authenticate log ingestion. Managed Identity is supported only for retrieving the workspace key from Azure Key Vault. It cannot replace the workspace key itself.

Because the primary and secondary workspace keys are disabled or not visible in your environment, Azure Synapse cannot authenticate with the Log Analytics ingestion endpoint. As a result, Spark throws the error indicating that the Log Analytics agent secret is invalid. This behavior occurs regardless of RBAC role assignments such as Log Analytics Contributor or Monitoring Metrics Publisher.

Microsoft documentation confirms that all supported configurations for sending Synapse Spark logs to Log Analytics require a workspace key. The key can be provided directly, stored in Azure Key Vault, or accessed through a Key Vault linked service. There is no supported configuration that allows Synapse Spark to send logs to Log Analytics using Managed Identity and RBAC alone without a workspace key.

The Log Analytics connector used by Azure Synapse Spark relies on the Log Analytics ingestion API, which authenticates using a workspace ID and workspace key. Enabling Managed Identity only allows Synapse to securely access secrets from Azure Key Vault and does not change the authentication mechanism used by the Log Analytics endpoint. If workspace keys are disabled by organizational security policy, Synapse Spark logging to Log Analytics cannot succeed. This is a product limitation rather than a misconfiguration.

To resolve the issue, one of the supported approaches must be used. If permitted by security policy, Log Analytics workspace keys must be enabled and the Spark pool configured using the workspace ID and primary key. If direct use of keys is restricted, the workspace key must be stored in Azure Key Vault and the Synapse workspace managed identity granted permission to read the secret. Spark can then retrieve the key securely from Key Vault and send logs successfully. Using Managed Identity without a workspace key is not supported at this time.

If workspace keys are disabled at the organization level, Azure Synapse Spark currently has no alternative authentication mechanism to send logs to Azure Log Analytics.
Reference links:
Monitor Apache Spark applications with Azure Log Analytics – Azure Synapse Analytics

Hope this helps. If you have any follow-up questions, please let me know. I would be happy to help.