SparkLoggingEvent_CL is not being gained in log analytics

Arudchayan Pirabaharan 20

I have configured log analytics and spark according to https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/apache-spark-azure-log-analytics
The metrics and listener have arrived to log analytics but logging event has not ,I have setup the pyspark logging as per the article and can view the logs through synapse monitor spark pool logs

Any help would be greatly appreciated
Thank you in advance

Update:

The problem seemed to be in the key vault secret setup as directly providing the log analytics key to conf works.
The key Vault has been linked though linked service and synapse has been given access policy access in key Vault.

Is there anything else I have overlooked @AnnuKumari-MSFT @C

Any pointers would be greatly appreciated

AnnuKumari-MSFT 30,676 Reputation points Microsoft Employee

2023-03-17T11:32:56.66+00:00

@Arudchayan Pirabaharan ,

Welcome to Microsoft Q&A platform and thanks for posting your question here.

As per my understanding you are able to see logging event while configuring log analytics through syanapse.

Kindly ensure that the logging level set in log4j.properties is set to an appropriate level that includes the logging events you want to see.

For example, if you have set the logging level to ERROR, you will only see logging events that are at the ERROR level or higher.

Please let us know how it goes. Thanks
Charls Koruthu Varghese 5 Reputation points Microsoft Employee

2023-03-17T12:37:01.8866667+00:00

@Arudchayan Pirabaharan

Can you please confirm the version of the Spark environment? Also, mention if it is a DEP-enabled workspace.

Thanks,
Charls
Arudchayan Pirabaharan 20 Reputation points

2023-03-19T16:59:52.1033333+00:00

Thank you for responding AnnuKumari,The log4j properties is at default,as we need to have all the logs at INFO and higher.
Arudchayan Pirabaharan 20 Reputation points

2023-03-19T17:08:05.3233333+00:00
Thank you for responding @Charls Koruthu Varghese ,The spark version is {Spark version: 3.3.1.5.2-83515052} and the workspace is not DEP enabled as managed virtual network is set to NO.

For further clarification

%%pyspark logger = sc._jvm.org.apache.log4j.LogManager.getLogger("com.contoso.PythonLoggerExample") logger.info("info message") logger.warn("warn message") logger.error("error message")

This is essentially the sample code we are using. The log analytics captures the metrics and listener but not logging event.

Any further assistance would be greatly appreciated.
Narayanan Jayaraman 0 Reputation points

2023-07-27T08:41:14.4933333+00:00

I am also seeing the similar problem, tried both the options of using linked service and explicitly specifying the secret. both of them don't work
Phan, Vinnie 1 Reputation point

2023-11-27T20:16:51.7333333+00:00

It looks like the schema of "SparkLoggingEvent_CL" has changed. It doesn't have Message column where log4j can send message to.

Also, the Spark Config UI has been changed, no longer provide "upload" config file as what I can see

Anyone has any idea when new changes will be reflected to microsoft document (ie: https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/apache-spark-azure-log-analytics) ?
James Cisneros 0 Reputation points

2023-12-13T19:16:58.8933333+00:00

@Arudchayan Pirabaharan did you solve this? I am seeing the same thing.