SparkLoggingEvent_CL is not being gained in log analytics

Question

SparkLoggingEvent_CL is not being gained in log analytics

I have configured log analytics and spark according to https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/apache-spark-azure-log-analytics
The metrics and listener have arrived to log analytics but logging event has not ,I have setup the pyspark logging as per the article and can view the logs through synapse monitor spark pool logs

Any help would be greatly appreciated
Thank you in advance

AnnuKumari-MSFT 20,996 Reputation points Microsoft Employee

2023-03-17T11:32:56.66+00:00

@Arudchayan Pirabaharan ,

Welcome to Microsoft Q&A platform and thanks for posting your question here.

As per my understanding you are able to see logging event while configuring log analytics through syanapse.

Kindly ensure that the logging level set in log4j.properties is set to an appropriate level that includes the logging events you want to see.

For example, if you have set the logging level to ERROR, you will only see logging events that are at the ERROR level or higher.

Please let us know how it goes. Thanks
Charls K V 5 Reputation points Microsoft Employee

2023-03-17T12:37:01.8866667+00:00

@Arudchayan Pirabaharan

Can you please confirm the version of the Spark environment? Also, mention if it is a DEP-enabled workspace.

Thanks,
Charls
Arudchayan Pirabaharan 10 Reputation points

2023-03-19T16:59:52.1033333+00:00

Thank you for responding AnnuKumari,The log4j properties is at default,as we need to have all the logs at INFO and higher.
Arudchayan Pirabaharan 10 Reputation points

2023-03-19T17:08:05.3233333+00:00
Thank you for responding @Charls K V ,The spark version is {Spark version: 3.3.1.5.2-83515052} and the workspace is not DEP enabled as managed virtual network is set to NO.

For further clarification

%%pyspark logger = sc._jvm.org.apache.log4j.LogManager.getLogger("com.contoso.PythonLoggerExample") logger.info("info message") logger.warn("warn message") logger.error("error message")

This is essentially the sample code we are using. The log analytics captures the metrics and listener but not logging event.

Any further assistance would be greatly appreciated.

AnnuKumari-MSFT 20,996 Reputation points Microsoft Employee

2023-03-17T11:32:56.66+00:00

@Arudchayan Pirabaharan ,

Welcome to Microsoft Q&A platform and thanks for posting your question here.

As per my understanding you are able to see logging event while configuring log analytics through syanapse.

Kindly ensure that the logging level set in log4j.properties is set to an appropriate level that includes the logging events you want to see.

For example, if you have set the logging level to ERROR, you will only see logging events that are at the ERROR level or higher.

Please let us know how it goes. Thanks
Charls K V 5 Reputation points Microsoft Employee

2023-03-17T12:37:01.8866667+00:00

@Arudchayan Pirabaharan

Can you please confirm the version of the Spark environment? Also, mention if it is a DEP-enabled workspace.

Thanks,
Charls
Arudchayan Pirabaharan 10 Reputation points

2023-03-19T16:59:52.1033333+00:00

Thank you for responding AnnuKumari,The log4j properties is at default,as we need to have all the logs at INFO and higher.
Arudchayan Pirabaharan 10 Reputation points

2023-03-19T17:08:05.3233333+00:00

Thank you for responding @Charls K V ,The spark version is {Spark version: 3.3.1.5.2-83515052} and the workspace is not DEP enabled as managed virtual network is set to NO.

For further clarification

%%pyspark logger = sc._jvm.org.apache.log4j.LogManager.getLogger("com.contoso.PythonLoggerExample") logger.info("info message") logger.warn("warn message") logger.error("error message")

This is essentially the sample code we are using. The log analytics captures the metrics and listener but not logging event.

Any further assistance would be greatly appreciated.

SparkLoggingEvent_CL is not being gained in log analytics

Activity