Custom logging in Azure Synapse Datapipeline to to log analytics Workspace - Help Requested

Johny Yagappan 20

Requirement:- Want to add custom logging in azure synapse pyspark job using python logging module and spark config

Getting error while implementing the same in the pyspark job

ModuleNotFoundError: No module named 'pyspark.SparkConf

pyspark script for sparkconf I am using

import logging

from pyspark.SparkConf import

SparkConf conf = SparkConf() \ .set("spark.spark.driver.extraJavaOptions", "-Dlog4j.configuration=appenders=synapseAppender,consoleAppender\n" "-Dlog4j.appender.synapseAppender=com.microsoft.azure.synapse.analytics.spark.AzureMonitorAppender\n" "-Dlog4j.appender.synapseAppender.logAnalyticsWorkspaceId=<workspace_id>\n" "-Dlog4j.appender.synapseAppender.logAnalyticsWorkspaceKey=<workspace_key>\n" "-Dlog4j.appender.consoleAppender=org.apache.log4j.ConsoleAppender\n" "-Dlog4j.appender.consoleAppender.layout=org.apache.log4j.PatternLayout\n" "-Dlog4j.appender.consoleAppender.layout.ConversionPattern=%d{ISO8601} %-5p [%t] %c{1} - %m%n")

Create a SparkSession with the configured SparkConf spark = SparkSession.builder \ .config(conf=conf) \ .getOrCreate() # Your Spark Job Logic Here # Use Python logging to write messages with different levelslogging.info("Spark Job Started")

As the pyspark.SparkConf is not available in synapse studio, I am trying to add the following spark configuration in the spark pool to enable custom logging to log analytics workspace. This is not working

"-Dlog4j.appender.synapseAppender.logAnalyticsWorkspaceKey=<workspace_key>\n" "-Dlog4j.appender.consoleAppender=org.apache.log4j.ConsoleAppender\n" "-

Could you please help implement custom logging in synapse data pipeline jobs to log analytics workspace? Your response is greatly appreciated.

Johny Yagappan 20 Reputation points

2024-06-26T23:21:06.53+00:00

Additional info

I am trying to use logAnalyticsWorkspaceId and logAnalyticsWorkspaceKey configuration parameters in the spark spool configuration for the spark.spark.driver.extraJavaOptions parameter in azure synapse studio.
PRADEEPCHEEKATLA-MSFT 83,886 Reputation points Microsoft Employee

2024-06-28T02:40:24.4766667+00:00

@Johny Yagappan - Thanks for the question and using MS Q&A platform.

It seems like you are trying to add custom logging in Azure Synapse Pyspark job using Python logging module and Spark config. However, you are getting an error while implementing the same in the Pyspark job.

The error message "ModuleNotFoundError: No module named 'pyspark.SparkConf'" indicates that the SparkConf module is not available in your environment. Make sure you are using Pyspark option on the Notebooks or use magic commands as shown her: Create, develop, and maintain Synapse notebooks in Azure Synapse Analytics

As you mentioned, the Pyspark.SparkConf is not available in Synapse Studio. Instead, you can try adding the following Spark configuration in the Spark pool to enable custom logging to log analytics workspace:

"-Dlog4j.appender.synapseAppender.logAnalyticsWorkspaceKey=<workspace_key>\n" "-Dlog4j.appender.consoleAppender=org.apache.log4j.ConsoleAppender\n"

Please note that you need to replace <workspace_key> with the actual key of your Log Analytics workspace.

If you are still facing issues, please provide more information about the error you are facing and the steps you have followed so far.
Looking forward for your response.

Share via

Custom logging in Azure Synapse Datapipeline to to log analytics Workspace - Help Requested

Create a SparkSession with the configured SparkConf spark = SparkSession.builder \ .config(conf=conf) \ .getOrCreate() # Your Spark Job Logic Here # Use Python logging to write messages with different levelslogging.info("Spark Job Started")