Custom logging in Azure Synapse Datapipeline to to log analytics Workspace - Help Requested
Requirement:- Want to add custom logging in azure synapse pyspark job using python logging module and spark config
Getting error while implementing the same in the pyspark job
ModuleNotFoundError: No module named 'pyspark.SparkConf
pyspark script for sparkconf I am using
import logging
from pyspark.SparkConf import
SparkConf conf = SparkConf() \ .set("spark.spark.driver.extraJavaOptions", "-Dlog4j.configuration=appenders=synapseAppender,consoleAppender\n" "-Dlog4j.appender.synapseAppender=com.microsoft.azure.synapse.analytics.spark.AzureMonitorAppender\n" "-Dlog4j.appender.synapseAppender.logAnalyticsWorkspaceId=<workspace_id>\n" "-Dlog4j.appender.synapseAppender.logAnalyticsWorkspaceKey=<workspace_key>\n" "-Dlog4j.appender.consoleAppender=org.apache.log4j.ConsoleAppender\n" "-Dlog4j.appender.consoleAppender.layout=org.apache.log4j.PatternLayout\n" "-Dlog4j.appender.consoleAppender.layout.ConversionPattern=%d{ISO8601} %-5p [%t] %c{1} - %m%n")
Create a SparkSession with the configured SparkConf spark = SparkSession.builder \ .config(conf=conf) \ .getOrCreate() # Your Spark Job Logic Here # Use Python logging to write messages with different levelslogging.info("Spark Job Started")
As the pyspark.SparkConf is not available in synapse studio, I am trying to add the following spark configuration in the spark pool to enable custom logging to log analytics workspace. This is not working
"-Dlog4j.appender.synapseAppender.logAnalyticsWorkspaceKey=<workspace_key>\n" "-Dlog4j.appender.consoleAppender=org.apache.log4j.ConsoleAppender\n" "-
Could you please help implement custom logging in synapse data pipeline jobs to log analytics workspace? Your response is greatly appreciated.