How to limit the logs that are forwarding to Log Analytics based on log level
I am using spark-listeners-loganalytics_3.0.1_2.12 and spark-listeners_3.0.1_2.12 in my spark application that is running on databricks. With the default configuration, I am getting too many logs in Azure Log Analytics. I want to limit the logs that are forwarding to Log Analytics.
I tried to configure it as below and added spark-monitoring.sh file as init script in my databricks job cluster.
tee -a "$SPARK_CONF_DIR/spark-env.sh" << EOF export DB_CLUSTER_ID=$DB_CLUSTER_ID export LOG_ANALYTICS_WORKSPACE_ID=xxxx export LOG_ANALYTICS_WORKSPACE_KEY=xxxx export AZ_SUBSCRIPTION_ID=xxxxx export LA_SPARKLOGGINGEVENT_NAME_REGEX="com.demo.sample.*" EOF
Now I am getting logs with logger_name_s matches the regexr "com.demo.sample.*"
Requirement is to forward logs specific to "com.demo.sample.*" package and also ERROR logs from all other packages to Log Analytics. Is there any way I can configure it !!
Hello @vikas katru and welcome to Microsoft Q&A.
After some digging I found some documentation on how to filter in more detail. Let me know if you can see the below link.
I know this isn't a final answer, but I wanted to share what I found in case you are faster with these things than I am.
Thanks @MartinJaffer-MSFT , I already gone through that documentation. I even tried with below config:
log4j.appender.logAnalyticsAppender.Threshold=ERROR export LA_SPARKLOGGINGEVENT_NAME_REGEX="com.demo.sample.*"
It's filtering out based on the package name I have given, but not with the Log level. I need something like logs from specific package + ERROR logs from all packages.
Sign in to comment