How to limit the logs that are forwarding to Log Analytics based on log level

vikas katru 6

I am using spark-listeners-loganalytics_3.0.1_2.12 and spark-listeners_3.0.1_2.12 in my spark application that is running on databricks. With the default configuration, I am getting too many logs in Azure Log Analytics. I want to limit the logs that are forwarding to Log Analytics.

I tried to configure it as below and added spark-monitoring.sh file as init script in my databricks job cluster.

tee -a "$SPARK_CONF_DIR/spark-env.sh" << EOF
export DB_CLUSTER_ID=$DB_CLUSTER_ID
export LOG_ANALYTICS_WORKSPACE_ID=xxxx
export LOG_ANALYTICS_WORKSPACE_KEY=xxxx
export AZ_SUBSCRIPTION_ID=xxxxx
export LA_SPARKLOGGINGEVENT_NAME_REGEX="com.demo.sample.*"
EOF

Now I am getting logs with logger_name_s matches the regexr "com.demo.sample.*"

Requirement is to forward logs specific to "com.demo.sample.*" package and also ERROR logs from all other packages to Log Analytics. Is there any way I can configure it !!

MartinJaffer-MSFT 26,211 Reputation points

2021-03-15T18:02:31.067+00:00

Hello @vikas katru and welcome to Microsoft Q&A.

After some digging I found some documentation on how to filter in more detail. Let me know if you can see the below link.

https://github.com/mspnp/spark-monitoring/blob/master/docs/filtering.md

I know this isn't a final answer, but I wanted to share what I found in case you are faster with these things than I am.
vikas katru 6 Reputation points

2021-03-16T06:00:16.11+00:00
Thanks @MartinJaffer-MSFT , I already gone through that documentation. I even tried with below config:

log4j.appender.logAnalyticsAppender.Threshold=ERROR export LA_SPARKLOGGINGEVENT_NAME_REGEX="com.demo.sample.*"

It's filtering out based on the package name I have given, but not with the Log level. I need something like logs from specific package + ERROR logs from all packages.

Share via

How to limit the logs that are forwarding to Log Analytics based on log level

Your answer