Not able to limit the size of STDERR and STDOUT logs for Azure HDInsight deployed Spark Application!!!

Goel, Akanksha 61 Reputation points
2021-04-20T09:16:53.993+00:00

I have tried editing the log4j properties file in

https://CLUSTERNAME.azurehdinsight.net/#/main/services/SPARK2/configs,

as mentioned in the following link:

https://learn.microsoft.com/en-us/azure/hdinsight/spark/troubleshoot-debug-wasb

89494-image.png
I have done the following additions as shown in screenshot:

log4j.appender.audit.triggeringPolicy=org.apache.log4j.rolling.SizeBasedTriggeringPolicy
log4j.appender.audit.triggeringPolicy.maxFileSize=50MB

![89416-image.png]2
But, the changes are not working, **what can I do to limit the size of logs as you can see the stderr log size is increasing **

Azure HDInsight
Azure HDInsight
An Azure managed cluster service for open-source analytics.
164 questions
{count} votes