syslog/messages/kern.log rollover not working on aks nodes

Question

We saw that after few months the aks node had diskpressure which lead to the pods being evicted, on debugging the cause for disk pressure it was observed that the syslog/messages and kern.log together were consuming around 14G of memory which was more than 50% of the free space available on the node

-rw-r-----   1 syslog    adm             1.2G Jan 21 16:22 kern.log.1
-rw-r-----   1 syslog    adm             6.4G Jan 21 16:22 syslog.1
-rw-r-----   1 syslog    adm             6.4G Jan 21 16:22 messages.1

Checking the syslog config in /etc/logrotate.d/rsyslog, I can confirm that the syslog is set to rotate daily with max backups set to 7.

If this was working there should have been 8 files in total i.e, syslog, syslog.1, syslog.2.gz .... syslog.7.gz which is not the case.

Also, I see that the logs are not being written to /var/log/syslog instead are being written to /var/log/syslog.1 which is also the case with /var/log/messages and /var/log/kern.log which might be the reason why the log rotation is not working.

-rw-r-----   1 syslog    adm                0 Jan 21 16:27 syslog
-rw-r-----   1 root      root               0 Jan 21 16:28 kern.log.

As you can see above the size of syslog, kern.log is 0

Here are the details about the node

Linux aks-xxxxx 5.4.0-1026-azure #26~18.04.1-Ubuntu SMP Thu Sep 10 16:19:25 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux


Node Image Version : AKSUbuntu-1804-2021.01.06

Distributor ID: Ubuntu
Description: Ubuntu 18.04.5 LTS
Release: 18.04
Codename: bionic

Answer 1

I'm seeing the same problem and it is causing pods to be evicted. Here is the disk usage for the relevant files in /var/log, note that both "messages" and "syslog" are 0 bytes, but there are files called "messages.1" and "syslog.1" which are both occupying 52GB:

12K localmessages
0     messages
52G messages.1
1.9M    messages.2.gz
0     syslog
52G syslog.1
1.1M    syslog.2.gz
876K    syslog.3.gz
4.0K    sysstat

Here are the contents of "/etc/logrotate.d/rsyslog":

/var/log/syslog
{
    rotate 7
    daily
    missingok
    notifempty
    delaycompress
    compress
    postrotate
        /usr/lib/rsyslog/rsyslog-rotate
    endscript
}

/var/log/mail.info
/var/log/mail.warn
/var/log/mail.err
/var/log/mail.log
/var/log/daemon.log
/var/log/kern.log
/var/log/auth.log
/var/log/user.log
/var/log/lpr.log
/var/log/cron.log
/var/log/debug
/var/log/messages
{
    rotate 4
    weekly
    missingok
    notifempty
    compress
    delaycompress
    sharedscripts
    postrotate
        /usr/lib/rsyslog/rsyslog-rotate
    endscript
}

Here is the node info:

Static hostname: aks-parallel-xxxxx-vmss000000
         Icon name: computer-vm
           Chassis: vm
        Machine ID: 3e2b3448a01e40ecaf65fc456f44a25d
           Boot ID: 5a9449cc806d4cb7a29018e26af124b7
    Virtualization: microsoft
  Operating System: Ubuntu 18.04.5 LTS
            Kernel: Linux 5.4.0-1036-azure
      Architecture: x86-64