VM Provision failing: running out of diskspace during creation of yum cache
My VPN create scripts are suddenly having an issue, without any modification to the script. The Azure provisioning process seems to be filling up the yum cache within /var/cache/yum//.
The error that shows up in the file cloud-init.log is a failure to create the yum cache. The /var partition is 100% full when the provisioning completes, almost all of the space taken up within the yum cache subdirectory. Perhaps the yum make cache is trying to reserve space and wants more than the 8Gb in the partition? No idea why this just started happening however.
From cloud-init.log
ProcessExecutionError: Unexpected error while running command.
Command: ['yum', '-t', '-y', 'makecache']
Exit code: 1
Reason: -
Stdout: -
Stderr: -
In the cloud-init-output.log, there is an immediate failure when trying to install my additional packages:
cloud-init v. 19.4 running 'modules:config' at Sun, 02 May 2021 21:29:53 +0000. Up 302.65 seconds.
Loaded plugins: langpacks, product-id, search-disabled-repos
Error: Error writing to file /var/cache/yum/x86_64/7Server/rhui-rhel-server-rhui-rhscl-7-rpms/gen/filelists.xml: [Errno 28] No space left on device
2021-05-02 21:34:22,956 - util.py[WARNING]: Package update failed
This exact same VM provisioning worked fine for a few months, and now suddenly just fails. I have not change the instance type (Standard_E8s_v3), and I've tried a RHEL 9 and RHEL 8 image.
I can run 'yum clean all' and it frees up all the space, however when I try the provisioning command 'yum -y -t makecache', the directory fills up again and the command fails with:
Error: Error writing to file /var/cache/yum/x86_64/7Server/rhui-rhel-server-rhui-rhscl-7-rpms/gen/filelists.xml: [Errno 28] No space left on device
The OS Disk details are:
Thanks,
Phil