vCenter Appliance root Partition 100% full. VMware, Inc. is a publicly traded software company listed on the NYSE under stock ticker VMW. Dell Technologies is a majority share holder. VMware provides cloud computing and virtualization software and services. It was one of the first commercially successful companies to virtualize the x86 architecture.
vCenter Appliance root Partition 100% full
What is vCenter Server Appliance – The vCenter Server Appliance is a preconfigured Linux virtual machine, which is optimized for running VMware vCenter Server® and the associated services on Linux.
100% capacity used for /dev/sda3.
Size of audit.log file is very large and /var/log/audit folder consumes majority of the space.
Saved logs from log rotate policy reference a date that is not in line with the policy.
Unable to connect to the vCenter Server as services are not started.
Running /etc/cron.daily/logrotate manually rotates logs as expected.
Accessing vSphere Web Client might fail with error: 503 service unavailable
To resolve this issue, truncate the audit.log file and verify the cron job is working correctly.
Step 1: Truncate audit.log
- Log in to the vCenter Server Appliance through SSH.
- Run this command to enable access the Bash shell:
$ shell.set --enabled true $ shell
Navigate to the /var/log/audit folder
$ cd /var/log/audit
Run below command to verify the issue is with the audit.log file being too large (a few GBs):
$ ls -lh
$ ls -lh total 4.1M drwx------ 2 root root 4.0K Nov 17 20:36 audispd -rw------- 1 root root 4.1G Nov 17 23:08 audit.log
Truncate (clean the content without deleting the file) the audit.log file with this command:
$ truncate -s 0 audit.log
Step 2: Verify that the cron job to rotate the audit.log is running
Run this command to see when the cron job was last ran successfully:
$ ls -l /var/spool/cron/lastrun/ total 0 -rw------- 1 root root 0 Apr 27 2017 cron.daily -rw------- 1 root root 0 Apr 27 2017 cron.hourly -rw------- 1 root root 0 Apr 26 2017 cron.weekly
Run below command to check for credential failures.
$ grep "Authentication token is no longer valid; new one required" /var/log/messages.0.log | head
grep "Authentication token is no longer valid; new one required" /var/log/messages.0.log | head 2018-11-07T00:20:01.617180+00:00 vcenter /usr/sbin/cron: Authentication token is no longer valid; new one required 2018-11-07T00:20:01.617183+00:00 vcenter /usr/sbin/cron: Authentication token is no longer valid; new one required
Run below command to check if the root password has expired:
$ chage -l root
Minimum: 0 Maximum: 365 Warning: 7 Inactive: -1 Last Change: Feb 03, 2018 Password Expires: Feb 03, 2019 Password Inactive: Never Account Expires: Never
If password expired please change root password
Restart all vCenter Server services.
$ service-control --stop --all $ service-control --start --all
Disable root password expiration
$ chage -m 0 -M 99999 -I -1 -E -1 root
More information @ VMware KB 2149278