vCenter Appliance root Partition 100% full

vCenter Appliance root Partition 100% full. VMware, Inc. is a publicly traded software company listed on the NYSE under stock ticker VMW. Dell Technologies is a majority share holder. VMware provides cloud computing and virtualization software and services. It was one of the first commercially successful companies to virtualize the x86 architecture.

vCenter Appliance root Partition 100% full

What is vCenter Server Appliance – The vCenter Server Appliance is a preconfigured Linux virtual machine, which is optimized for running VMware vCenter Server® and the associated services on Linux.

Symptoms

100% capacity used for /dev/sda3.

Size of audit.log file is very large and /var/log/audit folder consumes majority of the space.

Saved logs from log rotate policy reference a date that is not in line with the policy.

Unable to connect to the vCenter Server as services are not started.

Running /etc/cron.daily/logrotate manually rotates logs as expected.

Accessing vSphere Web Client might fail with error: 503 service unavailable

To resolve this issue, truncate the audit.log file and verify the cron job is working correctly.

Step 1: Truncate audit.log

1. Log in to the vCenter Server Appliance through SSH.
2. Run this command to enable access the Bash shell:

$ shell.set --enabled true
$ shell

Navigate to the /var/log/audit folder 

$ cd /var/log/audit

Run below command to verify the issue is with the audit.log file being too large (a few GBs):

$ ls -lh

Example output;

$ ls -lh

total 4.1M
drwx------ 2 root root 4.0K Nov 17 20:36 audispd
-rw------- 1 root root 4.1G Nov 17 23:08 audit.log

Truncate (clean the content without deleting the file) the audit.log file with this command:

$ truncate -s 0 audit.log

Step 2: Verify that the cron job to rotate the audit.log is running

Run this command to see when the cron job was last ran successfully:

$ ls -l /var/spool/cron/lastrun/


total 0
-rw------- 1 root root 0 Apr 27 2017 cron.daily
-rw------- 1 root root 0 Apr 27 2017 cron.hourly
-rw------- 1 root root 0 Apr 26 2017 cron.weekly

Run below command to check for credential failures.

$ grep "Authentication token is no longer valid; new one required" /var/log/messages.0.log | head

grep "Authentication token is no longer valid; new one required" /var/log/messages.0.log | head

2018-11-07T00:20:01.617180+00:00 vcenter /usr/sbin/cron[18972]: Authentication token is no longer valid; new one required
2018-11-07T00:20:01.617183+00:00 vcenter /usr/sbin/cron[18974]: Authentication token is no longer valid; new one required

Run below command to check if the root password has expired:

$ chage -l root

Minimum: 0
Maximum: 365
Warning: 7
Inactive: -1
Last Change: Feb 03, 2018
Password Expires: Feb 03, 2019
Password Inactive: Never
Account Expires: Never

If password expired please change root password

Restart all vCenter Server services.

$ service-control --stop --all
$ service-control --start --all

Disable root password expiration

$ chage -m 0 -M 99999 -I -1 -E -1 root

More information @ VMware KB 2149278