VMFS-6 heap memory exhaustion on vSphere 7.0 ESXi hosts (80188)

Issue

I recently experienced the problem indicated in KB80188 ("VMFS-6 heap memory exhaustion on vSphere 7.0 ESXi hosts"). Not having the possibility to upgrade to later versions where the problem has been fixed. So, to solve the problem, I created a small script that checks the VMFS-6 volumes mounted and executes the workaround indicated in the KB.

Solution

Disclaimer: Use it at your own risk.

The workaround is to create Eager zeroed thick disk on all of the mounted VMFS6 datastores and then delete it.
Below the script:

# Author: Lorenzo Moglie (ver.1.1 04.04.2022)
#
# VMFS-6 heap memory exhaustion on vSphere 7.0 ESXi hosts (80188)
# https://kb.vmware.com/s/article/80188
# filename: kb80188.sh
#
# WARNING : Use the script at your own risk
#

esxcli storage filesystem list | while read -r LINE; do
TYPE=`echo $LINE | awk -e '{print $5}'`
if [ $TYPE == "VMFS-6" ]; then
 VOLUME=`echo $LINE | awk -e '{print $1}'`
 vmkfstools -c 10M -d eagerzeroedthick $VOLUME/eztDisk`hostname`
 esxcli system syslog mark --message="KB80188 - Created disk  $VOLUME/eztDisk`hostname`"
 vmkfstools -U $VOLUME/eztDisk`hostname`;  echo "Deleted."
 esxcli system syslog mark --message="KB80188 - Deleted disk  $VOLUME/eztDisk`hostname`"
fi
done

Workaround has to be done for each datastore on each host.

So I suggest to copy it on each ESXi hosts root and scheduling it in the cron of the host. This because If you copy it on a shared datastore may not work properly on every hosts. A great explanation written by Mike Da Costa of how to schedule tasks in cron on esxi can be found here.

For example

1. Copy the workaround script into the environment. (In my case /kb81088.sh)
2. Give the script executable permissions
   chmod +x /kb81088.sh
3. On each hosts, edit /var/spool/cron/crontabs/root
4. Add the line to the above file, to schedule the execution every 5 hours
   0 */5 * * * /kb81088.sh
5. Now, we need to kill the crond PID.
   First, get the crond PID (process identifier) by running the command "cat /var/run/crond.pid"
6. Next, kill the crond PID. Be sure to change the PID number to what you obtained in the previous step.
   Example running the command "kill 2098332"
7. Once the process is stopped, you can use BusyBox to launch it again, running the command "/usr/lib/vmware/busybox/bin/busybox crond" to restart crond process

That's it.