[NSX - KB406460 ] NSX_OPSAGENT on ESXi node

Issue

Today has been release the KB 406460 related "The memory usage of agent NSX_OPSAGENT on ESXi node <UUID> has reached <kb> kilobytes which is at or above the high threshold value of 80%"

Solution

As a temporary workaround to the issue as mentioned in option 1, which consist in to restart OpsAgent on the affected hosts; I wrote a short prowershell script to restart the agent on all hosts connected to vCenter Cluster.
Let's see it below:

##########
# 
# Run remote commands (Linux like) on esxi hosts to restart /etc/init.d/nsx-opsagent
#
# How it works:
# 	Connect to vCenter
# 	Get the list of ESXi hosts from the cluster
# 	Enable SSH on host
# 	Restart "/etc/init.d/nsx-opsagent" service on the host
# 	Disable SSH on host
#
# Requirement: Install-Module -Name Posh-SSH
#
# LM 22.05.2025
##
Import-Module -Name Posh-SSH

#Replace the parameter below with your values
$esxiUser = "root"
$esxiPassword = "<ESXi - PASSWORD>"
$vc = "<vCenter IP or FQDN>"
$vcUser = "administrator@vsphere.local"
$vcPassword = "<vCenter Password>"
$clusterName = "<Cluster Name>"

Connect-VIServer -Server $vc -User $vcUser -Password $vcPassword

$count=0
foreach ($esxiIP in (Get-Cluster -Name $clusterName | Get-VMHost)) {
  $count = $count + 1      
  Write-Host " ----------------------- $($esxiIP) ----------------------------------"
  # Enable SSH 
  Write-Host " Enabling SSH! " -ForegroundColor Green
  Get-VMHost -Name $esxiIP| Get-VMHostService | ?{"TSM-SSH" -eq $_.Key} | Start-VMHostService

  #SSH connection and service restart 
    $session = New-SSHSession -ComputerName $esxiIP -Credential (New-Object System.Management.Automation.PSCredential($esxiUser, (ConvertTo-SecureString $esxiPassword -AsPlainText -Force)))  -Force
    if ($session.Connected) {
        $command = "/etc/init.d/nsx-opsagent restart"
        $result = Invoke-SSHCommand -SessionId $session.SessionId -Command $command 
    
        if ($result.ExitStatus -eq 0) {
            Write-Host "Service restarted! on Host ->"$esxiIP -ForegroundColor Green
        } else {
            Write-Host "Error on host $($esxiIP): $($result.Error)" -ForegroundColor Red
        }
    
        Remove-SSHSession -SessionId $session.SessionId | Out-Null
    } else {
        Write-Host "SSH Connection failed! on Host ->"$esxiIP -ForegroundColor Red
    }

  sleep 1
  # Disable SSH
  Get-VMHost -Name $esxiIP| Get-VMHostService | ?{"TSM-SSH" -eq $_.Key} | Stop-VMHostService -Confirm:$false

  Write-Host " ----------------------------------------------------------------------------"
  Write-Host
}

Disconnect-VIServer -Server $vc -Confirm:$false

    

That's it.

VMFS-6 heap memory exhaustion on vSphere 7.0 ESXi hosts (80188)

Issue

I recently experienced the problem indicated in KB80188 ("VMFS-6 heap memory exhaustion on vSphere 7.0 ESXi hosts"). Not having the possibility to upgrade to later versions where the problem has been fixed. So, to solve the problem, I created a small script that checks the VMFS-6 volumes mounted and executes the workaround indicated in the KB.

Solution

Disclaimer: Use it at your own risk.

The workaround is to create Eager zeroed thick disk on all of the mounted VMFS6 datastores and then delete it.
Below the script:

# Author: Lorenzo Moglie (ver.1.1 04.04.2022)
#
# VMFS-6 heap memory exhaustion on vSphere 7.0 ESXi hosts (80188)
# https://kb.vmware.com/s/article/80188
# filename: kb80188.sh
#
# WARNING : Use the script at your own risk
#

esxcli storage filesystem list | while read -r LINE; do
TYPE=`echo $LINE | awk -e '{print $5}'`
if [ $TYPE == "VMFS-6" ]; then
 VOLUME=`echo $LINE | awk -e '{print $1}'`
 vmkfstools -c 10M -d eagerzeroedthick $VOLUME/eztDisk`hostname`
 esxcli system syslog mark --message="KB80188 - Created disk  $VOLUME/eztDisk`hostname`"
 vmkfstools -U $VOLUME/eztDisk`hostname`;  echo "Deleted."
 esxcli system syslog mark --message="KB80188 - Deleted disk  $VOLUME/eztDisk`hostname`"
fi
done

Workaround has to be done for each datastore on each host.

So I suggest to copy it on each ESXi hosts root and scheduling it in the cron of the host. This because If you copy it on a shared datastore may not work properly on every hosts. A great explanation written by Mike Da Costa of how to schedule tasks in cron on esxi can be found here.

For example

1. Copy the workaround script into the environment. (In my case /kb81088.sh)
2. Give the script executable permissions
   chmod +x /kb81088.sh
3. On each hosts, edit /var/spool/cron/crontabs/root
4. Add the line to the above file, to schedule the execution every 5 hours
   0 */5 * * * /kb81088.sh
5. Now, we need to kill the crond PID.
   First, get the crond PID (process identifier) by running the command "cat /var/run/crond.pid"
6. Next, kill the crond PID. Be sure to change the PID number to what you obtained in the previous step.
   Example running the command "kill 2098332"
7. Once the process is stopped, you can use BusyBox to launch it again, running the command "/usr/lib/vmware/busybox/bin/busybox crond" to restart crond process

That's it.