[VCF 9.0 - SDDC Manger] Doesn't boot properly after upgrade

Issue

A few days ago, I was testing a VMware Cloud Foundation (VCF) upgrade in my lab, specifically moving from version 9.0.0 to 9.0.1.0.

NOTE: Before proceeding with any upgrades, always make sure you have reliable backups of your various components. Additionally, before taking any action, it's highly recommended to take snapshots of the involved components.

During the SDDC Manager upgrade phase...

After a few minutes, following the automatic reboot of the SDDC Manager appliance, I was greeted with this error message:

Authorization Error : Unauthorized access.

This message was present in both the SDDC Manager UI...

...and in the Lifecycle Manager.

At this point, the upgrade seemed to be completely stuck.

Solution

Disclaimer: Some of the procedures described below may not be officially supported by VMware. Use it at your own risk.

Googling around, I found the following Broadcom article: VCF Operations 'SDDC Manager' tab shows "Authorization Error: Unauthorized access".

As indicated in the KB, I went to the VM console to check if the SDDC Manager issue matched the symptoms described:

SDDC Manager is inaccessible and keeps spinning
SDDC manager displays CPU errors similar to :
[2989.634241] NMI watchdog: BUG: soft lockup - CPU#0 stuck for 42s! [jsvc:8372]

However, my finding was completely different from what the KB described. The SDDC Manager was actually booting into Emergency Mode, showing the following errors:

[    2.8952191 integrity: Problem loading X.509 certificate -22
[FAILED] Failed to mount /boot/efi.
[DEPEND] Dependency failed for Local File Systems.
"journalctl
You are in emergency mode. After logging in, type "systemctl default" or
^D to boot into default mode.
Give root password for maintenance (or press Control-D to continue):

It appeared the system was unable to correctly mount the /boot/efi file system.

I entered the root password and tried a simple reboot just to see if it would clear up, but without success.
Checking the file system, I confirmed it was failing on /boot/efi. I started investigating the mounts.

cat /etc/fstab

blkid

lsblk -f

I initially tried to fix the issue by replacing the UUID in /etc/fstab with the new ID retrieved from the lsblk command, but that didn't work.

vi /etc/fstab

So, I decided to comment out the /boot/efi line entirely. The appliance booted up successfully, but threw these firewall errors:

At this point, I realized the system wasn't properly loading its IP address.
I forced the manual configuration of the IP and gateway directly from the command line:

ifconfig eth0 10.1.1.5 netmask 255.255.255.0
route add default gw 10.1.1.1

Once the network was up, I restarted the SDDC Manager services using the built-in script:

/opt/vmware/vcf/operationsmanager/scripts/cli/sddcmanager_restart_services.sh

After the services restarted, I opened the SDDC Manager web interface to force or continue the upgrade process then I checked the logs.

Surprisingly, after a while, the logs showed that the upgrade had actually completed successfully!

However, doing a quick check on VCF Operation, it still appeared as disconnected.

But when I verified the target version in the SDDC Manager UI, it displayed the correct upgraded version.

To clean things up, I went ahead and rebooted the VCF Operation appliance.

After waiting for it to reboot, I logged back in to verify the status, and everything was finally green and fully connected.

With this roadblock cleared, I was able to safely continue upgrading the rest of the lab environment.

That's it.

[NSX - KB406460 ] NSX_OPSAGENT on ESXi node

Issue

Today has been release the KB 406460 related "The memory usage of agent NSX_OPSAGENT on ESXi node <UUID> has reached <kb> kilobytes which is at or above the high threshold value of 80%"

Solution

As a temporary workaround to the issue as mentioned in option 1, which consist in to restart OpsAgent on the affected hosts; I wrote a short prowershell script to restart the agent on all hosts connected to vCenter Cluster.
Let's see it below:

##########
# 
# Run remote commands (Linux like) on esxi hosts to restart /etc/init.d/nsx-opsagent
#
# How it works:
# 	Connect to vCenter
# 	Get the list of ESXi hosts from the cluster
# 	Enable SSH on host
# 	Restart "/etc/init.d/nsx-opsagent" service on the host
# 	Disable SSH on host
#
# Requirement: Install-Module -Name Posh-SSH
#
# LM 22.05.2025
##
Import-Module -Name Posh-SSH

#Replace the parameter below with your values
$esxiUser = "root"
$esxiPassword = "<ESXi - PASSWORD>"
$vc = "<vCenter IP or FQDN>"
$vcUser = "administrator@vsphere.local"
$vcPassword = "<vCenter Password>"
$clusterName = "<Cluster Name>"

Connect-VIServer -Server $vc -User $vcUser -Password $vcPassword

$count=0
foreach ($esxiIP in (Get-Cluster -Name $clusterName | Get-VMHost)) {
  $count = $count + 1      
  Write-Host " ----------------------- $($esxiIP) ----------------------------------"
  # Enable SSH 
  Write-Host " Enabling SSH! " -ForegroundColor Green
  Get-VMHost -Name $esxiIP| Get-VMHostService | ?{"TSM-SSH" -eq $_.Key} | Start-VMHostService

  #SSH connection and service restart 
    $session = New-SSHSession -ComputerName $esxiIP -Credential (New-Object System.Management.Automation.PSCredential($esxiUser, (ConvertTo-SecureString $esxiPassword -AsPlainText -Force)))  -Force
    if ($session.Connected) {
        $command = "/etc/init.d/nsx-opsagent restart"
        $result = Invoke-SSHCommand -SessionId $session.SessionId -Command $command 
    
        if ($result.ExitStatus -eq 0) {
            Write-Host "Service restarted! on Host ->"$esxiIP -ForegroundColor Green
        } else {
            Write-Host "Error on host $($esxiIP): $($result.Error)" -ForegroundColor Red
        }
    
        Remove-SSHSession -SessionId $session.SessionId | Out-Null
    } else {
        Write-Host "SSH Connection failed! on Host ->"$esxiIP -ForegroundColor Red
    }

  sleep 1
  # Disable SSH
  Get-VMHost -Name $esxiIP| Get-VMHostService | ?{"TSM-SSH" -eq $_.Key} | Stop-VMHostService -Confirm:$false

  Write-Host " ----------------------------------------------------------------------------"
  Write-Host
}

Disconnect-VIServer -Server $vc -Confirm:$false

    

That's it.

vRealize Log Insight Internal Certificate issue

Issue

A KB91441 was recently published that affects vRLI internal certificate that will expire on April 30th. The expiration on the certificate will leading to a comunication failures in internode comunication in a vRLI cluster. However, this problem affects both cluster and single instance installation of a vRLI, so I invite you to read carefully the KB and act as soon as possible.

Before to act on the customer's production cluster environment I prefer to do some tests on the safe VMware HOL environment, using this LAB "Getting Started with Aria Operations for Logs (HOL-2201-03-CMP)".

Let's see below the tasks, following the KB "Updating the vRealize Log Insight Internal Certificate (91441)" step by step.

Solution

As the KB says:

This is a known issue affecting vRealize Log Insight 8.x.
The official resolution will be in the next vRealize Log Insight due out before April 30th 2023.

In my case the vRLI is a single instance version 8.4.

Below the verification steps before the fix:

1. Let's check the external certificates, also because the problem may not arise with the expiration of the external certificate, but it is actually the internal certificate that expires:
   
   Open a web browser and open the vRLI URL and verify the certificate
   
   
   
   
2. Get access to the vRealize Log Insight with user admin (Default built-in) > Administration > SSL.
   
   Click on “VIEW DETAILS…” of Existing Certificate
   
   
   
   As you can see here, the external certificates don't seems to expire imminently
   
   
3. Connect via SSH to the vRealize Log Insight appliance with root user and run the following command:
   
   

   # openssl x509 -in /storage/core/loginsight/cidata/cassandra/config/cacert.pem -enddate -noout

   
   If the result is as indicated in the figure above, it must be updated, even if in points 1. and 2. the certificate is not expired. If the result is different and expires beyond April 30, 2023, no certificate reconfiguration activities are required.
   
   

Below the workaround:

1. Take a cold snapshot of the vRealize Log Insight VM
   
   
   
2. Generating the new self-signed certificate:
   

   openssl req -newkey rsa:2048 -keyout domain.key -x509 -days 3650 -out domain.crt -nodes 

   When prompted by openssl, provide the required values for your company.
   
   Then run the following command to concatenate the key and cert into a pem file

   cat domain.key domain.crt > /tmp/cert.pem

   
3. Download the cert.pem file and upload it to vRealize Log Insight:
   Navigate to Configuration > SSL, click Choose File, browse to the cert.pem file previusly downloaded and click Open.
   Click Save. This will automatically distribute the new cert across all nodes in the vRealize Log Insight cluster. Wait for the SSL certificate to be updated.
   
   
4. Run the "update_default_cert.sh" script downloaded from VMware KB:

   # ./update_default_cert.sh --all

   Once completed, stop the loginsight service on the node by running the following command:

   # systemctl stop loginsight

   Start the loginsight service by running the following command:

   # systemctl start loginsight

   Execute the script with the verify option:

   # ./update_default_cert.sh --verify

   
5. Run the following command to validate that the new certificate is in place:

   # openssl x509 -in /storage/core/loginsight/cidata/cassandra/config/cacert.pem -enddate -noout

   
   Verify the certificate by UI as well:
   
   
   
6. If everything seems to be OK, remove the snapshot.

Another way to address the criticality is to upgrade vRealize Log Insight to new version before the April 30 deadline because the new 8.12 version is not affected by this issue.
At the time I'm posting this article (April 20th, 2023), the new release 8.12 has just been released, and is available for download.
VMware Aria Operations for Logs 8.12 release notes are available here.

That's it.