lunedì 16 marzo 2026

[VCF 9.0] How to Set Up a HTTPS VCF Offline Depot

I'm deploying a new Holodeck 9.0.2 for a personal lab; and the first task to perform to set up the environment is to download the following components (as shown in the official page):

  1. HoloRouter OVA
  2. VCF Binaries
  3. Offline Depot (For air-gapped or internet-restricted environments)
As an offline depot, we'll download, deploy, and use the OVA already present in the link above (3). We'll configure the appliance's NGINX service with a new self-signed certificate to accept HTTPS requests.

Some circumstances may require using the offline method. In this post, I'll explain exactly how to implement it. I've created a bash script to automate these settings (available at the bottom of the page). Let's see how:
  • Download the Offline Depot OVA
  • Deploy the OVA "vcf-offline-depot-appliance-0.1.3.ova" (detailed steps on how to distribute an OVA are beyond the scope here).
    Start the appliance ...

  • Download the VCF Offline Depot metadata from the Broadcom Portal.
  • Once the appliance is up and running, copy the previous metadata file to the appliance under the /tmp directory. Use the "admin" user.
    In my case,
    scp ./Downloads/vcf-9.0.2.0-offline-depot-metadata.zip admin@192.168.1.96:/tmp
  • SSH on the appliance using "admin" user ... and then become "root" typing
    sudo -i
  • Move the "vcf-9.0.2.0-offline-depot-metadata.zip" file under the root folder ...
    mv /tmp/vcf-9.0.2.0-offline-depot-metadata.zip .
    Note: later on, if the *.metadata.zip file is present into the root directory the script "configure-https-nginx.sh" will use it to create the structure.

  • Create a new bash script "configure-https-nginx.sh". In my case ....
    vi configure-https-nginx.sh
  • Copy and paste the script at the bottom of the page into that file ...
  • Edit the file settings/variables according to your needs (as shown in the photo below) ... specifically customize SERVER_NAME (This variable will also be used in the "nginx.conf" configuration file) ...
  • ... do the same thing for the values to be inserted in the certificate generation request file, modifying the req_distinguished_name, alt_names parameters as desired (from line 42 to 59)...
  • When you think you've made the changes to your liking, save the file and grant it execute permission.
  • Run the script ...
    ./configure-https-nginx.sh
  • Check the service's reachability via web browser by connecting to the server in HTTPS (in my case https://192.168.1.96) ...
    ... check the certificate ...
    ... points to the right URL, in my case https://192.168.1.96/PROD/, and login with one of the two previously created default users, "admin" or "admin-depot" (the default password for both is VMware123!)
  • Next, download all required VCF binaries.
  • After downloading the files, create the following directory structure and move the appliance files into their respective folders.

    Once this is completed, your local depot structure will be ready.
  • Your Offline Depot is ready to use


Below is the "configure-https-nginx.sh" bash script ...

#!/bin/bash
# configure-https-nginx.sh
# Create new Self-signed Certificate
# Enable Https on NGINX
# Create .htpassword access file
# Create the metadata structure
# Lorenzo Moglie
#
SERVER_NAME="vcf-depot.vcf.lab"
DIR_NGINX="/etc/nginx"
NGINX_CONF_ORIG="${DIR_NGINX}/nginx.conf.orig"
VCF_METADATA="vcf-9.0.2.0-offline-depot-metadata.zip"

# ----
# Back up the original nginx.conf file, if not already present
if [ -f "$NGINX_CONF_ORIG" ]; then
    echo "0. Backup file $NGINX_CONF_ORIG present."
else
    echo "0. Backup file $NGINX_CONF_ORIG created."
    cp ${DIR_NGINX}/nginx.conf ${NGINX_CONF_ORIG}
fi

# ----
# Create the folder and new certificates based on the provided cert-req.cfg file
DIR_SSL="${DIR_NGINX}/ssl"
if [ -d "$DIR_SSL" ]; then
    echo "1. Direcroty $DIR_SSL present."
else
    mkdir -p $DIR_SSL
    echo "1. Directory $DIR_SSL created."
fi

# Create the cert-req.cfg file
CONFIG_FILE="${DIR_SSL}/cert-req.cfg"
echo "2. Creating the $CONFIG_FILE configuration file..."
cat <<EOF > $CONFIG_FILE
[req]
distinguished_name = req_distinguished_name
req_extensions = req_ext
prompt = no

[req_distinguished_name]
C = IT
ST = Rome
L = Rome
O = LorenzoMoglie
OU = vcf.lab
CN = $SERVER_NAME

[req_ext]
subjectAltName = @alt_names

[alt_names]
IP.1 = 172.16.10.98
IP.2 = 10.1.10.251
IP.3 = 192.168.1.96
DNS.1 = $SERVER_NAME
DNS.2 = $SERVER_NAME
DNS.3 = $SERVER_NAME
EOF


echo "3. Generating the Private Key and CSR...."
openssl req -new -nodes -out ${DIR_SSL}/vcf-depot.csr -newkey rsa:2048 -keyout ${DIR_SSL}/vcf-depot.key -config $CONFIG_FILE
if [ $? -eq 0 ]; then
    echo "   [OK] CSR and Key generated successfully."
else
    echo "   [ERROR] CSR generation failed."
    exit 1
fi

echo "4. Generating a Self-Signed Certificate (valid for 10 years)..."
openssl x509 -req -days 3650 -in ${DIR_SSL}/vcf-depot.csr -signkey ${DIR_SSL}/vcf-depot.key -out ${DIR_SSL}/vcf-depot.crt -extensions req_ext -extfile $CONFIG_FILE
if [ $? -eq 0 ]; then
    echo "   [OK] vcf-depot.crt certificate successfully created."
else
    echo "   [ERROR] Certificate creation failed."
    exit 1
fi

echo "------------------------------------------------------"
echo "Certificate generation completed. Generated files in the current folder:"
ls -l ${DIR_SSL}/vcf-depot.*
echo "------------------------------------------------------"


# ----
# Create the a new nginx.conf file
NGINX_CONF="${DIR_NGINX}/nginx.conf"
echo "5. Creating a new ${NGINX_CONF} file ... "
cat <<EOF > $NGINX_CONF
events {
    worker_connections  1024;
}

http {
  server {
    listen 80;
    server_name $SERVER_NAME;

    root /var/www;  

    # default
    location = / { return 200 "OK\n"; add_header Content-Type text/plain; }

    location /www/ {
        alias /var/www/;
        autoindex on;
    }
  }

  # --------------------------
  # HTTPS for VCF Offline Depot
  # --------------------------
  server {
    listen 443 ssl;
    server_name $SERVER_NAME;

    # Path
    root /var/www;

    # Bundle's path
    location /PROD/ {
            alias /var/www/build/PROD/;
            autoindex on;
            auth_basic "Restricted";
            auth_basic_user_file /etc/nginx/.htpasswd;
    }

    # Certificato self-signed
    ssl_certificate     /etc/nginx/ssl/vcf-depot.crt;
    ssl_certificate_key /etc/nginx/ssl/vcf-depot.key;

    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers HIGH:!aNULL:!MD5;

    access_log /var/log/nginx/vcf_repo_ssl_access.log;
    error_log  /var/log/nginx/vcf_repo_ssl_error.log;
  }
}
EOF

echo "6. Check ${NGINX_CONF} consistency configuration...."
nginx -t
if [ $? -eq 1 ]; then
    echo "   [ERROR] ${NGINX_CONF} not properly configured "
    exit 1
fi


HTP="${DIR_NGINX}/.htpasswd" 
echo "7. Creating ${HTP} file .... with the following users:"
echo "   [Default user]: admin / VMware123! "
echo "   [Default user]: admin-depot / VMware123! "
cat <<EOF > $HTP
admin-depot:\$6\$PlYrZXU7oL2SOF/d\$yrRCifrKhsSL8KB18SXnoQ.EgkRMhV5ocdkDXaeefXg1ZkTAYj3IVijfhc4Rvsz1/O7CSda8iOavaRB0tce9s.
admin:\$6\$n8HWw9EPBJus5PJw\$ncuE7HIQjGo4c1PDupxm5znRegK8N8CESesedQia3x5Ocypy8ISF5TNQLTZBdaNE1B5zemVvpTXmQZtp6KcP4.
EOF
chown nginx $HTP

#Restart NGINX
echo "8. Restart NGINX service."
systemctl restart nginx.service

echo "9. Enabling IPTABLES to accept request on port 443."
iptables -I INPUT -p tcp --dport 443 -m state --state NEW,ESTABLISHED -j ACCEPT
iptables-save > /etc/systemd/scripts/ip4save
if [ $? -eq 0 ]; then
    echo "   [OK] IPTABLES configuration properly saved "
else    
    echo "   [FAIL] IPTABLES configuration not saved "
fi

echo "10. Creating the new PROD folder "
#echo "   [unzip the metadata file ${VCF_METADATA} under the folder --> /var/www/build/PROD]"
mkdir -p /var/www/build/PROD
chown nginx /var/www/build/PROD

echo "11. Metadata file ... "
if [ -f "$VCF_METADATA" ]; then
    echo "   [OK] unziping metadata file ${VCF_METADATA} under the folder --> /var/www/build/PROD"
    unzip ${VCF_METADATA} -d /var/www/build/
else
    echo "   [FAILED] Metadata file not present. DOwnload and manually unzip the file under the folder --> /var/www/build/PROD"
fi

That's it.

lunedì 9 marzo 2026

[Quick TIP - VCF 9.0] - Operation not allowed by the credentials remediate in progress

Issue

I am in the situation where I have to change some settings to the current VCF 9.0.2.0 backup configuration ..... and the "EDIT" button is grayed out and I get the following message:

This operation is not allowed because the system lock is held by the Credentials remediate operation operation in progress.



Solution

Connect to the SDDC Manager

Go to ... Security > Password Management > Cancel (1) and confirm by pressing YES (2)

Switch back to the VCF Operations UI and the buttons are clickable again.



That's it.

mercoledì 4 marzo 2026

[VCF 9.0 - SDDC Manger] Doesn't boot properly after upgrade

Issue

A few days ago, I was testing a VMware Cloud Foundation (VCF) upgrade in my lab, specifically moving from version 9.0.0 to 9.0.1.0.

NOTE: Before proceeding with any upgrades, always make sure you have reliable backups of your various components. Additionally, before taking any action, it's highly recommended to take snapshots of the involved components.

During the SDDC Manager upgrade phase...

After a few minutes, following the automatic reboot of the SDDC Manager appliance, I was greeted with this error message:

Authorization Error : Unauthorized access.

This message was present in both the SDDC Manager UI...

...and in the Lifecycle Manager.

At this point, the upgrade seemed to be completely stuck.



Solution

Disclaimer: Some of the procedures described below may not be officially supported by VMware. Use it at your own risk.

Googling around, I found the following Broadcom article: VCF Operations 'SDDC Manager' tab shows "Authorization Error: Unauthorized access".

As indicated in the KB, I went to the VM console to check if the SDDC Manager issue matched the symptoms described:

SDDC Manager is inaccessible and keeps spinning
SDDC manager displays CPU errors similar to :
[2989.634241] NMI watchdog: BUG: soft lockup - CPU#0 stuck for 42s! [jsvc:8372]

However, my finding was completely different from what the KB described. The SDDC Manager was actually booting into Emergency Mode, showing the following errors:

[    2.8952191 integrity: Problem loading X.509 certificate -22
[FAILED] Failed to mount /boot/efi.
[DEPEND] Dependency failed for Local File Systems.
"journalctl
You are in emergency mode. After logging in, type "systemctl default" or
^D to boot into default mode.
Give root password for maintenance (or press Control-D to continue):

It appeared the system was unable to correctly mount the /boot/efi file system.

I entered the root password and tried a simple reboot just to see if it would clear up, but without success.
Checking the file system, I confirmed it was failing on /boot/efi. I started investigating the mounts.

cat /etc/fstab
blkid
lsblk -f

I initially tried to fix the issue by replacing the UUID in /etc/fstab with the new ID retrieved from the lsblk command, but that didn't work.

vi /etc/fstab

So, I decided to comment out the /boot/efi line entirely. The appliance booted up successfully, but threw these firewall errors:

At this point, I realized the system wasn't properly loading its IP address.
I forced the manual configuration of the IP and gateway directly from the command line:

ifconfig eth0 10.1.1.5 netmask 255.255.255.0
route add default gw 10.1.1.1

Once the network was up, I restarted the SDDC Manager services using the built-in script:

/opt/vmware/vcf/operationsmanager/scripts/cli/sddcmanager_restart_services.sh

After the services restarted, I opened the SDDC Manager web interface to force or continue the upgrade process then I checked the logs.

Surprisingly, after a while, the logs showed that the upgrade had actually completed successfully!

However, doing a quick check on VCF Operation, it still appeared as disconnected.

But when I verified the target version in the SDDC Manager UI, it displayed the correct upgraded version.

To clean things up, I went ahead and rebooted the VCF Operation appliance.

After waiting for it to reboot, I logged back in to verify the status, and everything was finally green and fully connected.

With this roadblock cleared, I was able to safely continue upgrading the rest of the lab environment.

That's it.

domenica 1 marzo 2026

[SSP 5.1.0] vDefend SSP Backup Failures: Stuck at 60%

Issue


If you manage a VMware vDefend Security Services Platform (SSP) environment, you know how critical automated backups are. Recently, I ran into a frustrating issue where scheduled backups started failing mysteriously. What initially seemed like a simple user and misconfigured settings issue, “network timeout” turned out to be a bizarre problem involving lost encryption keys and crashing containers.
Looking on the SSP UI at the System > Backup and Restore dashboard, I noticed a long list of failed backup jobs. Hovering over the failure status yielded a very generic message: "operation timed out, please contact administrator".
I tried manually forcing the backup, but it seems to stuck at 60% and then time out after a while.
The generic timeout error on the Backup and Restore dashboard


Solution


My first instinct was to check the SFTP credentials, re-type the password, SAVE and seemed fine.
Googling around, I found this interesting link "Backup and Restore goes into Failed State" which helped me in the final resolution.

Since the UI wasn't giving me enough technical details, it was time to drop into the CLI. I logged into the SSPI appliance via SSH as sysadmin to check what was happening at the Kubernetes pod level.

I ran a quick ...
kubectl get pods -n nsxi-platform | grep backup
... and immediately spotted the problem. The backup job pods weren't just timing out; they were actively crashing, throwing Error and OOMKilled statuses.
To understand why the pods were in this state ("OOMKilled", "Error"), I checked the logs of both pods....
k logs job-backup-4fv4k-5gq5s -n nsxi-platform
Scrolling through the container logs, I could see ... the system passed a malformed argument to the OpenSSL encryption command.
The log explicitly stated:
Error executing command: [openssl enc -aes-256-cbc -salt -in ... -out ... -k ******]. 
Error: fork/exec /usr/bin/bash: argument list too long.
Scrolling through the container logs, I could see the backup job actually doing its work: generating the .obj and .dump files from the Postgres database and config files.
However, the process abruptly halted right after the ...
<INFO> ... Started encrypting backup bundle
... message.


Conclusion

How did I fix it? I re-entered the password and retyped the passphrase to ensure the openssl process received a valid string, allowing the container to properly encrypt the dump files and upload them to the SFTP server.

The mystery was solved. The "timeout" in the UI was just a side effect. The real issue was that the missing encryption key caused the openssl command to fail spectacularly (resulting in an argument list too long OS error), which crashed the container pod.

It appeared the platform had somehow lost track of the encryption key. The configuration was stuck in a weird state where trying to "Reuse prior passphrase" wasn't working.

Now starting the backup in manual mode, it completes successfully.


That's it.