[Holodeck] - Troubleshooting VCF 9 Online Depot Connectivity in a Holodeck Environment

Issue

If you are deploying VMware Cloud Foundation 9 (VCF 9) within a Holodeck nested lab environment, you might hit a roadblock when trying to configure the Online Depot.

The Online Depot is crucial for pulling down software bundles and compatibility data from Broadcom. However, in a nested environment where networking layers can get complicated, simple internet connectivity isn't always guaranteed.

Here is a walkthrough of a problem I encountered recently, how I diagnosed it, and the fix involving the Holorouter.

I was attempting to configure the Online Depot in the VCF Operations console (Lifecycle > VCF Management > Depot Configuration).

I entered my Broadcom Token as required. However, almost immediately after clicking "OK," I was greeted with a red banner error:

Error in setting Online depot configuration

I also attempted to simply view the certificate details to check the connection. That failed as well with a timeout error:

Connection failed - connect timed out

Solution

To understand what was happening under the hood, I SSH'd into the Fleet Management VM (OPSLCM). This is the appliance responsible for handling Lifecycle Manager operations.
I navigated to the log directory:

# cd /var/log/vrlcm/

I then tailed the main log file to watch the traffic in real-time while I retried the configuration in the UI:

# tail -f vmware_vrlcm.log

The logs painted a clear picture. The appliance was trying to reach Broadcom's download servers but was timing out.

INFO ... Fetching certificate from https://dl.broadcom.com
INFO ... Endpoint : https://dl.broadcom.com
ERROR ... IOException occurred - connect timed out

The log confirmed that the application was working fine, but the network wasn't. I tried running a simple curl command from the OPSLCM appliance to the internet, and that timed out too.

Here is the architecture of the issue:

1. OPSLCM (Nested VM) sends a packet to the internet.
2. The packet goes to its default gateway: the Holo-Router (10.1.1.1).
3. The Holo-Router forwards the packet out of its WAN interface (eth0) to the physical router (192.168.1.1 in my case).
4. The packet reaches the physical router with a source IP from the nested environment (e.g., 10.1.1.x). The physical router/firewall has no idea where 10.1.1.x is located—it has no route back to the nested environment managed by Holodeck. Consequently, the return traffic is dropped.
5. Usually, you might solve this by adding a static route on your physical router pointing to the Holorouter. However, in many lab scenarios (including mine), we don't have access to modify the physical network infrastructure.

The solution is to enable NAT on Holorouter.
To fix this, we need to ensure that traffic leaving the Holorouter looks like it's coming from the Holorouter's WAN IP (which the physical network does know how to route). We need to enable Source NAT (Masquerading).

I logged into the Holorouter (root@holorouter) and performed the following steps.

1. Verify IP Forwarding
   First, ensure the kernel allows forwarding (it usually does in Holodeck, as FRR is running):

   # sysctl net.ipv4.ip_forward

   Should return = 1
   
2. Apply the NAT Rule
   I added an iptables rule to masquerade traffic coming from the internal VLANs (e.g., the VLAN interface eth0.10 or the specific subnet) when it exits the WAN interface (eth0).
   In my case I created a generic rule to NAT all outgoing traffic on eth0

   # iptables -t nat -I POSTROUTING 1 -o eth0 -j MASQUERADE

   With the following command verify the insertion of the NAT rule:

   # iptables -t nat -L POSTROUTING -v -n

   
3. Make it Persistent
   Since the Holorouter might be rebooted or affected by Kubernetes network refreshes, I saved the configuration:

   # iptables-save > /etc/systemd/scripts/ip4save

Conclusions:
Immediately after applying the NAT rule, the "return path" for the traffic was established. The physical router now sees traffic coming from the Holorouter's valid WAN IP and returns it correctly. The Holorouter then untranslates the address and hands the packet back to the Fleet Management VM.

I went back to the VCF UI, clicked "Configure," and the Online Depot connected successfully.

Now can I proceed with the upgrade proces!!!

That's it.