Issue
Holodeck is a powerfull toolkit designed to provide a standardized and automated method to deploy nested VMware Cloud Foundation (VCF) environments on a VMware ESX host or a vSphere cluster for your homelab learning test.
All informations about holodeck are available here.
The appliance can be deployed with a static IP or with DHCP.
What happens if I deploy the appliance using DHCP, then shut it down, and when I power it back on, it receives a different IP address?
The answer is that after reboot, previously configured services may fail to start properly, causing the Kubernetes control plane to become unresponsive.
I retrieve the new IP address and attempt to access the appliance via the web interface ...
Solution
Disclaimer: Use it at your own risk.
As a quick fix, if the old IP is still available, simply set the old IP in the network configuration as a static IP and restart the network services. The pods will be activated fairly quickly.
If the IP is unavailable, follow the steps below.
1. Stop iptables
First of all I stop the iptables/Firewall service to connect via SSH to the VM.
# systemctl stop iptables
2. Check status
Connect via SSH to the Holorouter and control the Kubernetes pod, with the following command:# kubectl get pods
Previous IP: 192.168.1.70
Current IP: 192.168.1.238
I check the network configuration as well ...
# cat /etc/systemd/network/50-static-en.network
3. Re-init the kubernetes control plane
To reinitialize the Kubernetes control plane and allow the server API and pods to function properly again, I created the following script (downloadable below). The script also changes the network settings from DHCP to statically with the new IP address obtained (in my case, 192.168.1.238).I create the new file in the root path:
# vi change-control-plane-ip.sh
I paste what you can see in the image (script below); I save and run the script...
4. Reboot and check results
I restart the appliance and perform the post-reboot check ...# reboot
To check if the pods have powered up, I log in to the appliance and run the following command:
# kubectl get pods
Boom!! It works
Below the script used change-control-plane-ip.sh
# change-control-plane-ip.sh
# Stop Services
systemctl stop kubelet docker
# Backup Kubernetes and kubelet
mv -f /etc/kubernetes /etc/kubernetes-backup
mv -f /var/lib/kubelet /var/lib/kubelet-backup
# Keep the certs we need
mkdir -p /etc/kubernetes
cp -r /etc/kubernetes-backup/pki /etc/kubernetes
rm -rf /etc/kubernetes/pki/{apiserver.*,etcd/peer.*}
# Start docker
systemctl start docker
# Get IP address
IP=`ip -o -4 addr show eth0 | awk '{print $4}' | cut -d/ -f1`
# Init cluster with new ip address
kubeadm init --control-plane-endpoint $IP --ignore-preflight-errors=all --v=5
# Verify resutl
kubectl cluster-info
# Change IP on the configuration file
cp /etc/systemd/network/50-static-en.network /etc/systemd/network/50-static-en.network.backup
cat > /etc/systemd/network/50-static-en.network << EOF
[Match]
Name=eth0
[Network]
Address=`ip -o -4 addr show eth0 | awk '{print $4}'`
Gateway=`ip route show 0.0.0.0/0 dev eth0 | cut -d\ -f3`
DNS=10.1.1.1 null
EOF
That's it.
