"log disk exhaustion" warning on vCenter due to vmafdd.log file not compressed

Issue

Everything start with "log disk exhaustion" warning on vCenter. This warning comes out when the /storage/log partition reach out 80% ot its space.

Log in vCenter via SSH and checking the disk space with following command:

# df -h

I remove some logs following the " KB83070 - How to clear space on VCSA /storage/log partition"
Then, checking storage logs for disk utilization performing the following command:

# du -a /storage/log | sort -n -r | head -n 20

... you find that the vmafdd.log is extremely large.

Solution

This happens, as the KB83238 says, because the logrotate.d configuration for the vmafd service expects the logs to be located in the following path "/var/log/vmware/vmafdd/vmafdd.log" instead of the current path "/var/log/vmware/vmafd/vmafdd.log".

# cat /etc/logrotate.d/vmware-vmafd.lr

Checking the vmafdd registry we can see the incorrect path:

# /opt/likewise/bin/lwregshell list_values "[HKEY_THIS_MACHINE\Services\vmafd\Parameters]"

Before to do any changes is a good practice to take a snapshot.

Taken the snapshot, access the vCenter in SSH again ... and is now possible to update the registry to match the log rotate configuration path, using the following command:

# /opt/likewise/bin/lwregshell set_value "[HKEY_THIS_MACHINE\Services\vmafd\Parameters]" "LogFile" "/var/log/vmware/vmafdd/vmafdd.log"

... then perform the following command to verify that the change as been applied:

# /opt/likewise/bin/lwregshell list_values "[HKEY_THIS_MACHINE\Services\vmafd\Parameters]"

Restart all services:

# service-control --stop --all && service-control --start --all

If we look now inside the folder "/var/log/vmware/vmafdd/" we can see the file "vmafdd.log" ...

... than we can remove the previous huge "vmafdd.log" file ...

checking again the /storage/log space, we don't see anymore the huge "vmafdd.log" file.

# du -a /storage/log | sort -n -r | head -n 20

Seeing the occupation of the entire disk, we can see that we have freed up enough space.

# df -h

That's it.

How to change the root password of ESXi hosts managed by VMware Cloud Foundation (VCF)

Issue

I need to change manually the root password of the ESXi hosts managed by VMware Cloud Foundation.

Solution

It is possible to manually change the password for the ESXi host root account (unlike password rotation, which generates a randomized password) managed by the VMware Cloud Foundation infrastructure, by logging in directly from the SDDC manager.

Log in to the SDDC Manager UI as a user with the ADMIN role.
From the navigation pane, select Administration (1) > Security (2) > Password Management (3), and be sure that ESXi (4) is selected.

Click on the ⋮ (5) > UPDATE (6)

Provide new password ...

... then Click UPDATE

Wait untill the credentioal are properly update on the host ...

... verify the Date in Last Modified field if properly reflect the changes

Repeat same procedure describe above for each host ESXi you want change the password.

Try logging into the host to verify the change.

Insert the new password.

Official documentation on how Manually Update Passwords on VCF is available at this link.

That's it.

OpenVPN Access Server on Photon OS (with docker)

Issue

How to build a fast and easy VPN solution for a secure connection to the own infrastructure.

Solution

As a solution I opted for VMware Photon OS, docker and took advantage of this "linuxserver/openvpn-as" project based on OpenVPN.

In this quick guide I assume that Photon OS (minimal) is already installed and properly configured (such as IP address, DNS, NTP server and so on). In my case Photon OS i configured with 1 vCPU, 2 GB of RAM and 16GB of Disk (even if for the purpose of this scope 8 GB are enough).

First thing to do, we check if docker is installed, if not we install it or start the service and enable it to start when the VM starts.

# systemctl status docker

# systemctl start docker

# systemctl enable docker

I then, created a script to automate the installation and running of the container.

The outcome of the script is the execution of the container, and we will be asked to enter a new password for the default admin account.
Below the script:

#/bin/bash
#
# Initial setup script (all-in-one) to access OpenVPN Web UI

# Install the package called OpenVPN access server
# Download and install
docker pull linuxserver/openvpn-as:latest

# Let's create a new docker container called openvpn-as with the following:
#  -v /home/docker/openvpn-as/config: /config - Sets the directory to store tehconfig files.
#  --restart=always - Ensures the container always starts on system boot. You can opt not to add this agument if you don't want to container to restart
# -e PGID=1001 -e PUID=1001 - Sets the user ID to eliminate permission issues between the host server and the container.
# -e TZ=Europe/Rome - Set Time Zone
# --net=host --privileged - Dictates how OpenVPN Access Server runs in the container.
docker create --name=openvpn-as --restart=always -v /home/docker/openvon-as/config:/config -e INTERFACE=eth0 -e PGID=1001 -e PUID=1001 -e TZ=Europe/Rome --net=host --privileged linuxserver/openvpn-as

# Start the container
docker start openvpn-as

# Get access to docker and change the default admin password
docker exec -it $(docker ps | grep openvpn-as | awk '{print $1}') /usr/bin/passwd admin
 

We give execute permissions to the script.

# chmod +x OpenVPN_initial_setup.sh

Let's execute it ...

# ./OpenVPN_initial_setup.sh

... and at the prompt enter the new password

Let's verify that the container is running correctly

# docker ps

Let's connect to the local IP address (https://local-IP:943/admin/) for a first configuration ... providing the username admin and the previous setted password.

We accept the terms of use

Once logged in, first of all let's set up in Configuration > Network Settings > Hostname or IP Address ... insert the public IP or the FQDN. Then, we click on Save Settings to save the configuration.

We enable Google Authenticator Multi-Factor Authentication in the menu Authentication > General ... and saving the settings.

We then enabling and saving the "Require user permissions record for VPN access" in User Management > User Permissions.

We create a new Profile in User Profiles and the we click on New Profile.

We download and save locally on its own computer the .ovpn file, that will be used later for the VPN connection.

Client configuration part

Configure one of the following Apps on your mobile device, Google Authenticator, Microsoft Authenticator, FortiToken. I have verified that the 2FA system works with all of the above authenticator apps.

Open the web client at the VPN endpoint address (https://Public-IP:9443/) and, once logged in with the admin account, scan the QR Code with the newly installed App.

Insert the 6-digit code and download the right VPN client for its own OS/device.

Once installed, double click on the previous .ovpn file downloaded to import the VPN configuration.

Click on Connect ...

Provide username and password ...

... insert the 6-digits of the 2FA ...

We can now able to reach the remote site in a safe mode.

Conclusion

OpenVPN Access server is very simple to install and configure. However, there are some limitations due to the free license (only two connections available), the "linuxserver/openvpn-as" project is deprecated and no longer maintained, consequently the latest available version of OpenVPN is version 2.9.0 (currently version available for download is 2.11.3).

I saw that there are also other interesting project like Wireguard, not yet available as a container for docker but available as a native package for Photon OS.
Must try as soon as possible.

That's it.