Quick tip for cURL users

Issue

I often use Rest API calls with the cURL command to interact with NSX manager, and every time I have to enter the login credentials.
It would be useful to have a place somewhere to store them so that you don't have to enter them every time (especially when you are on a call with customer, and you cannot write in clear text the password with the -u option ..... and you are therefore forced to type and/or copy password several times).

Solution

Looking around in "Using curl" site I discovered .netrc .
In short, it is possible to store username, password and IP/FQDN of the machine to connect to, in file ~/.netrc so that you do not need to type username and password in every API call you invoke.
The ~/.netrc file format is simple: you specify lines with a machine name and follow that with the login and password that are associated with that machine, and looks like the below:

% cat .netrc 
machine <IP/FQDN_1> login <username_here> password <password_1_here>
machine <IP/FQDN_2> login <username_here> password <password_2_here>
% 

Below an example

lorenzo@moglielL0KPF ~ % cat .netrc 
machine 172.25.251.31 login admt1lm@dominio.local password VMware1!VMware1!
machine nsxtmgr01.customer2.local login admin password VMware!123VMware!123
lorenzo@moglielL0KPF ~ % 

It is now possible to invoke the Rest API call with the -n switch to cURL to use netrc file.
We can check NSX Manager FQDNs using NSX-T Data Center API with -n option as below:

curl -k -n -X GET https://172.25.251.31/api/v1/configs/management

Further information about the parameters you can use in file .netrc or how to use it in Windows can be found on this site.

That's it.

NSX-T host preparation - Upgrade VIB(s) "loadesx" is required

Issue

I was trying to perform NSX-T host preparation on a cluster (based on HPe Simplivity) composed of two Esxi hosts, when I received the following error message:

Failed to install software on host. Failed to install software on host. Simplivity.host.local : java.rmi.RemoteException: [InstallationError] Upgrade VIB(s) "loadesx" is required for the transaction. Please use a depot with a complete set of ESXi VIBs. Please refer to the log file for more details.

Solution

After investigating, I don't actually find the installed VIB...

# esxcli software vib list | grep load

I check the profile on the ESXi host ...

# esxcli software profile get

The current update was done with custom bundles,

The customer confirms that during the update phase, he skipped the installation because otherwise he would not have been able to update the drivers.

I then asked the customer to retrieve the Offline Bundle package used for the update.

I copied the same Offline Bundle used for the upgrade into a shared folder by the cluster hosts.

I checked the Offline Bundle profile ...

# esxcli software sources profile list -d /vmfs/volumes/SVT-VDI/Temp/HPe/Q8A57-11137_hpe-esxi7.0u3c-19193900-703.0.0.10.8.1-3-offline-bundle.zip

... and then the contents of the VIBs, to verify that was present "loadesx" ...

# esxcli software sources profile get -d /vmfs/volumes/SVT-VDI/Temp/HPe/Q8A57-11137_hpe-esxi7.0u3c-19193900-703.0.0.10.8.1-3-offline-bundle.zip -p HPE-ESXi-7.0-Update3c-19193900-customized

Verified the presence, I proceed with the update of the profile in this way:

# esxcli software profile update -d /vmfs/volumes/SVT-VDI/Temp/HPe/Q8A57-11137_hpe-esxi7.0u3c-19193900-703.0.0.10.8.1-3-offline-bundle.zip -p HPE-ESXi-7.0-Update3c-19193900-customized

As we can see above, there are a number of packages that have been installed/updated including “loadesx”.

Since a reboot is required, let's proceed with rebooting the ESXi host.

Post Reboot we verify that the module has been properly loaded ...

# esxcli software vib list | grep load

Back to NSX-T UI

Click on Install Failed of the host we just updated, then VIEW ERRORS

Select the error message and click RESOLVE

Click RESOLVE again.

I check the progress of the installation process ...

I also check via command line ...

# esxcli software vib list | grep -i nsx

Verified that the NSX-T packages have been correctly installed on the ESXi host (NSX Configuration: Success), and the status of the host in NSX-T is UP... I proceed to perform the same tasks with the next host.

Now, all hosts are UP and running.

That's it.

vRA 8.5.1 REST Api calls - API Authentication

Issue

I need a quick guide (step-by-step) on how to authenticate to vRA 8.5.1 via REST API calls.

Solution

To do that, I found a VMware well documented guide "vRealize Automation 8.5 API Programming Guide"

First of all, we need to know that the process to obtain the access token is different depending upon the vRealize Automation version.
In our case, we need to get the token used to authenticate our session, we use the Identity Service API to get an API token. Then we use the API token as input to the IaaS API to get an access token.

Let's see below the steps on how to do it, using Postman and Curl:

1. Open Postman.
   
   
2. Execute the following REST API call:
   
   

   URL: https://<vRA-FQDN>/csp/gateway/am/api/login?access_token
   Method: POST
   Headers: 'Content-Type: application/json'
   Body: {
   	"username": "username",
   	"password": "password"
   }

   NOTE: If you don't need to authenticate locally (as in our case) replace the above username with your own username in the form username@domain.
   
3. Take note of the refresh_token.
   
4. Execute the following REST API call:
   
   

   URL: https://<vRA-FQDN>/iaas/api/login
   Method: POST
   Headers: 'Content-Type: application/json'
   Body: {
   	"refreshToken": "api_token"
   }

   ... and as output you will receive the token to be used for subsequent vRA queries.
   

Let's see the same procedure, this time using Curl:

1. Open a session Terminal with both command, curl and jq already installed.
   
   
2. Assign values to the variables for the hostname of our vRealize Automation appliance, our user name, and password.

   url='https://<vRA-FQDN>'
   username='username'
   password='password'

3. Execute the following curl command to retrive the API token:
   

   api_token=`curl -k -X POST \
     "$url/csp/gateway/am/api/login?access_token" \
     -H 'Content-Type: application/json' \
     -H 'Accept: application/json' \
     -d '{
     "username": "'$username'",
     "password": "'$password'"
   }' | jq -r .refresh_token`

   
4. With the API token assigned, execute the following curl command to retrieve the Access token:
   

   access_token=`curl -k -X POST \
     "$url/iaas/api/login" \
     -H 'Content-Type: application/json' \
     -H 'Accept: application/json' \
     -d '{
     "refreshToken": "'$api_token'"
   }' | jq -r .token`

   
   Note: After 25 minutes of inactivity, the access token times out and we must request it again.
   
   
5. We can now try to obtain more information such as the Organization ID, using the Access Token, by executing the command ...
   

   curl -k -X GET "$url/csp/gateway/am/api/loggedin/user/orgs" -H "csp-auth-token: $access_token"

   

That's it.

"log disk exhaustion" warning on vCenter due to vmafdd.log file not compressed

Issue

Everything start with "log disk exhaustion" warning on vCenter. This warning comes out when the /storage/log partition reach out 80% ot its space.

Log in vCenter via SSH and checking the disk space with following command:

# df -h

I remove some logs following the " KB83070 - How to clear space on VCSA /storage/log partition"
Then, checking storage logs for disk utilization performing the following command:

# du -a /storage/log | sort -n -r | head -n 20

... you find that the vmafdd.log is extremely large.

Solution

This happens, as the KB83238 says, because the logrotate.d configuration for the vmafd service expects the logs to be located in the following path "/var/log/vmware/vmafdd/vmafdd.log" instead of the current path "/var/log/vmware/vmafd/vmafdd.log".

# cat /etc/logrotate.d/vmware-vmafd.lr

Checking the vmafdd registry we can see the incorrect path:

# /opt/likewise/bin/lwregshell list_values "[HKEY_THIS_MACHINE\Services\vmafd\Parameters]"

Before to do any changes is a good practice to take a snapshot.

Taken the snapshot, access the vCenter in SSH again ... and is now possible to update the registry to match the log rotate configuration path, using the following command:

# /opt/likewise/bin/lwregshell set_value "[HKEY_THIS_MACHINE\Services\vmafd\Parameters]" "LogFile" "/var/log/vmware/vmafdd/vmafdd.log"

... then perform the following command to verify that the change as been applied:

# /opt/likewise/bin/lwregshell list_values "[HKEY_THIS_MACHINE\Services\vmafd\Parameters]"

Restart all services:

# service-control --stop --all && service-control --start --all

If we look now inside the folder "/var/log/vmware/vmafdd/" we can see the file "vmafdd.log" ...

... than we can remove the previous huge "vmafdd.log" file ...

checking again the /storage/log space, we don't see anymore the huge "vmafdd.log" file.

# du -a /storage/log | sort -n -r | head -n 20

Seeing the occupation of the entire disk, we can see that we have freed up enough space.

# df -h

That's it.

How to change the root password of ESXi hosts managed by VMware Cloud Foundation (VCF)

Issue

I need to change manually the root password of the ESXi hosts managed by VMware Cloud Foundation.

Solution

It is possible to manually change the password for the ESXi host root account (unlike password rotation, which generates a randomized password) managed by the VMware Cloud Foundation infrastructure, by logging in directly from the SDDC manager.

Log in to the SDDC Manager UI as a user with the ADMIN role.
From the navigation pane, select Administration (1) > Security (2) > Password Management (3), and be sure that ESXi (4) is selected.

Click on the ⋮ (5) > UPDATE (6)

Provide new password ...

... then Click UPDATE

Wait untill the credentioal are properly update on the host ...

... verify the Date in Last Modified field if properly reflect the changes

Repeat same procedure describe above for each host ESXi you want change the password.

Try logging into the host to verify the change.

Insert the new password.

Official documentation on how Manually Update Passwords on VCF is available at this link.

That's it.

OpenVPN Access Server on Photon OS (with docker)

Issue

How to build a fast and easy VPN solution for a secure connection to the own infrastructure.

Solution

As a solution I opted for VMware Photon OS, docker and took advantage of this "linuxserver/openvpn-as" project based on OpenVPN.

In this quick guide I assume that Photon OS (minimal) is already installed and properly configured (such as IP address, DNS, NTP server and so on). In my case Photon OS i configured with 1 vCPU, 2 GB of RAM and 16GB of Disk (even if for the purpose of this scope 8 GB are enough).

First thing to do, we check if docker is installed, if not we install it or start the service and enable it to start when the VM starts.

# systemctl status docker

# systemctl start docker

# systemctl enable docker

I then, created a script to automate the installation and running of the container.

The outcome of the script is the execution of the container, and we will be asked to enter a new password for the default admin account.
Below the script:

#/bin/bash
#
# Initial setup script (all-in-one) to access OpenVPN Web UI

# Install the package called OpenVPN access server
# Download and install
docker pull linuxserver/openvpn-as:latest

# Let's create a new docker container called openvpn-as with the following:
#  -v /home/docker/openvpn-as/config: /config - Sets the directory to store tehconfig files.
#  --restart=always - Ensures the container always starts on system boot. You can opt not to add this agument if you don't want to container to restart
# -e PGID=1001 -e PUID=1001 - Sets the user ID to eliminate permission issues between the host server and the container.
# -e TZ=Europe/Rome - Set Time Zone
# --net=host --privileged - Dictates how OpenVPN Access Server runs in the container.
docker create --name=openvpn-as --restart=always -v /home/docker/openvon-as/config:/config -e INTERFACE=eth0 -e PGID=1001 -e PUID=1001 -e TZ=Europe/Rome --net=host --privileged linuxserver/openvpn-as

# Start the container
docker start openvpn-as

# Get access to docker and change the default admin password
docker exec -it $(docker ps | grep openvpn-as | awk '{print $1}') /usr/bin/passwd admin
 

We give execute permissions to the script.

# chmod +x OpenVPN_initial_setup.sh

Let's execute it ...

# ./OpenVPN_initial_setup.sh

... and at the prompt enter the new password

Let's verify that the container is running correctly

# docker ps

Let's connect to the local IP address (https://local-IP:943/admin/) for a first configuration ... providing the username admin and the previous setted password.

We accept the terms of use

Once logged in, first of all let's set up in Configuration > Network Settings > Hostname or IP Address ... insert the public IP or the FQDN. Then, we click on Save Settings to save the configuration.

We enable Google Authenticator Multi-Factor Authentication in the menu Authentication > General ... and saving the settings.

We then enabling and saving the "Require user permissions record for VPN access" in User Management > User Permissions.

We create a new Profile in User Profiles and the we click on New Profile.

We download and save locally on its own computer the .ovpn file, that will be used later for the VPN connection.

Client configuration part

Configure one of the following Apps on your mobile device, Google Authenticator, Microsoft Authenticator, FortiToken. I have verified that the 2FA system works with all of the above authenticator apps.

Open the web client at the VPN endpoint address (https://Public-IP:9443/) and, once logged in with the admin account, scan the QR Code with the newly installed App.

Insert the 6-digit code and download the right VPN client for its own OS/device.

Once installed, double click on the previous .ovpn file downloaded to import the VPN configuration.

Click on Connect ...

Provide username and password ...

... insert the 6-digits of the 2FA ...

We can now able to reach the remote site in a safe mode.

Conclusion

OpenVPN Access server is very simple to install and configure. However, there are some limitations due to the free license (only two connections available), the "linuxserver/openvpn-as" project is deprecated and no longer maintained, consequently the latest available version of OpenVPN is version 2.9.0 (currently version available for download is 2.11.3).

I saw that there are also other interesting project like Wireguard, not yet available as a container for docker but available as a native package for Photon OS.
Must try as soon as possible.

That's it.