[SSPi] - Backup Process fail

Issue

The SSPI backup process fails and displays the following error message: “Management cluster backup failed. Please check logs for more details.” as shown below ..

management cluster backup failed, please check logs for more details.
Command '/usr/local/bin/clusterctl move --to-directory /image/br/sspi/mgmt-cluster/move/ --config /config/clusterctl/1/clusterctl-init.yaml --namespace ssp-rm-w01 --v 5' execution is failed. Output: Using configuration file="/config/clusterctl/1/clusterctl-init.yaml" 
.....
.....
ExecuteC github.com/spf13/cobra@v1.9.1/command.go:1148 github.com/spf13/cobra.(*Command).
Execute github.com/spf13/cobra@v1.9.1/command.go:1071 sigs.k8s.io/cluster-api/cmd/clusterctl/cmd.Execute sigs.k8s.io/cluster-api/cmd/clusterctl/cmd/root.go:113 main.main sigs.k8s.io/cluster-api/cmd/clusterctl/main.go:27 runtime.main runtime/proc.go:272 runtime.goexit runtime/asm_amd64.s:1700 , 
Error:exit status 1 

Solution

Disclaimer: Some of the procedures described below may not be officially supported by VMware. Use it at your own risk. I suggest to contact broadcom support to troubleshoot this issue.

Googling around, I found the KB that indicates exactly my error "Management cluster backup failed. Please check logs for more details.", and which tells me to contact support.

I tried to investigate by connecting via SSH to the SSPI and typing ...

df -h

The root "/" directory is 100% used, which is not good at all, regardless.

In my cases, I emptied the root folder by compressing the "log" folder (13 GB) (which was under the "/root" directory) previously used for troubleshooting.
Then I tried to manually launch the backup and it worked.

That's it.

[Quick TIP - VCF 9.0] - Operation not allowed by the credentials remediate in progress

Issue

I am in the situation where I have to change some settings to the current VCF 9.0.2.0 backup configuration ..... and the "EDIT" button is grayed out and I get the following message:

This operation is not allowed because the system lock is held by the Credentials remediate operation operation in progress.

Solution

Connect to the SDDC Manager

Go to ... Security > Password Management > Cancel (1) and confirm by pressing YES (2)

Switch back to the VCF Operations UI and the buttons are clickable again.

That's it.

[SSP 5.1.0] vDefend SSP Backup Failures: Stuck at 60%

Issue

If you manage a VMware vDefend Security Services Platform (SSP) environment, you know how critical automated backups are. Recently, I ran into a frustrating issue where scheduled backups started failing mysteriously. What initially seemed like a simple user and misconfigured settings issue, “network timeout” turned out to be a bizarre problem involving lost encryption keys and crashing containers.
Looking on the SSP UI at the System > Backup and Restore dashboard, I noticed a long list of failed backup jobs. Hovering over the failure status yielded a very generic message: "operation timed out, please contact administrator".
I tried manually forcing the backup, but it seems to stuck at 60% and then time out after a while.

Solution

My first instinct was to check the SFTP credentials, re-type the password, SAVE and seemed fine.

Googling around, I found this interesting link "Backup and Restore goes into Failed State" which helped me in the final resolution.

Since the UI wasn't giving me enough technical details, it was time to drop into the CLI. I logged into the SSPI appliance via SSH as sysadmin to check what was happening at the Kubernetes pod level.

I ran a quick ...

kubectl get pods -n nsxi-platform | grep backup

... and immediately spotted the problem. The backup job pods weren't just timing out; they were actively crashing, throwing Error and OOMKilled statuses.

To understand why the pods were in this state ("OOMKilled", "Error"), I checked the logs of both pods....

k logs job-backup-4fv4k-5gq5s -n nsxi-platform

Scrolling through the container logs, I could see ... the system passed a malformed argument to the OpenSSL encryption command.
The log explicitly stated:

Error executing command: [openssl enc -aes-256-cbc -salt -in ... -out ... -k ******]. 
Error: fork/exec /usr/bin/bash: argument list too long.

Scrolling through the container logs, I could see the backup job actually doing its work: generating the .obj and .dump files from the Postgres database and config files.
However, the process abruptly halted right after the ...

<INFO> ... Started encrypting backup bundle

... message.

Conclusion

How did I fix it? I re-entered the password and retyped the passphrase to ensure the openssl process received a valid string, allowing the container to properly encrypt the dump files and upload them to the SFTP server.

The mystery was solved. The "timeout" in the UI was just a side effect. The real issue was that the missing encryption key caused the openssl command to fail spectacularly (resulting in an argument list too long OS error), which crashed the container pod.

It appeared the platform had somehow lost track of the encryption key. The configuration was stuck in a weird state where trying to "Reuse prior passphrase" wasn't working.

Now starting the backup in manual mode, it completes successfully.

That's it.

NSX-T - Error to fetch SSH Fingerprint

Issue
Recently I needed to set up backups of a NSX-T Manager for a one time backup. I have performed the procedures shown in this link: I logged in with admin privileges to an NSX Manager at https://<nsx-manager-ip-address>. Selected System > Backup & Restore then Edit (in the upper right of the page to configure backups) and filled up all forms except the once regarding the SSH fingerprint of the server that stores the backups, because as indicated, I could leave it blank.
Doing so, I received the following error message:

Error while fetching fingerprint of fileserver Algorithm negotiation fail. Possibly, there is no ecdsa and ed25519 support for public keys (Error code: 29259)
- Please retry or provide the fingerprint.

By carefully reading the documentation about the Prerequisites, at the link says:

Verify that you have the SSH fingerprint of the backup file server. Only an SHA256 hashed ECDSA key is accepted as a fingerprint. See Find the SSH Fingerprint of a Remote Server.

Solution
In case was not practicable to follow the instructions suggested in the link, is it possible to get the SSH Fingerprint of a Remote Server through a REST API request. Info about NSX-T Data Center API Guide regarding NSX-T Data Center 2.5.0 can be found here.

So, to fetch the SHA256 fingerprint of the SFTP backup target, in my case, I used the following CURL command:

curl --user admin -H 'Content-Type: application/json' --request POST  https://<nsx-manager-ip-address>/api/v1/cluster/backups?action=retrieve_ssh_fingerprint -k -d '{ "server":"<Target-Backup-IP-Address>","port":22}'

The NSX-T Manager API responds with the SHA256 fingerprint of the SFTP server:

A very great post on configuring and troubleshooting NSX-T SFTP backups, wrote by Gary Hills is available here.

That's it.

VEEAM - Configure a NAS backup Job

New capabilities that redefine NAS data protection to deal with the world of massive file data, have been released with the new veeam v10 version.
Below I will covering step by step how to setup and configure on Veeam 10 to backup file from a NAS file share.
First of all let's see the official documentation available here, to understand which components must be present or added to the backup infrastructure and to understand "How File Share Backup Works".

Before to begin to adding an SMB file share to the inventory of the virtual infrastructure is possible to check prerequisites, limitations and the official procedures here.

To add an SMB file share, once opened Veeam Backup and Replication console proceed as follows:

• Open the Inventory view.
  
  Click on File Shares and then ADD FILE SHARE.
  
  
  
  
  
  
• Click SMB share (in my case I have a NAS that run a CIFS service)
  
  
  
  
  
  
• In the Shared folder field, specify the path to an SMB file share in the \\server\folder format.
  
  Select the This share requires access credentials check box and provide proper credentials to get full access control to shared folder.
  
  
  
  
  
  
• Clicking on Advanced the window shows several options.
  
  Leave the enabled the default option "Backup directly from the file share" clicking on OK and then click Next >.
  
  
  
  
  
  
• Click Choose... next to the File proxy field to select a file proxy.
  
  From the Cache repository drop-down list, select a cache repository where temporary cache files must be stored.
  
  Use the Backup I/O control slider to define how fast all proxies can read data from the source file share.
  
  Click Apply to save the configured settings.
  
  
  
  
  
  
• Wait while required components are installed and configured, then click Next>.
  
  
  
  
  
  
• At the Summary step, reviewed the configuration, click Finish to finalize the setup and exit the wizard.
  
  
  
  
  
  

In the following steps we will see how to set up a backup job of the NAS file share content.

• Open the Inventory view.
  
  Right click on SMB file share select Add to backup job and then click on New job...
  
  
  
  
  
  
• In the Name field, specify the name of the backup job and click Next >.
  
  
  
  
  
  
• Select File or folder and click Add...
  
  
  
  
  
  
• Explore Folders, select which ones you prefer and click OK.
  
  
  
  
  
  
• In Advanced is possible to create extension masks to Include or Exclude files.
  
  
  
  
  
  
• Select the desired Backup repository.
  
  Use the Keep all file versions for the last field to define short-term retention policy: specify how long copies of all recent file versions must be kept in the short-term backup repository.
  
  Click Next >.
  
  
  
  
  
  
• Add a Secondary Target otherwise click Next >.
  
  
  
  
  
  
• Schedule the job to run automatically.
  
  
  
  
  
  
• Review details and click Finish.
  
  
  
  
  
  
• The job run as expected.
  
  
  
  
  
  
  

That's it.

Further info: v10: Seriously Powerful NAS Backup at Scale