Change VMNIC name to fit NSX-T Transport Node Profile

Issue
During the design phase of an NSX-T Datacenter implementation, there are a lot of Design Desicions to be addressed and taken, such as the size of the appliances, how many TZ to create, how many Edges (sizing, Active/Active, Active/Standby) .... and so on.

In this design decision I found myself having to consider the following:

• The customer have many ESXi host clusters to configure with NSX-T where the vmnics that should be used for the creation of the N-VDS are not homogeneous (at the naming convention level). This means, for example, that in a Host-A the vmnic to be used could be named vmnic4 while in a Host-B the vmnic to be associated to the N-VDS could be vmnic9 and so on for other hosts.
  
  
• Second consideration, we are operating in a brown-field environment, where VMs are already up and running.
  
  

Well, not having homogeneous vmnic and given the objective need to create a single Transport Node Profile (also applicable to multiple clusters) it was decided to rename the vmnic that will be used/associated with the specific N-VDS in vmnic100 (as showed from the below picture).

As implication for sure we will have to configure each ESXi hosts of the cluster, putting it in maintenance mode (so it to evacuate all running VMs away), rename the vmnic o the vmnics and then restart the system.

Solution
In order to rename/renumbering the vmnics, I found a link to KB article 2091560 that describe how to do this with vSphere 6.x!

So, in our case we want to change the aliases assigned by the ESXi host, for example from vmnic4 to vmnic100, not using Host Profiles but by command line.

Onces gotten the SSH access to the ESXi, from the shell, run the command to see the current assignment of aliases to device locations:

localcli --plugin-dir /usr/lib/vmware/esxcli/int/ deviceInternal alias list

Output

When a nic is controlled by a native driver, then there are actually two aliases associated with the device: a pci alias for the pci device and a logical alias for the uplink logical device.
For each alias that you want to reassign, use the command:

localcli --plugin-dir /usr/lib/vmware/esxcli/int/ deviceInternal alias store --bus-type pci --alias vmnicN --bus-address B

When the logical alias is present, then both the pci alias and logical alias need to be renamed:

localcli --plugin-dir /usr/lib/vmware/esxcli/int/ deviceInternal alias store --bus-type logical --alias vmnicN --bus-address B

where the vmnicN and B are the new assignment that you want to make. For instance, in my case the command is ...

localcli --plugin-dir /usr/lib/vmware/esxcli/int/ deviceInternal alias store --bus-type pci --alias vmnic100 --bus-address s00000002.00

In order the changes take effect it is necessary to perform a clean shut down and reboot the system.

That's it.

NIC network Stats

Issue
Today I face out a singular issue, concerning networks micro disconnection and the needs verify information on the physical NICs regarding packet drops and so on ..

Solution
To do that, I wrote a powershell script, available below.
It is enough just replace the variables<VCENTER_FQDN_or_IP> with it's own vCenter (FQDN or IP) references and <CLUSTER_NAME> with the name of the Cluster where the ESXi Hosts are placed.

Connect-VIServer -Server <VCENTER_FQDN_or_IP> 

$hosts = Get-Cluster <CLUSTER_NAME> |Get-VMHost
$Report = @()
foreach($ESXi in $hosts){
  $esxcli = get-vmhost $ESXi | Get-EsxCli
  $nics = $esxcli.network.nic.list.Invoke()
  foreach($pNic in $nics){
               $tempModule=@{}
               $tempModule.Name = $ESXi.name
               $tempModule.NIC_Name =  $pNic.Name
                $tempModule.NIC_Link =  $pNic.Link
                $tempModule.NIC_LinkStatus =  $pNic.LinkStatus
                $tempModule.NIC_Driver =  $pNic.Driver
                $tempModule.NIC_Duplex =  $pNic.Duplex
                $tempModule.NIC_MACAddress =  $pNic.MACAddress
                $tempModule.NIC_MTU =  $pNic.MTU
                $tempModule.NIC_Speed =  $pNic.Speed
 
                $vmnicstats = $esxcli.network.nic.stats.get($pNIC.Name)
                $tempModule.vmnic_Broadcastpacketsreceived =  $vmnicstats.Broadcastpacketsreceived
                $tempModule.vmnic_Broadcastpacketssent =  $vmnicstats.Broadcastpacketssent
                $tempModule.vmnic_Bytesreceived =  $vmnicstats.Bytesreceived
                $tempModule.vmnic_Bytessent =  $vmnicstats.Bytessent
                $tempModule.vmnic_Multicastpacketsreceived =  $vmnicstats.Multicastpacketsreceived
                $tempModule.vmnic_Multicastpacketssent =  $vmnicstats.Multicastpacketssent
                $tempModule.vmnic_Packetsreceived =  $vmnicstats.Packetsreceived
                $tempModule.vmnic_Packetssent =  $vmnicstats.Packetssent
                $tempModule.vmnic_ReceiveCRCerrors =  $vmnicstats.ReceiveCRCerrors
                $tempModule.vmnic_ReceiveFIFOerrors =  $vmnicstats.ReceiveFIFOerrors
                $tempModule.vmnic_Receiveframeerrors =  $vmnicstats.Receiveframeerrors
                $tempModule.vmnic_Receivelengtherrors =  $vmnicstats.Receivelengtherrors
                $tempModule.vmnic_Receivedmissederrors =  $vmnicstats.Receivedmissederrors
                $tempModule.vmnic_Receiveovererrors =  $vmnicstats.Receiveovererrors
                $tempModule.vmnic_Receivepacketsdropped =  $vmnicstats.Receivepacketsdropped
                $tempModule.vmnic_Totalreceiveerrors =  $vmnicstats.Totalreceiveerrors
                $tempModule.vmnic_Totaltransmiterrors =  $vmnicstats.Totaltransmiterrors
                $tempModule.vmnic_TransmitFIFOerrors =  $vmnicstats.TransmitFIFOerrors
                $tempModule.vmnic_Transmitabortederrors =  $vmnicstats.Transmitabortederrors
                $tempModule.vmnic_Transmitcarriererrors =  $vmnicstats.Transmitcarriererrors
                $tempModule.vmnic_Transmitheartbeaterrors =  $vmnicstats.Transmitheartbeaterrors
                $tempModule.vmnic_Transmitpacketsdropped =  $vmnicstats.Transmitpacketsdropped
                $tempModule.vmnic_Transmitwindowerrors =  $vmnicstats.Transmitwindowerrors
               
                $temp = New-Object -TypeName PSObject -Property $tempModule
               $Report += $temp
  }
}
$Report | Select * | Sort-Object -Property Name | Out-Gridview

That's it.

NSX-T - How to retrieve TEP IPs by PowerShell

Issue
Today I needed to quickly retrieve all the TEP IPs assigned by NSX-T (from an IP Pool) to the Hosts of a specific ESXi Cluster.

Solution
To do that, I wrote a powershell script, available below.
It is enough just replace the variables<VCENTER_FQDN_or_IP> with it's own vCenter (FQDN or IP) references and <CLUSTER_NAME> with the name of the Cluster where the ESXi Hosts are placed.

Connect-VIServer -Server <VCENTER_FQDN_or_IP> 

$hosts = Get-Cluster <CLUSTER_NAME> |Get-VMHost
$Report = @()
foreach($ESXi in $hosts){
  $tempModule=@{}
  $tempModule.Name = $ESXi.name
  $esxcli = get-vmhost $ESXi | Get-EsxCli
  $IPApp = $esxcli.network.ip.interface.ipv4.get("","vxlan")| Select @{N="IPv4Address";E={$_.IPv4Address}}
  $tempModule.IP = $($IPApp.IPv4Address)
  $temp = New-Object -TypeName PSObject -Property $tempModule
  $Report += $temp
} 
$Report | Select Name,IP | Sort-Object -Property Name | Out-Gridview

The script has been successfully tested on a production environment and on VMware' HOL as it is possible to see below.

That's it.

NSX-T Manager: The account specified has been locked

Issue
Today, trying to access via UI to an NSX-T Manager appliance (version 2.5.0) with admin user; I received the following error message:

Your login attempt was not successful.
The username/password combination is incorrect or the account specified has been locked.

.. and even via SSH I had no possibility to log in with the same user.

Solution
I logged in via SSH with the root user and I reset the admin user password by running the following commands:

• root@nsxmgr-01:/# su admin
  
  
  
  
  
• root@nsxmgr-01> set user admin
  Current password: type OLD password here
  New password: enter new password
  Confirm new password: re-type new password to confim here
  
  
  
  
  

Now, trying to get access via UI with admin user again ...

... we are able to get in.

Now, it is also possible to Modifying the Default Admin Password Expiration providing from CLI console the following commands:

• ... to change the password expiration period for admin user (between 1 and 9999 days).
  
  

  nsxcli> set user admin password-expiration <1 - 9999>

  
  
  
• ... to remove the password expiration policy on any NSX Manager simply typing.
  
  

  nsxcli> clear user audit password-expiration

  
  

That's it.

NSX-T - Error to fetch SSH Fingerprint

Issue
Recently I needed to set up backups of a NSX-T Manager for a one time backup. I have performed the procedures shown in this link: I logged in with admin privileges to an NSX Manager at https://<nsx-manager-ip-address>. Selected System > Backup & Restore then Edit (in the upper right of the page to configure backups) and filled up all forms except the once regarding the SSH fingerprint of the server that stores the backups, because as indicated, I could leave it blank.
Doing so, I received the following error message:

Error while fetching fingerprint of fileserver Algorithm negotiation fail. Possibly, there is no ecdsa and ed25519 support for public keys (Error code: 29259)
- Please retry or provide the fingerprint.

By carefully reading the documentation about the Prerequisites, at the link says:

Verify that you have the SSH fingerprint of the backup file server. Only an SHA256 hashed ECDSA key is accepted as a fingerprint. See Find the SSH Fingerprint of a Remote Server.

Solution
In case was not practicable to follow the instructions suggested in the link, is it possible to get the SSH Fingerprint of a Remote Server through a REST API request. Info about NSX-T Data Center API Guide regarding NSX-T Data Center 2.5.0 can be found here.

So, to fetch the SHA256 fingerprint of the SFTP backup target, in my case, I used the following CURL command:

curl --user admin -H 'Content-Type: application/json' --request POST  https://<nsx-manager-ip-address>/api/v1/cluster/backups?action=retrieve_ssh_fingerprint -k -d '{ "server":"<Target-Backup-IP-Address>","port":22}'

The NSX-T Manager API responds with the SHA256 fingerprint of the SFTP server:

A very great post on configuring and troubleshooting NSX-T SFTP backups, wrote by Gary Hills is available here.

That's it.

NSX - Enable and Disable Firewall are greyed out

Issue
Today I face out with a singular issue, whole clusters show me Communication Channels DOWN and Firewall status ERROR.

Checking inside ACTIONS in Host Preparation tab I saw that Resolve, Enable and Disable Firewall was greyed out as well even if EAM was UP and running Normal.

Looking inside logs, I haven't found something interesting ... NSX Controllers were UP and Running Normal, NSX Cluster was UP, communication between various components of the NSX environment has been tested successfully with the ping ... so, everything seemed to work properly.

I tried to create new objects (examples Logical Switches) and was created and synced correctly on every hosts.

Then alternatively I tried to reboot both NSX Manager and vCenter ... without success. Communication channel were still DOWN and Firewall status still in ERROR.

I also tried to restart the on Control Plane Agent (netcpad service) on some hosts, randomly chosen, but without success .... communication between NSX's objects works fine but at GUI level I continue to see the error.

Solution
The fastest way I found to resolve the issue was to put into maintenance mode the ESXi host and then, one by one, reboot every single host of the whole cluster.
After the reboot of the ESXi host, the Communication channel and the Firewall status was properly recognized.

After all hosts have been restarted, the grayed out option in ACTIONS became available.

That's it.

What's new in NSX-T Data Center 3.0

What's new in NSX-T Data Center 3.0
NSX-T Data Center 3.0 version has been released today, with this release VMware introduce a variety of new features to provide new functionality for virtualized networking and security for private, public, and multi-clouds. Highlights include the following focus areas and new features:

• Cloud-scale Networking: NSX Federation
  
• Intrinsic Security: Distributed IDS, Micro-Segmentation for Windows Physical Servers, Time-based Firewall Rules, and a feature preview of URL Analysis
  
• Modern Apps Networking: NSX-T for vSphere with Kubernetes, container networking and security enhancements
  
• Next-Gen Telco Cloud: L3 EVPN for VM mobility, accelerated data plane performance, NAT64, IPv6 support for containers, E-W service chaining for NFV
  

Some of the new features will be discussed down here, most o them and enhancements are available in the NSX-T Data Center 3.0.0 release Notes.

Before to drill down into the new features I would like to remember that VMware NSX-T provide functionality, all the way from layer two layer seven on the stacks of connectivity. They have logical layer two layer three across physical bare metal and container workloads. They also support layer to VPN and layer three base route and policy VPN. and so on .. the are continuing to add more Layer seven capabilities and more Context Aware inside the Firewall engine.
Below a simple picture of a NSX-T Single Heterogeneous SDN Platform before NSX-T Data Center version 3.0

Release of the new features concern Full-stack Networking and Security Virtualization ...

... let's have a look below of some new features:

NSX Federation
NSX-T 3.0 introduces the ability to federate multiple on-premises data centers through a single pane of glass, called Global Manager (GM). GM provides a graphical user interface and an intent-based REST API endpoint. Through the GM, you can configure consistent security policies across multiple locations and stretched networking objects: Tier0 and Tier1 gateways and segments.

VRF Lite in NSX-T 3.0
Now is possible to have multi-tenant data plane isolation through Virtual Routing Forwarding (VRF) in Tier-0 gateway. VRF has its own isolated routing table, uplinks, NAT and gateway firewall services; the solution can scale up to 100 VRFs per Tier0 Gateway.

Converged VDS
With new vSphere version 7.0 is possible to run NSX straight on VDS 7.0 that's part of it, giving the possibility to use the existing VDS dvPortGroups for NSX switching.

Actually this features is for GreenField only, where you start with a fresh deploy. Where we already have a previous NSX-T installation we can continue to leverage on N-VDS. However, it is recommended that new deployments of NSX and vSphere take advantage of this close integration and start to move toward the use of NSX-T on VDS. The N-VDS NSX-T host switch will be deprecated in a future release. Going forward, the plan is to converge NSX-T and ESXi host switches. The N-VDS remains the switch on the KVM, NSX-T Edge Nodes, native public cloud NSX agents and for bare metal workloads.


Some UI enahcement
In previous NSX-T 2.4 version was introduced the whole policy UI mode and Advanced mode. So now we have a switch to seamlessly switch between the policy and the manager view to configure objects

New Getting Started Wizard to prepare clusters for micro-segmentation has been introduced.

Networking topology visualizations has been introduced. This feature provide graphical visualization of the network infrastructure with the possibility to export the topology in PDF.
With this new features, we can see containers attached if we have containers networking in our views; it is also possible to drill deep inside a particular topology and look the routing table, the forwarding tables etc.

New Alarm Dashboard for managing alarms.

Distributed Intrusion Detection System (D-IDS)
Challenges with the traditional Data Center IDS/IPS (picture below) most of the time deployed in a physical IDS/IPS appliance in large clusters that are deployed behind firewall with their own management console (when IDS/IPS in not integrated into firewall) and traffic is hairpin across the physical network.

Introducing in NSX Platform the capability of Distributed Intrusion Detection as a part of the platform's Threat & Vulnerability Detection capabilities. This feature allows you to enable intrusion detection capabilities within the hypervisor to detect vulnerable network traffic. This distributed mechanism can be enabled on a per VM and per vNIC of a VM basis with granular rule inspection. As part of this feature set, the NSX Manager is able to download the latest signature packs from the NSX Signature Service. This keeps the NSX Distributed IDS updated with the latest threat signatures in the environment.

Service Insertion and Guest Introspection
E-W Service Chaining for NFV-SFC at the Edge - The ability to chain multiple services was earlier available only to distributed traffic but is now available for edge traffic. The East-West service chains can now also be extended to redirect edge traffic.
Disable cloning of NSX Service VMs - Cloning of Service VMs is now prevented from the vSphere Client to prevent malfunctioning of the VMs.

Container Networking and Security
Container Inventory & Monitoring in User Interface - Container cluster, Namespace, Network Policy, Pod level inventory can be visualized in the NSX-T User Interface. Visibility is also provided into co-relation of Container/K8 objects to NSX-T logical objects.
IPAM Flexibility - The NSX Policy IP Block API has been enhanced to carve out IP subnets of variable sizes. This functionality helps the NSX Container Plugin carve out variable size subnets for Namespaces using Policy API.
NCP Component Health Monitoring - The NSX Container Plugin and related component health information like NCP Status, NSX Node Agent Status, NSX Hyperbus Agent Status can be monitored using the NSX Manager UI/API.

..... most of the features and enhancements are available in the NSX-T Data Center 3.0.0 release Notes

Enjoy!!

Notes: All information, texts and images contained in this article are owned by VMware.

VMs with active Snapshots

Issue
I need to get a list of VMs with active snapshots.

Solution
Below a very simple PowerCLI script that will return the list of snapshots that are in the environment.
First of all, after you have opened a powershell terminal (doesn't mind if Windows/Linux/MacOS), you need to connect to the vCenter Server, providing a proper User and Password:

Connect-VIServer -Server “IP/hostname/FQDN”

Once connected you can execute the following command:

Get-VM | Get-Snapshot | Select VM, Name, Created, @{N="SizeGB";E={[math]::round($_.SizeGB,4)}}

That's it.

VEEAM - Invalid Remote Certificate – Veeam Backup & Replication

Issue
Recently, following an update of the VMware infrastructure to the latest 6.7 P01 (build number 15160138), all backup jobs on some servers, already present in the inventory in stand alone mode, have failed with the error message:

"Disk and volumes discovery failed Error: The remote certificate is invalid according to the validation procedure."

.... as shown in the picture below.

As first attempt I went to the INVENTORY then in Standalone Hosts right click on the Host affected and tried to Refresh informations already kept on the Veeam Manager.....

after a while I faced out the message:

Failed to login to "" by SOAP, port 443, user "root", proxy srv: port:0
The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel.
The remote certificate is invalid according to the validation procedure.

Solution
Here is a quick fix for it.

• Open the Veeam Backup & Replication Console.
  
• Select Backup Infrastructure.
  
• Select Managed Servers.
  
  
  
  
  
• Right click on the Server with the issue and select Properties...
  
  
  
  
  
• In the Edit VMware Server screen Name.
  Leave DNS Name or IP address and Description unchanged clicking Next.
  
  
  
  
  
• You should not to change anything in the tab Credential as well and click Apply.
  
  
  
  
  
• You will get an error about the certificate and be asked if you want to continue anyway.
  
  
  
  
  Before continuing, check the certificate by clicking on View... and then OK and Continue
  
  
  
  
  
• Review summary information and continue the process saving the server configuration clicking Finish.
  
  
  
  
  
  
  
  
• Go back on the Standalone Hosts in the INVENTORY and select the host affected; now you should be able to see again the list of the VMs ....
  
  
  
  
  
• Let's try to re-run the backup job by selecting "Retry" or "Start"
  
  
  
  
  
• This should clear the error and the backup job should run now as before!!!
  
  
  
  
  

That's it!

DataDomain - Upgrade by command line

Issue
Today I had to update the firmware of a DataDomain 3300 to the latest version 7.1.0.10-646016, however, there is an issue that prevents you from updating from the GUI ...

Solution

1. First of all we should download locally the firmware .rpm file from the DellEMC support site. In our case we have to download the 7.1.0.10-646016.rpm file.
   
   
2. Then we must copy the .rpm file into a specific path "/ddr/var/releases".
   
   scp 7.1.0.10-646016.rpm sysadmin@dd3300:/ddr/var/releases/
   
   
3. Got access the Data Domain appliance via SSH, we proceed giving the pre-check command ...
   
   sysadmin@dd3300# system upgrade precheck 7.1.0.10-646016.rpm
   
   
4. If no issues were found, we go ahead starting the upgrade ...
   
   sysadmin@dd3300# system upgrade start 7.1.0.10-646016.rpm
   
   and then typing "yes" and enter.
   
   
   
   
   
5. System will reboot automatically...
   
   
   
   
   
6. When the appliance become reachable again, gained access by ssh, we can check the status of the upgrade with the command ..
   
   sysadmin@dd3300# system upgrade status
   
   
   
   
   To be sure that the upgrade is completed successfully, we have to wait the message (as shown below in the picture):
   Current Upgrade Status: DD OS upgrade Succeeded
   
   
   
   
   
   
7. We can now get access to the DataDomain Appliance by the GUI
   
   
   

That's it.

Notes: The information and some links provided in this article may be confidential and owned by DellEMC, accessible only after authentication on the DellEMC website.
Further information available at https://emcservice.force.com/CustomersPartners/kA23a000000GI38CAG