VMware 6.7 PSC Decommission: Failed to get the PSC thumbprint

As part of our recent VMware 6.7 upgrade we where migrating from external PSC to embedded PSC.

All went fine until we tried to decommission the old PSC’s, when trying to view the thumbprint we got the below error:

Failed to get the PSC thumbprint. Ensure PSC port is correct.

PSCWith out getting the thumbprint you can continue.

We tested accessing the PSC using web browser,

using openssl from the vCenter appliance

openssl s_client -connect PSC01.domain.local:443

We also tested access using telnet to confirm if port 443 was open

curl -v telnet://PSC01.domain.local:443

All test came back fine and there was no ports being blocked.

In the end the issue was cause by using the short name for the vCenter servers https://VCSA/UI

When we changed to the full FQDN, https://VCSA.domain.local/UI this fixed the issue and we where able to complete the decommission. The issue looks to be related to the SAN name on the cert only having the FQDN.


VMware ESXi 6.7 Upgrade: Missing dependencies VIBs Error

Recently we have been upgrading some VMware host from ESXi 6.0 to ESXi 6.7, We where applying the image using VMware update manager and a HPE custom ESXi image.

When applying the image we where getting incompatible warring and where not able to apply the image to upgrade ESXi on some hosts.

The issues was related to VIBS but they where not showing in the html 5 clientvibs1

To find the missing VIBS we ended up having to mount the ISO through HPE ILO and try a manual upgrade which did show the conflicting VIBS. vibs2

In our case the VIBS causing issue was the below.


The issue seem to be related to older hosts that where previously upgraded from ESXi 5.5.

Next we needed to find out if the VIBS where in use by either the storage or network adapters,

below is the VMware KB that explain how to do this.


To get the list of storage and network adapter use the esxcli commands

esxcli storage core adapter list

esxcli network nic list

To get check the VIBS version we can use

esxcli software vib list | grep Mel

esxcli software vib list | grep scsi-lpfc820


Once we know the version numbers of the VIBS, we just need to confirm they are not used and if not used remove them.

If they where in use we would need to look at creating a custom image or wipe and reload the ESXi host.

We use esxcli to view if the drivers are in use and what version each is

esxcli system module list | grep lpfc820, esxcli network nic get -n vmnic0


Once we confirm that none of the VIBS are required the last step is to remove each one. Below is the KB from VMware on removing VIBS.


There might be some VIBS that have dependency’s on others in our case the net-mlx4-core needed to be removed after the net-mlx4-en as it was dependent on it.

To remove we use esxcli software vib remove -n ‘vibname’


Reboot the ESXi host if required,

After the reboot, scan the host again from updates tab in VMware vSphere web client and it should now show as non-complaint and not incompatible. vibs6

The host should now upgrade as normal.

HPE Gen 10 SUM Integrated Smart Update Tools VMware

I was installing some new ESXi host using HPE Gen 10 blade servers and was getting a warning when trying to update the firmware using HPE SPP (Service pack for proliant). The issue is due to HPE changing from installing updates directly to the ESXi OS level and instead using HPE ILO.


If iSUT is not installed the update will be staged on the host but wont install. To install, download iSUT from the HPE Support site. Below is the link to version 2.3.6 which was the version I used. ISUT_Tool

HPE15Once download and extracted we need to copy the file to the ESXi host, the easiest way to copy is by enabling SSH on the host and using WinSCP.     HPE04I created a folder called hpe_isut on the ESXi host to copy the file to. HPE05Once the files are copied over, use either SSH or ESXi Shell to install the tools. I used SSH with putty as it was easier.

To install use esxcli command, this is the command I used.

esxcli software vib install -d /hpe_sut/sut-esxi6.0-bundle- the install has completed, a restart is required to finish the install. After the reboot the next step is to set the iSUT mode there are 4 <OnDemand/AutoStage/AutoDeploy/AutodeployReboot>

I chose AutoDeploy, to set the mode use the below command.

sut –set mode=AutoDeployHPE09

Once this has completed, run the inventory again from the SPP and the warning should now be gone and the firmware and driver updates should now apply. HPE10


Install and Configure VMware NSX

Recently we have been looking to implement zero trust networking. One way to achieve this was to use physical firewall and multiple VLAN’s to break out traffic and restrict access to each VLAN this would take a long time to complete and is quite difficult to manage.

It would require adding between 30 to 60 additional VLAN to our physical servers and VMware and re assinging IP to each server which would cause a lot of downtime.

As an alternative to this I have been looking at VMware NSX to try achieve this same segmentation without the need to redesign the entire VMware networks.

NSX consists of multiple components under different planes like management, control, and data plane’s below is an image of the different plane’s. 

In the next set of posts I am going to go thorough install and configuring a basic NSX deployment. I will be setting this up in a Lab environment and will use nested ESXi and appliances.

It is recommended to have NSX installed on its own management cluster along with vCenter.

First step is to download the OVA for NSX current version is 6.4.4


below are the system requirments to deploy NSX

NSX Component Hard Drive Memory vCpu
NSX Manager 60 16 4
NSX Controller 20 4 4

NSX 6.4.4 is not supported on vSphere 5.5 below are the supported and recommed verison of vSphere to run NSX 6.4.4:

  • For vSphere 6.0:
    Supported: 6.0 Update 2, 6.0 Update 3
    Recommended: 6.0 Update 3. vSphere 6.0 Update 3 resolves the issue of duplicate VTEPs in ESXi hosts after rebooting vCenter server. SeeVMware Knowledge Base article 2144605 for more information.
  • For vSphere 6.5:
    Supported: 6.5a, 6.5 Update 1
    Recommended: 6.5 Update 1. vSphere 6.5 Update 1 resolves the issue of EAM failing with OutOfMemory. See VMware Knowledge Base Article 2135378 for more information.

Once the OVA is downloaded logon to vCenter right-click on datacenter and deploy OVF Template.


Select the location of OVANSX2

Give the appliance a nameNSX3

Select the Cluster that will run the applianceNSX4

Click next NSX5

Accept the licence agreement and click continueNSX6

Chose Thick ProvisionNSX7

Select the network that will be used for the management networkNSX8

The next screen is where all the customization will be setup

Appliance Password:


Network settings: management IP, subnet, gateway, DNS and NTP. Leave blank if  you want to use DHCP but its recommend to use static addressesNSX9NSX10

Once all setting are configured click next and confirm all settings on the last screen. Once finished the OVA should start to deploy. (Note that this failed the first time for me as I selected a host and there seems to be an issue with this in vCenter 6.7, once I selected the cluster the OVA deployed without issue)NSX11

Once the OVA had been deployed I decided to edit the memory size as I was running low on memory so I change it from 16Gb to 8Gb but for production this should be left at 16Gb.

After this you can connect using DNS name configured above or through the management IPNSX12

The last step in this post is to connect NSX to vCenter

Logon using  admin and the password specified in the config of the OVA

Click on Manage vCenter Registration NSX13

both the lookup and vCenter server connection will need to be configuredNSX15

Add vCenter server and user name / passwordNSX16NSX18

There will be a prompt to trust the vCenter certificate click yes to continueNSX17

Once configured both status should show as connectedNSX19

Open the vCenter web client and once logged on there should now be an addtional tab for Networking & Security. (At the time of this post this option is only available in the Flash version of the Web client not the HTML 5 version) 


In the next post we will start to configure the NSX and controllers.

Using VMware PowerCli Part 1

Since VMware 6.0 I have started to use VMware PowerCli module to automate task and checks that I do daily or for large task that would take a long time to do manually. I am going to go through installing PowerCli and some of the useful command and script that can be use to check VMware.

To install PowerCli there are some pre-req’s

OS Type
  • Windows Server 2012 R2
  • Windows Server 2008 R2 Service Pack 1
  • Windows 10
  • Windows 8.1
  • Windows 7 Service Pack 1
  • Windows PowerShell 3.0, 4.0, 5.0, or 5.1
  • .NET Framework 4.5, 4.5.x, 4.6, or 4.6.x

I would recommend installing the latest version of PowerShell which is currently 5.1

To check current version of PowerShell run $PSVersionTablePcli

To install the latest version install the latest Windows Management Framework 5.1 link to download page is below.


Step 1 is to install PowerCli

The old PowerCli was a PowerShell snap in and required downloading an exe to install, the new version is module based and can be installed directly from the PowerShell console. To install run the below command.

Install-Module VMware.PowerCliPcli1

If you need to update the module to a new release run

Update-Module VMWare.PowerCliPcli2

If a path is not specified the default location that the module files will be placed in is

C:\Program Files\WindowsPowerShell\Modules

Once the install has finished to verify that PowerCli is installed run the below commandPcli3

Step 2 is to connect to vCenter

To connect to vCenter open an elevated PowerShell console and import the VMware Module

Import-Module VMware.PowerCliPcli4

Connect-VIServer lab-VC vCenterServerPcli5

Once connected we can now start to run command against vCenter.

To get host information run



To find all VM’s that have snapshots over a certain date. I want to get all snapshots older than 1 day to change this just edit the $date variable.

$date = (Get-Date).AddDays(-1)
$Snapshot = get-vm | get-snapshot
$Snapshot | where {$_.Created -lt $date}Pcli7

To check datastores you can run



To get additional info you can do some math’s and use arrays to get % free space of the datastores. The below will get all datastores that have less than 25% free space.

Get-Datastore | select Name,@{N=”UsedSpaceGB”;E={[math]::Round(($_.CapacityGB),2)}},
@{N=”%Free”;E={[math]::Round(($_.FreeSpaceGB)/($_.CapacityGB)*100,2)}} |
where %Free -lt “25”

VMware 6.5 Migrate From vSS To vDS

Migrating from standard vSwitches to distributed vSwitches has a lot of advantages. I have listed a few below.

  1. Central management for all host in the vDS
  2. Uniform configuration for all hosts
  3. Easily add new port groups

The one major disadvanatage is if vCenter is down you cannot manage the vDS.

To use vDS you will need to have Enterprise or Enterprise Plus VMware licences.

To migrated from vSS to vDS go to networking in the VMware web client and right-click on vCenter server and go to distributed switch.


Give the vDS a suitable name


Select the version of ESXi you have in production if you select a newer version than you have running in vCenter the host will show as incompatibleVDS3

The defualt number of uplinks is 4. I only had two uplinks so I changed this to two.  VDS4VDS5

After the vSwitch has been created I usually add a port group for management and vMotion VMkernel networks.

To migrated from the standard switch right-click on the vDS and go to Add and Manage Hosts.


Select add hosts. VDS8

Select the host that you want to add to the vDSVDS9

The next page is where you manage the physical uplinks. Click assign uplinks and assign to the vDS. VDS10

Next we can migrate the VMkernel adapters. I only have a management network but this is the same process if you have vMotion or any other VMkernel adapter. Click assign port group and select the relevant port group in my case it was dv_Managment. VDS11

Last step is to migrate VM’s. If you have multiple networks / port groups you migrated each individual VM to its own port group I only have one so I assigned each VM to the same port group. VDS12


Once the task have completed you should now see the Host and VM’s running on the new port group. VDS14VDS15

Deploy Multiple VM’s using PowerCLI and VMware Template

I wanted to create a few different VM’s so I can test a VMware daily report script. This can be done using deploy VM from template using the vSphere web client but this can take a long time. I decided to write a quick PowerShell script to deploy the VM’s use PowerCLI. I first ran Get-Template to get the name of the template I wanted to use.


I decided to use numerical numbering for my VM’s . So I used a while loop so that it would start at 1 and I added to the $i variable at the end of the loop so it would run until it hit 10. I used the variable $DS to get the datastores and piped that to get-random so the VM’s would be spread across both datastores. If you want to select just one datastore just remove the $DS variable and change the ($DS | Get-Random) to the datastore you want to select.

$DS = Get-Datastore -Name Local*
$i = 1
while ($i -le 10){
New-VM -Name “Linux-VM$i” -Template “Linux-VM_Template” -Datastore ($DS | Get-Random) -VMHost “Esxihost to deploy to”
Deploy OVF template

I then used the below command to get the list of VM’s and their datastores.

Get-VM -Name Linux-VM* | Select Name,@{N=”Datastore”;E={(Get-Datastore -Id $_.DatastoreIdList)}}



OVF parameter chunkSize with value “XXXXXXXXXX” error in vCenter Server 6.5

I was trying to deploy some of my old Linux ovf but I was getting an error about ovf chunkSize.


I had a quick look and found this VMware KB and the error is due to VMware no longer supporting ChunkSize in vSphere 6.5. To fix this issue I had to extract the ovf, I used 7-zip.


If there are multiple disks you may have to combine them using command like the below.

copy /b vmName-disk1.vmdk.000000 + vmName-disk1.vmdk.000001 + ….. + <until the last fragment> vmName-disk1.vmdk

I only had one disk so I used the below.

copy /b Linux-VM-disk1.vmdk.000000000 Linux-VM-disk1.vmdk



Once the file copy command has completed I needed to edit the Linux-VM.ovf file to remove the ChunkSize. I used Notepad ++ to edit this. You can search for chunksize.

<File ovf:chunkSize=”7516192768″ ovf:href=”Linux-VM-disk1.vmdk” ovf:id=”file1″ ovf:size=”58041344″/>


Below is the updated ovf file with the ChunkSize removed.

<File ovf:href=”Linux-VM-disk1.vmdk” ovf:id=”file1″ ovf:size=”58041344″/>


After making the change I tried to import the ovf again. To import use the individual files.


I then got a new error.


This was due to the manifest file the Linux-vm.mf not matching the updated ovf file. To get the filehash run the below command in Powershell. The hash has to be in lower case so use the .tolower to convert the response to lower case.

(Get-FileHash .\Linux-vm.ovf -Algorithm SHA1).hash.tolower()


I then had to edit the Linux-vm.mf file.


Change the SHA1 Linux-VM.OVF file hash, To the new hash that has been export from PowerShell.


Below if the updated .mf file.


After this I was then able to complete the OVF import.


Once Imported we can then export the file to an OVA again so that it can be imported when needed.


How to Join vCSA 6.5 to an Windows AD Domain

To give access to single sign-on for Windows Active Directory users the Platform service controller (PSC) can be added to AD and an identity source can be added.

To join the PSC to an AD domain. Logon to the vCSA and go to AdministrationDJ1

Once in administration go to Deployment > System Configuration > then select the nodes since I am using  an embedded PSC and vCenter there is only one Node. DJ4

Enter Domain details and username password.DJ6

After the domain join has completed a reboot will be required to complete the domain join.


The domain should now show and the Join tab will be grayed out. DJ8

To add permission for the new domain user go to Administration > Single Sign-on and add the domain as an Identity SourcesDJ2

Once click add identity source go to AD windows authenticationDJ3

The domain name should be already populated and I used the machine account. DJ9


Once completed the identity source should show. DJ11


To add a group to the global permissions go to Administration > Access Control > Global permissions > Manage.DJ12

Select the domain and account to add. DJ15

Select role to assign. DJ13

The group or user should know show in the Global Permissions. DJ14


Installing and configuring vCSA 6.5

VMware are moving away from using Windows vCenter server to only using the vCenter Server Appliance as this give simpler management, lower licensing cost and it integrates VMware update manager (VUM) as part of the appliance.  You can still currently download a Windows vCenter server but this is being deprecated and will be removed in the next major release after 6.7. To deploy the vCSA requires the ISO to be downloaded and for the device that will launch the install to have network connectivity with the ESXi host that will run vCenter appliance.

The system requirements for my deployment which is the tiny version is

  • Disk: 120GB
  • Memory: 8GB
  • 2 vCPU

A Tiny Environment can have up to 10 Hosts, 100 Virtual Machines.

Link to download the vCSA is below.


To deploy the vCSA mount or extract the ISO for 6.5. Go to drive letter for the mounted ISO or the extracted folder and then in to vcsa-ui-installer\win32\ and run the installer.exe.


Select installVC2

Click next on stage 1 screen, Accept the licence agreement and click next.

Since this is a lab I just went with an embedded PSC and vCenter server. If you want to test enhanced link mode you will need to use an external PSC. VC5

Select the ESXi host that the appliance will be deployed to.VC6

Select the VM name and the root password.VC7

Select the deployment size I went with tiny as I will only have a few nested hosts. VC8

Select the datastore to be used. VC9

Assign VM Network,  a static IP address, System name and DNS server.  If a FQDN is used for the appliance name and is not registered in DNS the deployment will fail. So make sure it is done before proceeding with the install. VC10

Next screen is just to confirm all settings are correct. VC11


The deployment should promt to start stage 2 of the deployment.

If the appliance doesn’t show the continue screen you can go to stage 2 of the deployed using the admin page for vCenter

go to https://vCentersystemname5480 or https://vCenterIPAddress:5480 and click on set up vCenter server appliance.


Set IP settings for the Embedded PSC.VC14

Set the SSO domain and site-name with administrator password. VC15

last page will be to confirm details for embedded PSC VC16

Click finish and the appliance will start to deploy. VC17

Once completed  the URL’s to access will be presented on-screen. You can access either the vSphere web client Flash version or HTML5. HTML5 is not full functional in version 6.5 so certain task will need to be run from the Flash client. VC18

The logon will be the account and password that was set on the PSC SSO setup. In my case the logon name is administrator@vsphere.local. To logon to vCenter either go to

https://vCentersystemname or https://vCenterIPAddress