VMware 6.7 External PSC to Embedded PSC migration

VMware have recently deprecated using external platform service controller (PSC) with vCenter 6.5 Update2 and 6.7

Deprecation of the external Platform Services Controller deployment model (60229) (vmware.com)

Since we where upgrading from Windows vCenter and PSC to appliance’s we had to migrate to new external PSC and then convert the external PSC after the migration.

In this post we will be going through the process of converting and decommissioning an external PSC.

In vCenter 6.7 there is a conversion tool built in to the vCenter web console that allow the PSC to be converted from an external to embedded PSC.

vCenter Server Converge Tool Enhancements in vSphere 6.7 Update 2 – VMware vSphere Blog

The conversion tool is available under Administration > deployments > system configuration

Select the vCenter server that has the external PSC and click converge to embedded

Enter in the logon details for the existing PSC SSO domain and select Join AD if you want to also join the vCenter appliance to AD (use this if the existing PSC has domain joined).

This image has an empty alt attribute; its file name is image-39.png

The converge tools will start and and it will require vCenter servers services to restart, this will cause an outage to vCenter while the service restart.

This image has an empty alt attribute; its file name is image-40.png

Once all the vCenter services have restarted, vCenter should now show as an having an embedded PSC.

The last step is to decommission the old PSC.

Select the old PSC from in the System configuration and use the decommission PSC action.

Click on porceed.

Enter the SSO logon details, view thumbprint and accept the acknowledgment tickbox.

Click decommission and the PSC will start to be removed.

Once completed vCenter will now be running an embedded PSC and the external PSC will be powered off and can be deleted.

If there are other vCenter in enhanced linked mode follow the same process for each additional vCenter

VMware 6.7 PSC Decommission: Failed to get the PSC thumbprint

As part of our recent VMware 6.7 upgrade we where migrating from external PSC to embedded PSC.

All went fine until we tried to decommission the old PSC’s, when trying to view the thumbprint we got the below error:

Failed to get the PSC thumbprint. Ensure PSC port is correct.

With out getting the thumbprint you can’t continue.

We tested accessing the PSC using web browser,

using openssl from the vCenter appliance

openssl s_client -connect PSC01.domain.local:443

We also tested access using telnet to confirm if port 443 was open

curl -v telnet://PSC01.domain.local:443

All test came back fine and there was no ports being blocked.

In the end the issue was caused by using the short name for the vCenter servers https://VCSA/UI

When we changed to the full FQDN, https://VCSA.domain.local/UI this fixed the issue and we where able to complete the decommission. The issue looks to be related to the SAN name on the cert only having the FQDN.

PSC2

VMware ESXi 6.7 Upgrade: Missing dependencies VIBs Error

Recently we have been upgrading some VMware host from ESXi 6.0 to ESXi 6.7, We where applying the image using VMware update manager and a HPE custom ESXi image.

When applying the image we where getting incompatible warring and where not able to apply the image to upgrade ESXi on some hosts.

The issues was related to VIBS but they where not showing in the html 5 client. vibs1

To find the missing VIBS we ended up having to mount the ISO through HPE ILO and try a manual upgrade which did show the conflicting VIBS. vibs2

In our case the VIBS causing issue was the below.

Mellanox_bootbank_net-mlx4-core_1.9.9.8-10EM.510.0.0.799733
Mellanox_bootbank_net-mlx4-en_1.9.9.0-10EM.510.0.0.799733
Emulex_bootbank_scsi-lpfc820_10.5.55.0-10EM.500.0.0.472560
Mellanox_bootbank_net-mst_2.0.0.0-10EM.500.0.0.472560

The issue seem to be related to older hosts that where previously upgraded from ESXi 5.5.

VMware recommends doing a fresh install if possible but in this case that was not possible, but if you get this error and can do a fresh install that would be the better option. If you can’t do a fresh install below is how I checked to see if the VIBS where in use and how to remove them. 

Next we needed to find out if the VIBS where in use by either the storage or network adapters,

below is the VMware KB that explain how to do this.

https://kb.vmware.com/s/article/1027206

To get the list of storage and network adapter use the esxcli commands

esxcli storage core adapter list

esxcli network nic list

To get check the VIBS version we can use

esxcli software vib list | grep Mel

esxcli software vib list | grep scsi-lpfc820

vibs3

Once we know the version numbers of the VIBS, we just need to confirm they are not used and if not used remove them.

If they where in use we would need to look at creating a custom image or wipe and reload the ESXi host.

We use esxcli to view if the drivers are in use and what version each is

esxcli system module list | grep lpfc820, esxcli network nic get -n vmnic0

vibs4

Once we confirm that none of the VIBS are required the last step is to remove each one. Below is the KB from VMware on removing VIBS.

https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.upgrade.doc/GUID-7FFEBD91-5D82-4E32-93AB-F10D8BFFECAA.html

There might be some VIBS that have dependency’s on others in our case the net-mlx4-core needed to be removed after the net-mlx4-en as it was dependent on it.

To remove we use esxcli software vib remove -n ‘vibname’

vibs5

Reboot the ESXi host if required,

After the reboot, scan the host again from updates tab in VMware vSphere web client and it should now show as non-complaint and not incompatible. vibs6

The host should now upgrade as normal.

HPE Gen 10 SUM Integrated Smart Update Tools VMware

I was installing some new ESXi host using HPE Gen 10 blade servers and was getting a warning when trying to update the firmware using HPE SPP (Service pack for proliant). The issue is due to HPE changing from installing updates directly to the ESXi OS level and instead using HPE ILO.

HPE03

If iSUT is not installed the update will be staged on the host but wont install. To install, download iSUT from the HPE Support site. Below is the link to version 2.3.6 which was the version I used. ISUT_Tool

HPE15Once download and extracted we need to copy the file to the ESXi host, the easiest way to copy is by enabling SSH on the host and using WinSCP.     HPE04I created a folder called hpe_isut on the ESXi host to copy the file to. HPE05Once the files are copied over, use either SSH or ESXi Shell to install the tools. I used SSH with putty as it was easier.

To install use esxcli command, this is the command I used.

esxcli software vib install -d /hpe_sut/sut-esxi6.0-bundle-2.3.6.0-16.zipHPE06Once the install has completed, a restart is required to finish the install. After the reboot the next step is to set the iSUT mode there are 4 <OnDemand/AutoStage/AutoDeploy/AutodeployReboot>

I chose AutoDeploy, to set the mode use the below command.

sut –set mode=AutoDeployHPE09

Once this has completed, run the inventory again from the SPP and the warning should now be gone and the firmware and driver updates should now apply. HPE10

HPE14

Install and Configure VMware NSX

Recently we have been looking to implement zero trust networking. One way to achieve this was to use physical firewall and multiple VLAN’s to break out traffic and restrict access to each VLAN this would take a long time to complete and is quite difficult to manage.

It would require adding between 30 to 60 additional VLAN to our physical servers and VMware and re assinging IP to each server which would cause a lot of downtime.

As an alternative to this I have been looking at VMware NSX to try achieve this same segmentation without the need to redesign the entire VMware networks.

NSX consists of multiple components under different planes like management, control, and data plane’s below is an image of the different plane’s. 

In the next set of posts I am going to go thorough install and configuring a basic NSX deployment. I will be setting this up in a Lab environment and will use nested ESXi and appliances.

It is recommended to have NSX installed on its own management cluster along with vCenter.

First step is to download the OVA for NSX current version is 6.4.4

https://my.vmware.com/web/vmware/details?productId=417&downloadGroup=NSXV_644

below are the system requirments to deploy NSX

NSX Component Hard Drive Memory vCpu
NSX Manager 60 16 4
NSX Controller 20 4 4

NSX 6.4.4 is not supported on vSphere 5.5 below are the supported and recommed verison of vSphere to run NSX 6.4.4:

  • For vSphere 6.0:
    Supported: 6.0 Update 2, 6.0 Update 3
    Recommended: 6.0 Update 3. vSphere 6.0 Update 3 resolves the issue of duplicate VTEPs in ESXi hosts after rebooting vCenter server. SeeVMware Knowledge Base article 2144605 for more information.
  • For vSphere 6.5:
    Supported: 6.5a, 6.5 Update 1
    Recommended: 6.5 Update 1. vSphere 6.5 Update 1 resolves the issue of EAM failing with OutOfMemory. See VMware Knowledge Base Article 2135378 for more information.

Once the OVA is downloaded logon to vCenter right-click on datacenter and deploy OVF Template.

NSX6_1

Select the location of OVANSX2

Give the appliance a nameNSX3

Select the Cluster that will run the applianceNSX4

Click next NSX5

Accept the licence agreement and click continueNSX6

Chose Thick ProvisionNSX7

Select the network that will be used for the management networkNSX8

The next screen is where all the customization will be setup

Appliance Password:

HostName:

Network settings: management IP, subnet, gateway, DNS and NTP. Leave blank if  you want to use DHCP but its recommend to use static addressesNSX9NSX10

Once all setting are configured click next and confirm all settings on the last screen. Once finished the OVA should start to deploy. (Note that this failed the first time for me as I selected a host and there seems to be an issue with this in vCenter 6.7, once I selected the cluster the OVA deployed without issue)NSX11

Once the OVA had been deployed I decided to edit the memory size as I was running low on memory so I change it from 16Gb to 8Gb but for production this should be left at 16Gb.

After this you can connect using DNS name configured above or through the management IPNSX12

The last step in this post is to connect NSX to vCenter

Logon using  admin and the password specified in the config of the OVA

Click on Manage vCenter Registration NSX13

both the lookup and vCenter server connection will need to be configuredNSX15

Add vCenter server and user name / passwordNSX16NSX18

There will be a prompt to trust the vCenter certificate click yes to continueNSX17

Once configured both status should show as connectedNSX19

Open the vCenter web client and once logged on there should now be an addtional tab for Networking & Security. (At the time of this post this option is only available in the Flash version of the Web client not the HTML 5 version) 

NSX20NSX21

In the next post we will start to configure the NSX and controllers.

Using VMware PowerCli Part 1

Since VMware 6.0 I have started to use VMware PowerCli module to automate task and checks that I do daily or for large task that would take a long time to do manually. I am going to go through installing PowerCli and some of the useful command and script that can be use to check VMware.

To install PowerCli there are some pre-req’s

OS Type
64-Bit
Server
  • Windows Server 2012 R2
  • Windows Server 2008 R2 Service Pack 1
Workstation
  • Windows 10
  • Windows 8.1
  • Windows 7 Service Pack 1
  • Windows PowerShell 3.0, 4.0, 5.0, or 5.1
  • .NET Framework 4.5, 4.5.x, 4.6, or 4.6.x

I would recommend installing the latest version of PowerShell which is currently 5.1

To check current version of PowerShell run $PSVersionTablePcli

To install the latest version install the latest Windows Management Framework 5.1 link to download page is below.

https://www.microsoft.com/en-us/download/details.aspx?id=54616

Step 1 is to install PowerCli

The old PowerCli was a PowerShell snap in and required downloading an exe to install, the new version is module based and can be installed directly from the PowerShell console. To install run the below command.

Install-Module VMware.PowerCliPcli1

If you need to update the module to a new release run

Update-Module VMWare.PowerCliPcli2

If a path is not specified the default location that the module files will be placed in is

C:\Program Files\WindowsPowerShell\Modules

Once the install has finished to verify that PowerCli is installed run the below commandPcli3

Step 2 is to connect to vCenter

To connect to vCenter open an elevated PowerShell console and import the VMware Module

Import-Module VMware.PowerCliPcli4

Connect-VIServer lab-VC vCenterServerPcli5

Once connected we can now start to run command against vCenter.

To get host information run

Get-VMHost

Pcli6

To find all VM’s that have snapshots over a certain date. I want to get all snapshots older than 1 day to change this just edit the $date variable.

$date = (Get-Date).AddDays(-1)
$Snapshot = get-vm | get-snapshot
$Snapshot | where {$_.Created -lt $date}Pcli7

To check datastores you can run

Get-Datastore

Pcli8

To get additional info you can do some math’s and use arrays to get % free space of the datastores. The below will get all datastores that have less than 25% free space.

Get-Datastore | select Name,@{N=”UsedSpaceGB”;E={[math]::Round(($_.CapacityGB),2)}},
@{N=”FreeSpaceGB”;E={[math]::Round(($_.FreeSpaceGB),2)}},
@{N=”%Free”;E={[math]::Round(($_.FreeSpaceGB)/($_.CapacityGB)*100,2)}} |
where %Free -lt “25”
Pcli9

VMware 6.5 Migrate From vSS To vDS

Migrating from standard vSwitches to distributed vSwitches has a lot of advantages. I have listed a few below.

  1. Central management for all host in the vDS
  2. Uniform configuration for all hosts
  3. Easily add new port groups

The one major disadvanatage is if vCenter is down you cannot manage the vDS.

To use vDS you will need to have Enterprise or Enterprise Plus VMware licences.

To migrated from vSS to vDS go to networking in the VMware web client and right-click on vCenter server and go to distributed switch.

VDS1

Give the vDS a suitable name

VDS2

Select the version of ESXi you have in production if you select a newer version than you have running in vCenter the host will show as incompatibleVDS3

The defualt number of uplinks is 4. I only had two uplinks so I changed this to two.  VDS4VDS5

After the vSwitch has been created I usually add a port group for management and vMotion VMkernel networks.

To migrated from the standard switch right-click on the vDS and go to Add and Manage Hosts.

VDS7

Select add hosts. VDS8

Select the host that you want to add to the vDSVDS9

The next page is where you manage the physical uplinks. Click assign uplinks and assign to the vDS. VDS10

Next we can migrate the VMkernel adapters. I only have a management network but this is the same process if you have vMotion or any other VMkernel adapter. Click assign port group and select the relevant port group in my case it was dv_Managment. VDS11

Last step is to migrate VM’s. If you have multiple networks / port groups you migrated each individual VM to its own port group I only have one so I assigned each VM to the same port group. VDS12

VDS13

Once the task have completed you should now see the Host and VM’s running on the new port group. VDS14VDS15

Deploy Multiple VM’s using PowerCLI and VMware Template

I wanted to create a few different VM’s so I can test a VMware daily report script. This can be done using deploy VM from template using the vSphere web client but this can take a long time.

I decided to write a quick PowerShell script to deploy the VM’s use PowerCLI and a CSV file. First we need to get the template that are available in vCenter server.

Connect to to vCenters server using PowerCLI and then run Get-Template to list all templates and then copy the name of the template. This will be added to the csv file.

This image has an empty alt attribute; its file name is image-82.png

Since I am going to be using a csv file to create the VM’s this needs to be created with headings that will be called in the script.

Below are the heading I used in the csv, these can either be used as is or the script can be updated using different headings.

NameDatastoreTemplateCluster

Below is the completed csv file I will be using.

Below is the script I used you just need to update the $vms variable with the path that has the csv file.

$vms = Import-Csv -Path D:\Scripts\VMware\Deploy_VMs\Deploy_VMs.csv
foreach ($vm in $vms) {
Write-Warning "Creating $($vm.Name) in $($vm.cluster)"
New-VM -Name $vm.Name -Datastore $vm.Datastore -Template $vm.Template  -ResourcePool $vm.Cluster
}

To confirm the VM’s are being created we can check vCenter Server for running tasks.

The deployment can take awhile depending on the size of the template.

I then used the below command to get the list of VM’s and their datastores just to confirm all VM’s have been created and are in the correct datastore.

Get-VM -Name LAB-Linux* | Select Name,@{N="Datastore";E={(Get-Datastore -Id $_.DatastoreIdList)}}

This is just a quick example of deploying a VM using a template

OVF parameter chunkSize with value “XXXXXXXXXX” error in vCenter Server 6.5

I was trying to deploy some of my old Linux ovf but I was getting an error about ovf chunkSize.

OVF3

I had a quick look and found this VMware KB and the error is due to VMware no longer supporting ChunkSize in vSphere 6.5. To fix this issue I had to extract the ovf, I used 7-zip.

OVF4

If there are multiple disks you may have to combine them using command like the below.

copy /b vmName-disk1.vmdk.000000 + vmName-disk1.vmdk.000001 + ….. + <until the last fragment> vmName-disk1.vmdk

I only had one disk so I used the below.

copy /b Linux-VM-disk1.vmdk.000000000 Linux-VM-disk1.vmdk

OVF6

 

Once the file copy command has completed I needed to edit the Linux-VM.ovf file to remove the ChunkSize. I used Notepad ++ to edit this. You can search for chunksize.

<File ovf:chunkSize=”7516192768″ ovf:href=”Linux-VM-disk1.vmdk” ovf:id=”file1″ ovf:size=”58041344″/>

OVF8

Below is the updated ovf file with the ChunkSize removed.

<File ovf:href=”Linux-VM-disk1.vmdk” ovf:id=”file1″ ovf:size=”58041344″/>

OVF9

After making the change I tried to import the ovf again. To import use the individual files.

OVF10

I then got a new error.

OVF11

This was due to the manifest file the Linux-vm.mf not matching the updated ovf file. To get the filehash run the below command in Powershell. The hash has to be in lower case so use the .tolower to convert the response to lower case.

(Get-FileHash .\Linux-vm.ovf -Algorithm SHA1).hash.tolower()

OVF12

I then had to edit the Linux-vm.mf file.

OVF13

Change the SHA1 Linux-VM.OVF file hash, To the new hash that has been export from PowerShell.

OVF14

Below if the updated .mf file.

OVF16

After this I was then able to complete the OVF import.

OVF15

Once Imported we can then export the file to an OVA again so that it can be imported when needed.

 

How to Join vCSA 6.5 to an Windows AD Domain

To give access to single sign-on for Windows Active Directory users the Platform service controller (PSC) can be added to AD and an identity source can be added.

To join the PSC to an AD domain. Logon to the vCSA and go to AdministrationDJ1

Once in administration go to Deployment > System Configuration > then select the nodes since I am using  an embedded PSC and vCenter there is only one Node. DJ4

Enter Domain details and username password.DJ6

After the domain join has completed a reboot will be required to complete the domain join.

DJ7

The domain should now show and the Join tab will be grayed out. DJ8

To add permission for the new domain user go to Administration > Single Sign-on and add the domain as an Identity SourcesDJ2

Once click add identity source go to AD windows authenticationDJ3

The domain name should be already populated and I used the machine account. DJ9

DJ10

Once completed the identity source should show. DJ11

 

To add a group to the global permissions go to Administration > Access Control > Global permissions > Manage.DJ12

Select the domain and account to add. DJ15

Select role to assign. DJ13

The group or user should know show in the Global Permissions. DJ14