VMware Site Recovery Manager 8.4 Upgrade Invalid certificate when reconnecting to vCenter

I was upgrading our VMware SRM appliance from version 8.2 to 8.4 recently. The upgrade completed successfully but when logged on to the SRM management console and I tried to reconfigure the appliance to connect back to vCenter I started to get a error.

To get the fixes for some of the errors I had to log a case with VMware support who where very helpfully in getting the issue resolved. So decided to do a post on this in case anyone else runs in to the same issue.

Below is the initial error I got when trying to reconnect to vCenter.

The error was about an invalid cert and failed soap response to vCenter

Exit code: 61

[backtrace begin] product: VMware vCenter Site Recovery Manager, version: 8.4.0, build: build-18048862, tag: drconfig, cpu: x86_64, os: linux, buildType: release
backtrace[03] libvmacore.so[0x001BC4BD]
backtrace[04] dr-configurator[0x00151AB4]
backtrace[05] dr-configurator[0x000663BA]
backtrace[06] dr-configurator[0x000EDCB4]
backtrace[07] dr-configurator[0x000F66FC]
backtrace[08] dr-configurator[0x000F79AF]
backtrace[09] dr-configurator[0x000F1788]
backtrace[10] dr-configurator[0x000BB3DD]
backtrace[11] libvmacore.so[0x002EFFED]
backtrace[12] libvmacore.so[0x002F1C32]
backtrace[13] libvmacore.so[0x0040327E]
backtrace[14] libpthread.so.0[0x00007F87]
backtrace[15] libc.so.6[0x000F35BF]
[backtrace end]
Caused by:
(vmodl.fault.InvalidArgument) {
faultCause = (vmodl.MethodFault) null,
faultMessage = ,
invalidProperty = “Invalid certificate”
msg = “Received SOAP response fault from []: create
“
}

I checked the existing PSC cert in the summary page of the SRM management web client but this was fine and showing as valid, I checked the vCenter cert that was also valid so I was a bit confused as to what cert the error was actually referring to as it just said invalid certificate which isn’t all that helpful.

I decided to to check the appliance certificate and that had expired about 4 months ago.

To re-issue the cert we should just need to go to change and select change to reissue the cert, but my issue was that the appliance had previously been set with just a host name and not FQDN so when I tried to generate the cert I got an error for no FQDN.

It looks like the older version of SRM allows cert to be generated without FQDN’s but not version 8.4.

This was going to cause an issue with the site paring as the names had already been configure so most likely once the new cert was generated with the FQDN and connected to vCenter I would probable need to reconfigure the site paring.

First I had to change the host name under networking to use a FQDN.

Once that was changed I tried to re-issued the self singed cert using the new FQDN name but this failed with the below error. (This is where I got stuck and had to open a service request 🙂 )

The certificate’s Subject Alternative Name does not contain one of the folowing: – an IP address that matches the SRM host IP – a DNS name that matches the SRM host name – a common Name field that matches the SRM host name.

VMware support thought the issue was due to the plugin caching the old host name, so I had to then go to reconfigure on the summary page and on name and extension had to change the local host to use the new FQDN to update the plugin settings as it was set to custom and using the old host name.

The reconfigure will most likely fail due to the cert but it will update the plugin hostname which then allow the cert to be generated.

After this I was able to generate the new cert, the web console took a few minutes to come back while the cert was updated.

Once both of the above where completed SRM successfully connect back to vCenter.

After the connection was made to vCenter I had to reconnect in the SRM site pairing to update the SRM URL and certificate from the SRM UI console.

I had to complete the exact same steps on the second appliance.

This was a bit of annoying issue to get around as the errors where pretty vague and the logs didn’t really give much information.

Hopefully the rest of my upgrade goes a bit smother than the SRM upgrade.

Migrate from Active Directory Integrated Windows Authentication VMware vSphere 7.0

VMware is depreciating Integrated Windows Authentication in vSphere 7.0. The feature will be removed in a later release. Below is from the VMware KB.

Support for IWA continues to be available in vSphere 7.0 and will be phased out in a future release. Although IWA can still be configured, we highly recommend using AD over LDAP or Federated Identity (AD FS).

Deprecation of Integrated Windows Authentication (78506) (vmware.com)

In this post we will be going through changing over to using Active Directory over LDAP. We will also be using LDAPS as this is secured with certificates and is much better from a security side and Microsoft are requiring this on applications that use LDAP.

2020 LDAP channel binding and LDAP signing requirements for Windows (microsoft.com)

If you haven’t configured a certificate on your domain controller yet to allow LDAPS I would configure this first before proceeding with the swap over to Active directory over LDAP identity provider.

If we check the existing AD IWA we can see the warning that the feature is depreciated.

I usually create a new account for each applications LDAP connections just so I keep track of what account is used where.

For LDAP authentication in a Windows domain a standard account with just domain users right should have enough permission as it best to use least privilege for service accounts.

To confirm in an Windows AD domain is setup to use LDAPS we can use the ldp on a devices that has the active directory tools enabled to confirm LDAPS connection.

Open and click connect and add in the server name, set port to 636 and tick SSL.

If the configuration is retuned then LDAPS is working.

Once we have the account created and confirmed that LDAPS is working we can start setting up AD over LDAP in vCenter.

Since we will be using the same domain name as the IWA source we need to remove this first or it will cause error when trying add the LDAPS source.

Logon to vCenter web client > Menu > Administration > single sign on > configuration.

Under Identity sources select the IWA and click remove.

Click ok to confirm removal.

Once the IWA is removed we can now add the AD LDAP connection.

Click Add in the Identity source page and select Active Directory over LDAP

Add in the required details.

Name: Friendly name for the identity source.

Base DN: Is the level at which search in AD will start for user or groups to search all AD just use the top level or select sub OU to limit the searches.

Domain name: FQDN of the domain

Domain alias: this is the NetBIOS / pre windows 2000 domain name

When I select any domain controller I was getting the below.

Cannot configure identity source due to Failed to probe provider connectivity [URI: ldaps://domainl ]; tenantName [vsphere.local], userName [User] Caused by: Can’t contact LDAP server.

To work around this I had to specific my DC manually.

As I have a certificate issue from an internal certificate authority I will be selecting the CA cert for LDAPS as this should trust any cert issued by the CA on my domain controllers.

Click Add to complete the AD over LDAP identity source.

If we check the websso.log under /var/log/vmware/sso on the vCenter appliance, we can see the certificate being verified when we logon with a domain account.

We have now move from IWA to AD over LDAP all existing groups and roles should still work.

Filtering VMware vCenter Server Events Using PowerCLI

Recently we needed to review some changes and remote console events to check what user was accessing a particular VM and what changes where made.

I find searching event in the vCenter web client is a bit slow, I prefer to use Get-VIEvent as it has multiple parameters that can be used to search and can also use regular expression to filter by patterns.

I decided to do a blog post on how to filter events to show the different options that I use regularly to filter events.

First we need to connect to vCenter sever using a computer that has PowerCLI installed

Connect-VIServer vCenterServer

Once connected we can start to use Get-VIEvent, to return event for a specific object we can use -entity parameter and the object name.

In the below example I am getting the last event for my LAB-Win10 VM

Get-VIEvent -Entity ObjectName -Maxsamples 1

We can also filter by entity and user name is we know the user is tied to the event.

Get-VIEvent -Entity VM -Username User

We can filter the events by time range.

Get-VIEvent -Start "11/07/2021 20:48" -Finish "11/07/2021 21:00" | Select-Object EventTypeId,CreatedTime

Another option for filtering is to use where-object and search for a specific event message.

Get-VIEvent -Entity VM | Where-Object {$_.FullFormattedMessage -Like "VM started"}

There are addtional properties like host, ComputeResource, datacenter…. that are not return as readable values unless you format the results.

To format the result we can use select-object and create an array to give a name to the property and select the property value. Below will show the datacenter, cluster and host the event was create on.

Get-VIEvent -Entity object -Username User | Select-Object Message,CreatedTime,UserName,@{N="DataCenter";E={$_.DataCenter.Name}},@{N="Compute";E={$_.ComputeResource.Name}},@{N="Host";E={$_.Host.Name}}

Below is an example showing what the event looks like before and after using the array’s to return the readable values.

Once we the events we want we can either review them in the PowerShell console or use Export-csv to export the results.

Audit VMware vCenter Server Permission Using PowerCLI

As part of our VMware 6.7 to 7.0 Upgrade we wanted to audit the existing vCenter server permission. We have a lot of contractors who come in to do work and users who have had permission assigned but these permission are not always removed.

We wanted to get a report that export each of the permission assigned in vCenter.

I could do this manually but this would take a while and is not that easily repeatable so I decided to create a quick script that will export the required information.

The script will be calling two command (Get-VIPermission to export permission and Get-VIRole to export the assigned privileges) and then formation the results.

The script also has some mandatory variables (one for the vCenter server and one for the export path) and there is some error handling incase there is no connection to vCenter server or the export folder doesn’t exist.

There are three type of object in VMware permissions.

Privilege: Allow specific actions (create, delete, manage.. ) or rights to view specific properties
Role : A set of privileges assigned to an object to allow assignment
Permission: Is either a set of a users or groups that have been assigned to a role

If we run Get-ViPermission on we will see all permission returned.

We can select one specific permission by using -principal and expand using format-list. This gives a bit more information but we are missing the assigned privilege’s.

This image has an empty alt attribute; its file name is image-28.png

This is where we use Get-VIRole as this has a property that shows privileges that have been assigned to the role.

Below is an example of the script running.

.\VMware_Permissions_Audit.ps1 -VCServer lab-vc.thesleepyadmin.local -ReportExport .\

Once completed the csv file should be exported with the vCenter server name.

Below is what the csv export should look like.

Below is an example of the error handling when connection to vCenter.

The full script can be downloaded from the below link to my GitHub.

Scripts/VMware/Permissions_Audit at master · TheSleepyAdmin/Scripts (github.com)

Updating VMware tools on ESXi 7.0 host using VMware Lifecycle Manager

There was a recent VMware local privilege escalation vulnerability in VMware tools version 11.2.6 and below. See VMware advisors VMSA-2021-0013 (vmware.com).

The vunerablity has been fixed in VMware tools version 11.3

VMware Tools 11.3.0 Release Notes

We needed to update the version of VMware tools running manually as the tools are not currently included in any of our standard baselines we apply to our ESXi hosts.

I decided to do a post on how to update the version of VMware tools using VMware Lifecycle manager baseline as it’s a little bit different than VMware Update Manager.

First we need to go to Lifecycle Manager, open the vSphere web console > Menu > Lifecycle Manager

In Lifecycle manager the tools should be synced as previously in VMware Update Manager the tools need to be manually uploaded.

The quickest way I find to check the latest tools that have been synced is by click on image depot and select components.

We could also check under updates and turn off show only rollup updates. (If the tools required a reboot it would show under impact)

Next we will create a baseline to apply the latest tools.

Go to baselines and select new baseline.

Give the baseline a name and select patch

Untick Automatically update this baseline

Untick show only rollup updates and filter for VMware tools, there will probable be a different VMware tools for 6.x and 7.x so check before adding to the baseline.

Click next and complete the baseline creation.

We can check the current tools status by going to the esxi host > Updates > VMware tools and check status.

We can now apply the baseline and run the check again and it should show as out of date.

The baseline can be applied either directly to the ESXi host or to the cluster we will be applying to the cluster as it saves time having to apply to each host individually .

Go to the cluster > Updates > attach and select attached baseline.