VMware PowerCLI Module 12.4 Update Error

With the release of VMware PowerCLI 12.4 I was updating the module on my LAB servers so I could check out the new command.

When upgrading I was getting the below error which is due to VMware changing the certificate authority used to publish the new module.

Authenticode issuer ‘E=noreply@vmware.com, CN=”VMware, Inc.”, O=”VMware, Inc.”, L=Palo Alto, S=California, C=US’ of the new module ‘VMware.VimAutomation.Sdk’ with version ‘12.4.0.18627054’ from root
certificate authority.

In the release notes from VMware there is a know issue with the certificate. The fix is to remove and re-install the module.

https://blogs.vmware.com/PowerCLI/2021/09/powercli-12-4-whats-new.html

I though I would do a quick post on this incase anyone doesn’t know the process.

To check where the module is installed run the below command.

Get-Module -ListAvailable VMware.PowerCLI
This image has an empty alt attribute; its file name is image-21.png

In my case the module was installed in C:\Program Files\WindowsPowerShell\Modules.

Next we need to remove the old modules, below is a list of the folders that make up the VMware PowerCLI 12.0 module.

After removing the folder we now have to install the new module using Install-Module

Install-Module VMware.PowerCLI

Once completed we should now have PowerCLI version 12.4

Using VMware PowerCLI Get-EsxCli

In this post we will be going through using the PowerCLI Get-EsxCli commandlet, which is used to call EsxCli command from PowerShell instead of having to SSH directly to the the ESXi host.

This can be useful when trying to gather information from multiple hosts instead of connecting on to each host with SSH or updating configuration settings.

First step is to connect to vCenter, once connected we can run the Get-EsxCli command and this will return the list of namespaces that can be use to gather information or set configuration settings.

We will be using -V2 as this sets Get-EsxCli to use version 2 interface as version 1 is being deprecated and will be removed in a later version.

Get-EsxCli -VMHost esxihost -V2

We can set the Get-EsxCli command to a variable so that we can call the namespaces, below will call the network namespace, once called there will be a list of addtiaonl namespaces that can be called.

$vmhostesxcli = Get-EsxCli -VMHost esxihost -V2
$vmhostesxcli.network

Once we select a namespace there should also be a list of methods that can be called these can be used to run specific actions like list or get to return information.

To list all nic’s we can call the nic and then list namespace

$vmhostesxcli.network.nic.list.invoke()

To gather info on details like driver version we can use the get method, in v1 you use to be able to call the nic name directly but if you try this in v2 the below error will be returned.

If specified, the arguments parameter must contain a single value of type Hashtable.

This is due to the method requiring a hash table parameter. If we call the help method it should return the valid parameters. If the parameter has a hyphens remove this or the command wont work.

For the nic namespace the parameter is nicname and needs to be called as a hash table.

$vmhostesxcli.network.nic.get.invoke(@{nicname="vmnic1"})

There might be some namespaces that have multiple properties like driverinfo under nic, these can be called by adding the property like below.

$vmhostesxcli.network.nic.get.invoke(@{nicname="vmnic1"}).DriverInfo

Now that we have the command we can start to build out a script to export information.

In this case we will be getting all hosts, listing all NIC’s and getting the drivers info.

The above shows that Get-EsxCli can be very good for retrieving information, if we wanted to set a configuration we can use similar command syntax but use the set method.

In this case we will update the Power policy settings on all host, we could do this manually by going to the Configure > Hardware > Overview > Power Management one each host and update the power policy settings

but doing this on a larger cluster is a lot of effort. To update using PowerCli we can first create the script to report on the current host power policy.

To update the policy settings we will use the set method like the below.

$vmhostesxcli.hardware.Power.policy.set.Invoke(@{id="1")})

Below are the 4 different power policy’s.

IDPower Policy
1High Performance
2Balanced
3Lower Power
4Custom

We can then take the above and create a re-usable script to report or set the power policy on all host in one go.

To download a copy of the either of the above script use the below link to my Github these can be used as reference if you want to create your own scripts for other settings.

https://github.com/TheSleepyAdmin/Scripts/tree/master/VMware/EsxCli

Upgrading VMware Tools Different Methods

Keeping VMware tools updated is an import task to improve performance and keep up with security / bug fixes. VMware tools are not required for a VM to run but are used to enable VMware features.

VMware tools can be upgrade in a few different methods. In this post we will go through some of these different methods these range from completely manual by upgrading to fully automated using VM options, scripting or application deployment.

First method is to manually mount the tools on the VM by right clicking and then install in the guest OS, this is the slowest method and is fine if there is a few VM to upgrade but not really great at scale.

Second method is to set the VMware tools to check for upgrades during power on. This method can be good when doing OS patching as when patching a VM the tools will also be upgraded.

To enabled this either edit the VM setting and go to VM Options > VMware tools > Tools Upgrade

To enabled on multiple VM’s we can use PowerCli to bulk update the VM options.

To view the current upgrade policy we can use

(Get-VM VMName  | Get-View).config.Tools

To update the VM Option using PowerCli we can create a new VM configuration and then apply to a list of VMs in a text file using a loop. To set the upgrade policy back to manual just update tools upgrade policy from upgradeAtPowerCycle to manual.

$VMs = Get-Content C:\Temp\VMs\VMware_Tools_Upgrade.txt
$Toolsconfig = New-Object VMware.Vim.VirtualMachineConfigSpec
$Toolsconfig.tools = New-Object VMware.Vim.ToolsConfigInfo
$Toolsconfig.tools.toolsUpgradePolicy = "upgradeAtPowerCycle"

foreach ($VM in $VMs) {
Write-Warning "Setting VM Tools to Upgrade on PowerOn for $VM"
(Get-VM -Name $VM | Get-View).ReconfigVM($Toolsconfig)

 }

vCenter should show the tasks that where run to update the VM option.

Now when the VM’s reboot they should auto upgrade to the newest version.

Third method is to use a script to update the tools. This can be used when we want to have more control over when VMware tools are upgrade.

This can be done by using the VMware PowerCli Update-Tools command

For one VM we could use

Update-Tools VMName

Below is an example of using a text file with VMs names. I used the no reboot parameter to stop the VM from rebooting after the tools are updated.



$VMs = Get-Content C:\Temp\VMs\VMware_Tools_Upgrade.txt
foreach ($VM in $VMs) {
Get-VM -Name $VM  | Update-Tools -NoReboot
}

If we check vCenter we should see the upgrade task start

The last method I would use would be an application deployment tool to push out the latest VMware tools.

I use Microsoft Endpoint Configuration Manager for most of my deployment so this will be where I will be creating the deployment package.

Mount the VMware tools on on of the VMs and copy the install files.

Open the ConfigMgr console > Software Library > Application Management > Application

Create a new application and set to manual specify.

Add in the application details.

Add a new Deployment Type

Set type to manual specify.

Give the deployment a name

I used the document for the command line switches for the tools upgrade its for version 5.5 but the same switches worked.

https://docs.vmware.com/en/VMware-vSphere/5.5/com.vmware.vmtools.install.doc/GUID-CD6ED7DD-E2E2-48BC-A6B0-E0BB81E05FA3.html

Add the path to the copied install files, install command line switches and the uninstall command line (not required)

Command line: setup.exe  /S /v “/qn REBOOT=R

Next set a detection method this can be either MSI install code if available but I used file version for the vmtoolsd.exe.

below is what the detection clause should look like.

Set the install behavior to Install for system and logon requirement to Whether or not a user is logged on.

I didn’t set any requirements or dependencies. Review the summary page and then complete the application wizard.

Once completed the wizard will go back to the create application wizard.

Follow the wizard and complete.

Now that the application is create we can deploy out to the required VM’s.

Right click and go to deploy.

Select the device collection that will be used to put VMs that will have there tools upgraded.

Add the application to the required distribution points so that content is transferred.

There are two install type available or required. To force the install use required.

For scheduling I setup as soon as possible but this can be set to what ever time the application should start to install.

I left user experience and alerts as default and completed the deployment wizard.

Once the application deployment has been assigned to the client the app should now show in software center on the client and start the upgrade.

After the upgrade completes the version of VMware tools should be 11.3 and will prompt for reboot if required.

These are the main way I manage VM tools upgrades, I usually try to use VM options for most severs as it works well and requires the least manual intervention to keep VMware tools up to date.

VMware Site Recovery Manager 8.4 Upgrade Invalid certificate when reconnecting to vCenter

I was upgrading our VMware SRM appliance from version 8.2 to 8.4 recently. The upgrade completed successfully but when logged on to the SRM management console and I tried to reconfigure the appliance to connect back to vCenter I started to get a error.

To get the fixes for some of the errors I had to log a case with VMware support who where very helpfully in getting the issue resolved. So decided to do a post on this in case anyone else runs in to the same issue.

Below is the initial error I got when trying to reconnect to vCenter.

The error was about an invalid cert and failed soap response to vCenter

Exit code: 61

[backtrace begin] product: VMware vCenter Site Recovery Manager, version: 8.4.0, build: build-18048862, tag: drconfig, cpu: x86_64, os: linux, buildType: release
backtrace[03] libvmacore.so[0x001BC4BD]
backtrace[04] dr-configurator[0x00151AB4]
backtrace[05] dr-configurator[0x000663BA]
backtrace[06] dr-configurator[0x000EDCB4]
backtrace[07] dr-configurator[0x000F66FC]
backtrace[08] dr-configurator[0x000F79AF]
backtrace[09] dr-configurator[0x000F1788]
backtrace[10] dr-configurator[0x000BB3DD]
backtrace[11] libvmacore.so[0x002EFFED]
backtrace[12] libvmacore.so[0x002F1C32]
backtrace[13] libvmacore.so[0x0040327E]
backtrace[14] libpthread.so.0[0x00007F87]
backtrace[15] libc.so.6[0x000F35BF]
[backtrace end]
Caused by:
(vmodl.fault.InvalidArgument) {
faultCause = (vmodl.MethodFault) null,
faultMessage = ,
invalidProperty = “Invalid certificate”
msg = “Received SOAP response fault from []: create

}

I checked the existing PSC cert in the summary page of the SRM management web client but this was fine and showing as valid, I checked the vCenter cert that was also valid so I was a bit confused as to what cert the error was actually referring to as it just said invalid certificate which isn’t all that helpful.

I decided to to check the appliance certificate and that had expired about 4 months ago.

To re-issue the cert we should just need to go to change and select change to reissue the cert, but my issue was that the appliance had previously been set with just a host name and not FQDN so when I tried to generate the cert I got an error for no FQDN.

It looks like the older version of SRM allows cert to be generated without FQDN’s but not version 8.4.

This was going to cause an issue with the site paring as the names had already been configure so most likely once the new cert was generated with the FQDN and connected to vCenter I would probable need to reconfigure the site paring.

First I had to change the host name under networking to use a FQDN.

Once that was changed I tried to re-issued the self singed cert using the new FQDN name but this failed with the below error. (This is where I got stuck and had to open a service request 🙂 )

The certificate’s Subject Alternative Name does not contain one of the folowing: – an IP address that matches the SRM host IP – a DNS name that matches the SRM host name – a common Name field that matches the SRM host name.

VMware support thought the issue was due to the plugin caching the old host name, so I had to then go to reconfigure on the summary page and on name and extension had to change the local host to use the new FQDN to update the plugin settings as it was set to custom and using the old host name.

The reconfigure will most likely fail due to the cert but it will update the plugin hostname which then allow the cert to be generated.

After this I was able to generate the new cert, the web console took a few minutes to come back while the cert was updated.

Once both of the above where completed SRM successfully connect back to vCenter.

After the connection was made to vCenter I had to reconnect in the SRM site pairing to update the SRM URL and certificate from the SRM UI console.

I had to complete the exact same steps on the second appliance.

This was a bit of annoying issue to get around as the errors where pretty vague and the logs didn’t really give much information.

Hopefully the rest of my upgrade goes a bit smother than the SRM upgrade.

Migrate from Active Directory Integrated Windows Authentication VMware vSphere 7.0

VMware is depreciating Integrated Windows Authentication in vSphere 7.0. The feature will be removed in a later release. Below is from the VMware KB.

Support for IWA continues to be available in vSphere 7.0 and will be phased out in a future release. Although IWA can still be configured, we highly recommend using AD over LDAP or Federated Identity (AD FS).

Deprecation of Integrated Windows Authentication (78506) (vmware.com)

In this post we will be going through changing over to using Active Directory over LDAP. We will also be using LDAPS as this is secured with certificates and is much better from a security side and Microsoft are requiring this on applications that use LDAP.

2020 LDAP channel binding and LDAP signing requirements for Windows (microsoft.com)

If you haven’t configured a certificate on your domain controller yet to allow LDAPS I would configure this first before proceeding with the swap over to Active directory over LDAP identity provider.

If we check the existing AD IWA we can see the warning that the feature is depreciated.

I usually create a new account for each applications LDAP connections just so I keep track of what account is used where.

For LDAP authentication in a Windows domain a standard account with just domain users right should have enough permission as it best to use least privilege for service accounts.

To confirm in an Windows AD domain is setup to use LDAPS we can use the ldp on a devices that has the active directory tools enabled to confirm LDAPS connection.

Open and click connect and add in the server name, set port to 636 and tick SSL.

If the configuration is retuned then LDAPS is working.

Once we have the account created and confirmed that LDAPS is working we can start setting up AD over LDAP in vCenter.

Since we will be using the same domain name as the IWA source we need to remove this first or it will cause error when trying add the LDAPS source.

Logon to vCenter web client > Menu > Administration > single sign on > configuration.

Under Identity sources select the IWA and click remove.

Click ok to confirm removal.

Once the IWA is removed we can now add the AD LDAP connection.

Click Add in the Identity source page and select Active Directory over LDAP

Add in the required details.

Name: Friendly name for the identity source.

Base DN: Is the level at which search in AD will start for user or groups to search all AD just use the top level or select sub OU to limit the searches.

Domain name: FQDN of the domain

Domain alias: this is the NetBIOS / pre windows 2000 domain name

When I select any domain controller I was getting the below.

Cannot configure identity source due to Failed to probe provider connectivity [URI: ldaps://domainl ]; tenantName [vsphere.local], userName [User] Caused by: Can’t contact LDAP server.

To work around this I had to specific my DC manually.

As I have a certificate issue from an internal certificate authority I will be selecting the CA cert for LDAPS as this should trust any cert issued by the CA on my domain controllers.

Click Add to complete the AD over LDAP identity source.

If we check the websso.log under /var/log/vmware/sso on the vCenter appliance, we can see the certificate being verified when we logon with a domain account.

We have now move from IWA to AD over LDAP all existing groups and roles should still work.

Filtering VMware vCenter Server Events Using PowerCLI

Recently we needed to review some changes and remote console events to check what user was accessing a particular VM and what changes where made.

I find searching event in the vCenter web client is a bit slow, I prefer to use Get-VIEvent as it has multiple parameters that can be used to search and can also use regular expression to filter by patterns.

I decided to do a blog post on how to filter events to show the different options that I use regularly to filter events.

First we need to connect to vCenter sever using a computer that has PowerCLI installed

Connect-VIServer vCenterServer

Once connected we can start to use Get-VIEvent, to return event for a specific object we can use -entity parameter and the object name.

In the below example I am getting the last event for my LAB-Win10 VM

Get-VIEvent -Entity ObjectName -Maxsamples 1

We can also filter by entity and user name is we know the user is tied to the event.

Get-VIEvent -Entity VM -Username User

We can filter the events by time range.

Get-VIEvent -Start "11/07/2021 20:48" -Finish "11/07/2021 21:00" | Select-Object EventTypeId,CreatedTime

Another option for filtering is to use where-object and search for a specific event message.

Get-VIEvent -Entity VM | Where-Object {$_.FullFormattedMessage -Like "VM started"}

There are addtional properties like host, ComputeResource, datacenter…. that are not return as readable values unless you format the results.

To format the result we can use select-object and create an array to give a name to the property and select the property value. Below will show the datacenter, cluster and host the event was create on.

Get-VIEvent -Entity object -Username User | Select-Object Message,CreatedTime,UserName,@{N="DataCenter";E={$_.DataCenter.Name}},@{N="Compute";E={$_.ComputeResource.Name}},@{N="Host";E={$_.Host.Name}}

Below is an example showing what the event looks like before and after using the array’s to return the readable values.

Once we the events we want we can either review them in the PowerShell console or use Export-csv to export the results.

Audit VMware vCenter Server Permission Using PowerCLI

As part of our VMware 6.7 to 7.0 Upgrade we wanted to audit the existing vCenter server permission. We have a lot of contractors who come in to do work and users who have had permission assigned but these permission are not always removed.

We wanted to get a report that export each of the permission assigned in vCenter.

I could do this manually but this would take a while and is not that easily repeatable so I decided to create a quick script that will export the required information.

The script will be calling two command (Get-VIPermission to export permission and Get-VIRole to export the assigned privileges) and then formation the results.

The script also has some mandatory variables (one for the vCenter server and one for the export path) and there is some error handling incase there is no connection to vCenter server or the export folder doesn’t exist.

There are three type of object in VMware permissions.

  • Privilege: Allow specific actions (create, delete, manage.. ) or rights to view specific properties
  • Role : A set of privileges assigned to an object to allow assignment
  • Permission: Is either a set of a users or groups that have been assigned to a role

If we run Get-ViPermission on we will see all permission returned.

We can select one specific permission by using -principal and expand using format-list. This gives a bit more information but we are missing the assigned privilege’s.

This image has an empty alt attribute; its file name is image-28.png

This is where we use Get-VIRole as this has a property that shows privileges that have been assigned to the role.

Below is an example of the script running.

.\VMware_Permissions_Audit.ps1 -VCServer lab-vc.thesleepyadmin.local -ReportExport .\

Once completed the csv file should be exported with the vCenter server name.

Below is what the csv export should look like.

Below is an example of the error handling when connection to vCenter.

The full script can be downloaded from the below link to my GitHub.

Scripts/VMware/Permissions_Audit at master · TheSleepyAdmin/Scripts (github.com)

Updating VMware tools on ESXi 7.0 host using VMware Lifecycle Manager

There was recent VMware local privilege escalation vulnerability in VMware tools below 11.2.6 and below. See VMware advisors VMSA-2021-0013 (vmware.com).

The vunerablity has been fixed in VMware tools version 11.3

VMware Tools 11.3.0 Release Notes

We needed to update the version of VMware tools running manually as the tools are not currently included in any other of our standard baselines we apply to our hosts.

I decided to do a to a post on how to update the version of VMware tools using VMware Lifecycle manager baseline as it a little bit different that VMware Update Manager.

First we need to go to Lifecycle Manager, open the vSphere web console > Menu > Lifecycle Manager

In Lifecycle manager the tools should be synced as previously in VMware Update Manager the tools need to be manually uploaded.

To quickest way I find to check the latest tools have been synced is by click on image depot and select components.

We could also check under updates and turn off show only rollup updates. (If the tools required a reboot it would show under impact)

Next we will create a baseline to apply the latest tools.

Go to baselines and select new baseline.

Give the baseline a name and select patch

Untick Automatically update this baseline

Untick show only rollup updates and filter for VMware tools, there will probable be a different VMware tools for 6.x and 7.x so check before adding to the baseline.

Click next and complete the baseline creation.

We can check the current tools status by going to the esxi host > Updates > VMware tools and check status.

We can now apply the baseline and run the check again and it should show as out of date.

The baseline can be applied either directly to the ESXi host or to the cluster we will be applying to the cluster as it save time having to apply to each host indiduvally .

Go to the cluster > Updates > attach and select attached baseline.

Select the VMware tools baseline and attach.

Next run a compliance check on the ESXi host.

Check the baseline status.

Next we will remediate the baseline to apply the latest tools.

If there are no issue with the pre-check click remediate.

Once the remediation is done the tools should show as compliant.

Once applied the VM should now pickup that there is a new tools version available.

The tools can now be applied to the VM either using a script, update on reboot or manually.

Report on users MFA status in Office 365 using PowerShell

During a recent audit we wanted to confirm what users had MFA enabled in Office 365. We use conditional access policy to enforce MFA.

We wanted to check each users to see if they had setup MFA and had a method configured. We also wanted to get information on licensing status and assigned licenses.

The only pre-req for using the script is that the MSOnline Powershell module is installed.

To install the MSOline module open and admin PowerShell windows and run

Install-Module -Name MSOnline

To confirm the module is installed run the below command.

Get-Module -ListAvailable MSOnline
This image has an empty alt attribute; its file name is image-26.png

First we need to connect to MS Online to do this run

Connect-MsolService 

Once connected to check the MFA status I will be using the StrongAuthenticationMethods properties as if MFA is configured for the user there will be a default method set.

For users that haven’t configured MFA no StrongAuthenticationMethods is set.

Below are the 4 methods available for MFA.

OneWaySMS
TwoWayVoiceMobile
PhoneAppOTP
PhoneAppNotification

In the script I only want to return the default method.

There is only one mandatory parameter for the export path where the report will be exported to.

The below is an example of how to run the report.

.\Office365_MFA_Report.ps1 -ExportPath C:\temp

Below is what the output will look like.

The full script can be downloaded from the below link.

Scripts/Office365_MFA_Report.ps1 at master · TheSleepyAdmin/Scripts (github.com)

Weekly Active Directory Audit Report PowerShell

Recently a request came in from our security team to audit recently create, deleted AD object, accounts due to expire (this is for third party users) and modified / created group policy objects so that they would be able to trace the changes happening in Active Directory.

I decided to write a PowerShell script that will export the required information and then send a the csv export to the user that require the information.

This could also be used to import the data to a dashboard by either using the CSV files or if the dashboard can use direct PowerShell script like PowerBI.

First there are some mandatory parameters. Exportpath and domain.

To allow the script to be run without emailing the csv I have left the smtpserver, to and from address as not mandatory parameters.

The script used two different modules

Group Policy:

ActiveDirectory:

To install these go on a Windows server go to add roles and features and select Group policy Management

and under RSAT enabled the Active Directory module.

Once all the features are enable we can run the script.

I have set the default time to last 7 days but if you want to go back further then update the date value.

To run the script so that it just export local without email the reports use the below.

.\WeeklyAD_AuditReport_V1.ps1 -exportPath c:\Temp\AD_Audit\ -domains domian.local

To email the report use the below

.\WeeklyAD_AuditReport_V1.ps1 -SMTPServer mailserver.domain.local -toAddress administrator@domain.local -FromAddress ADreport@domain.local -exportPath c:\Temp\AD_Audit\ -domains domian.local

Once the script completes we can check that the csv files have been created.

If the SMTP server parameter is set, the script will send a email and add the csv as attachments.

Below is what the outputs should look like.

GPO:

Deleted Objects:

Account expire:

The full script can be downloaded from the below link to my GitHub.

Scripts/ActiveDirectory/WeeklyReport at master · TheSleepyAdmin/Scripts (github.com)

The script can then be set to run as a scheduled task to run on a weekly scheduled.