With the release of VMware PowerCLI 12.4 I was updating the module on my LAB servers so I could check out the new command.
When upgrading I was getting the below error which is due to VMware changing the certificate authority used to publish the new module.
Authenticode issuer ‘E=noreply@vmware.com, CN=”VMware, Inc.”, O=”VMware, Inc.”, L=Palo Alto, S=California, C=US’ of the new module ‘VMware.VimAutomation.Sdk’ with version ‘12.4.0.18627054’ from root certificate authority.
In the release notes from VMware there is a know issue with the certificate. The fix is to remove and re-install the module.
In this post we will be going through using the PowerCLI Get-EsxCli commandlet, which is used to call EsxCli command from PowerShell instead of having to SSH directly to the the ESXi host.
This can be useful when trying to gather information from multiple hosts instead of connecting on to each host with SSH or updating configuration settings.
First step is to connect to vCenter, once connected we can run the Get-EsxCli command and this will return the list of namespaces that can be use to gather information or set configuration settings.
We will be using -V2 as this sets Get-EsxCli to use version 2 interface as version 1 is being deprecated and will be removed in a later version.
Get-EsxCli -VMHost esxihost -V2
We can set the Get-EsxCli command to a variable so that we can call the namespaces, below will call the network namespace, once called there will be a list of addtiaonl namespaces that can be called.
Once we select a namespace there should also be a list of methods that can be called these can be used to run specific actions like list or get to return information.
To list all nic’s we can call the nic and then list namespace
$vmhostesxcli.network.nic.list.invoke()
To gather info on details like driver version we can use the get method, in v1 you use to be able to call the nic name directly but if you try this in v2 the below error will be returned.
If specified, the arguments parameter must contain a single value of type Hashtable.
This is due to the method requiring a hash table parameter. If we call the help method it should return the valid parameters. If the parameter has a hyphens remove this or the command wont work.
For the nic namespace the parameter is nicname and needs to be called as a hash table.
Now that we have the command we can start to build out a script to export information.
In this case we will be getting all hosts, listing all NIC’s and getting the drivers info.
The above shows that Get-EsxCli can be very good for retrieving information, if we wanted to set a configuration we can use similar command syntax but use the set method.
In this case we will update the Power policy settings on all host, we could do this manually by going to the Configure > Hardware > Overview > Power Management one each host and update the power policy settings
but doing this on a larger cluster is a lot of effort. To update using PowerCli we can first create the script to report on the current host power policy.
To update the policy settings we will use the set method like the below.
We can then take the above and create a re-usable script to report or set the power policy on all host in one go.
To download a copy of the either of the above script use the below link to my Github these can be used as reference if you want to create your own scripts for other settings.
Keeping VMware tools updated is an import task to improve performance and keep up with security / bug fixes. VMware tools are not required for a VM to run but are used to enable VMware features.
VMware tools can be upgrade in a few different methods. In this post we will go through some of these different methods these range from completely manual by upgrading to fully automated using VM options, scripting or application deployment.
First method is to manually mount the tools on the VM by right clicking and then install in the guest OS, this is the slowest method and is fine if there is a few VM to upgrade but not really great at scale.
Second method is to set the VMware tools to check for upgrades during power on. This method can be good when doing OS patching as when patching a VM the tools will also be upgraded.
To enabled this either edit the VM setting and go to VM Options > VMware tools > Tools Upgrade
To enabled on multiple VM’s we can use PowerCli to bulk update the VM options.
To view the current upgrade policy we can use
(Get-VM VMName | Get-View).config.Tools
To update the VM Option using PowerCli we can create a new VM configuration and then apply to a list of VMs in a text file using a loop. To set the upgrade policy back to manual just update tools upgrade policy from upgradeAtPowerCycle to manual.
$VMs = Get-Content C:\Temp\VMs\VMware_Tools_Upgrade.txt
$Toolsconfig = New-Object VMware.Vim.VirtualMachineConfigSpec
$Toolsconfig.tools = New-Object VMware.Vim.ToolsConfigInfo
$Toolsconfig.tools.toolsUpgradePolicy = "upgradeAtPowerCycle"
foreach ($VM in $VMs) {
Write-Warning "Setting VM Tools to Upgrade on PowerOn for $VM"
(Get-VM -Name $VM | Get-View).ReconfigVM($Toolsconfig)
}
vCenter should show the tasks that where run to update the VM option.
Now when the VM’s reboot they should auto upgrade to the newest version.
Third method is to use a script to update the tools. This can be used when we want to have more control over when VMware tools are upgrade.
This can be done by using the VMware PowerCli Update-Tools command
For one VM we could use
Update-Tools VMName
Below is an example of using a text file with VMs names. I used the no reboot parameter to stop the VM from rebooting after the tools are updated.
Add the path to the copied install files, install command line switches and the uninstall command line (not required)
Command line: setup.exe /S /v “/qn REBOOT=R
Next set a detection method this can be either MSI install code if available but I used file version for the vmtoolsd.exe.
below is what the detection clause should look like.
Set the install behavior to Install for system and logon requirement to Whether or not a user is logged on.
I didn’t set any requirements or dependencies. Review the summary page and then complete the application wizard.
Once completed the wizard will go back to the create application wizard.
Follow the wizard and complete.
Now that the application is create we can deploy out to the required VM’s.
Right click and go to deploy.
Select the device collection that will be used to put VMs that will have there tools upgraded.
Add the application to the required distribution points so that content is transferred.
There are two install type available or required. To force the install use required.
For scheduling I setup as soon as possible but this can be set to what ever time the application should start to install.
I left user experience and alerts as default and completed the deployment wizard.
Once the application deployment has been assigned to the client the app should now show in software center on the client and start the upgrade.
After the upgrade completes the version of VMware tools should be 11.3 and will prompt for reboot if required.
These are the main way I manage VM tools upgrades, I usually try to use VM options for most severs as it works well and requires the least manual intervention to keep VMware tools up to date.
I was upgrading our VMware SRM appliance from version 8.2 to 8.4 recently. The upgrade completed successfully but when logged on to the SRM management console and I tried to reconfigure the appliance to connect back to vCenter I started to get a error.
To get the fixes for some of the errors I had to log a case with VMware support who where very helpfully in getting the issue resolved. So decided to do a post on this in case anyone else runs in to the same issue.
Below is the initial error I got when trying to reconnect to vCenter.
The error was about an invalid cert and failed soap response to vCenter
I checked the existing PSC cert in the summary page of the SRM management web client but this was fine and showing as valid, I checked the vCenter cert that was also valid so I was a bit confused as to what cert the error was actually referring to as it just said invalid certificate which isn’t all that helpful.
I decided to to check the appliance certificate and that had expired about 4 months ago.
To re-issue the cert we should just need to go to change and select change to reissue the cert, but my issue was that the appliance had previously been set with just a host name and not FQDN so when I tried to generate the cert I got an error for no FQDN.
It looks like the older version of SRM allows cert to be generated without FQDN’s but not version 8.4.
This was going to cause an issue with the site paring as the names had already been configure so most likely once the new cert was generated with the FQDN and connected to vCenter I would probable need to reconfigure the site paring.
First I had to change the host name under networking to use a FQDN.
Once that was changed I tried to re-issued the self singed cert using the new FQDN name but this failed with the below error. (This is where I got stuck and had to open a service request 🙂 )
The certificate’s Subject Alternative Name does not contain one of the folowing: – an IP address that matches the SRM host IP – a DNS name that matches the SRM host name – a common Name field that matches the SRM host name.
VMware support thought the issue was due to the plugin caching the old host name, so I had to then go to reconfigure on the summary page and on name and extension had to change the local host to use the new FQDN to update the plugin settings as it was set to custom and using the old host name.
The reconfigure will most likely fail due to the cert but it will update the plugin hostname which then allow the cert to be generated.
After this I was able to generate the new cert, the web console took a few minutes to come back while the cert was updated.
Once both of the above where completed SRM successfully connect back to vCenter.
After the connection was made to vCenter I had to reconnect in the SRM site pairing to update the SRM URL and certificate from the SRM UI console.
I had to complete the exact same steps on the second appliance.
This was a bit of annoying issue to get around as the errors where pretty vague and the logs didn’t really give much information.
Hopefully the rest of my upgrade goes a bit smother than the SRM upgrade.
VMware is depreciating Integrated Windows Authentication in vSphere 7.0. The feature will be removed in a later release. Below is from the VMware KB.
Support for IWA continues to be available in vSphere 7.0 and will be phased out in a future release. Although IWA can still be configured, we highly recommend using AD over LDAP or Federated Identity (AD FS).
In this post we will be going through changing over to using Active Directory over LDAP. We will also be using LDAPS as this is secured with certificates and is much better from a security side and Microsoft are requiring this on applications that use LDAP.
If you haven’t configured a certificate on your domain controller yet to allow LDAPS I would configure this first before proceeding with the swap over to Active directory over LDAP identity provider.
If we check the existing AD IWA we can see the warning that the feature is depreciated.
I usually create a new account for each applications LDAP connections just so I keep track of what account is used where.
For LDAP authentication in a Windows domain a standard account with just domain users right should have enough permission as it best to use least privilege for service accounts.
To confirm in an Windows AD domain is setup to use LDAPS we can use the ldp on a devices that has the active directory tools enabled to confirm LDAPS connection.
Open and click connect and add in the server name, set port to 636 and tick SSL.
If the configuration is retuned then LDAPS is working.
Once we have the account created and confirmed that LDAPS is working we can start setting up AD over LDAP in vCenter.
Since we will be using the same domain name as the IWA source we need to remove this first or it will cause error when trying add the LDAPS source.
Logon to vCenter web client > Menu > Administration > single sign on > configuration.
Under Identity sources select the IWA and click remove.
Click ok to confirm removal.
Once the IWA is removed we can now add the AD LDAP connection.
Click Add in the Identity source page and select Active Directory over LDAP
Add in the required details.
Name: Friendly name for the identity source.
Base DN: Is the level at which search in AD will start for user or groups to search all AD just use the top level or select sub OU to limit the searches.
Domain name: FQDN of the domain
Domain alias: this is the NetBIOS / pre windows 2000 domain name
When I select any domain controller I was getting the below.
Cannot configure identity source due to Failed to probe provider connectivity [URI: ldaps://domainl ]; tenantName [vsphere.local], userName [User] Caused by: Can’t contact LDAP server.
To work around this I had to specific my DC manually.
As I have a certificate issue from an internal certificate authority I will be selecting the CA cert for LDAPS as this should trust any cert issued by the CA on my domain controllers.
Click Add to complete the AD over LDAP identity source.
If we check the websso.log under /var/log/vmware/sso on the vCenter appliance, we can see the certificate being verified when we logon with a domain account.
We have now move from IWA to AD over LDAP all existing groups and roles should still work.
Recently we needed to review some changes and remote console events to check what user was accessing a particular VM and what changes where made.
I find searching event in the vCenter web client is a bit slow, I prefer to use Get-VIEvent as it has multiple parameters that can be used to search and can also use regular expression to filter by patterns.
I decided to do a blog post on how to filter events to show the different options that I use regularly to filter events.
First we need to connect to vCenter sever using a computer that has PowerCLI installed
Connect-VIServer vCenterServer
Once connected we can start to use Get-VIEvent, to return event for a specific object we can use -entity parameter and the object name.
In the below example I am getting the last event for my LAB-Win10 VM
Get-VIEvent -Entity ObjectName -Maxsamples 1
We can also filter by entity and user name is we know the user is tied to the event.
Another option for filtering is to use where-object and search for a specific event message.
Get-VIEvent -Entity VM | Where-Object {$_.FullFormattedMessage -Like "VM started"}
There are addtional properties like host, ComputeResource, datacenter…. that are not return as readable values unless you format the results.
To format the result we can use select-object and create an array to give a name to the property and select the property value. Below will show the datacenter, cluster and host the event was create on.
Get-VIEvent -Entity object -Username User | Select-Object Message,CreatedTime,UserName,@{N="DataCenter";E={$_.DataCenter.Name}},@{N="Compute";E={$_.ComputeResource.Name}},@{N="Host";E={$_.Host.Name}}
Below is an example showing what the event looks like before and after using the array’s to return the readable values.
Once we the events we want we can either review them in the PowerShell console or use Export-csv to export the results.
As part of our VMware 6.7 to 7.0 Upgrade we wanted to audit the existing vCenter server permission. We have a lot of contractors who come in to do work and users who have had permission assigned but these permission are not always removed.
We wanted to get a report that export each of the permission assigned in vCenter.
I could do this manually but this would take a while and is not that easily repeatable so I decided to create a quick script that will export the required information.
The script will be calling two command (Get-VIPermission to export permission and Get-VIRole to export the assigned privileges) and then formation the results.
The script also has some mandatory variables (one for the vCenter server and one for the export path) and there is some error handling incase there is no connection to vCenter server or the export folder doesn’t exist.
There are three type of object in VMware permissions.
Privilege: Allow specific actions (create, delete, manage.. ) or rights to view specific properties
Role : A set of privileges assigned to an object to allow assignment
Permission: Is either a set of a users or groups that have been assigned to a role
If we run Get-ViPermission on we will see all permission returned.
We can select one specific permission by using -principal and expand using format-list. This gives a bit more information but we are missing the assigned privilege’s.
This is where we use Get-VIRole as this has a property that shows privileges that have been assigned to the role.
There was recent VMware local privilege escalation vulnerability in VMware tools below 11.2.6 and below. See VMware advisors VMSA-2021-0013 (vmware.com).
The vunerablity has been fixed in VMware tools version 11.3
We needed to update the version of VMware tools running manually as the tools are not currently included in any other of our standard baselines we apply to our hosts.
I decided to do a to a post on how to update the version of VMware tools using VMware Lifecycle manager baseline as it a little bit different that VMware Update Manager.
First we need to go to Lifecycle Manager, open the vSphere web console > Menu > Lifecycle Manager
In Lifecycle manager the tools should be synced as previously in VMware Update Manager the tools need to be manually uploaded.
To quickest way I find to check the latest tools have been synced is by click on image depot and select components.
We could also check under updates and turn off show only rollup updates. (If the tools required a reboot it would show under impact)
Next we will create a baseline to apply the latest tools.
Go to baselines and select new baseline.
Give the baseline a name and select patch
Untick Automatically update this baseline
Untick show only rollup updates and filter for VMware tools, there will probable be a different VMware tools for 6.x and 7.x so check before adding to the baseline.
Click next and complete the baseline creation.
We can check the current tools status by going to the esxi host > Updates > VMware tools and check status.
We can now apply the baseline and run the check again and it should show as out of date.
The baseline can be applied either directly to the ESXi host or to the cluster we will be applying to the cluster as it save time having to apply to each host indiduvally .
Go to the cluster > Updates > attach and select attached baseline.
Select the VMware tools baseline and attach.
Next run a compliance check on the ESXi host.
Check the baseline status.
Next we will remediate the baseline to apply the latest tools.
If there are no issue with the pre-check click remediate.
Once the remediation is done the tools should show as compliant.
Once applied the VM should now pickup that there is a new tools version available.
The tools can now be applied to the VM either using a script, update on reboot or manually.
During a recent audit we wanted to confirm what users had MFA enabled in Office 365. We use conditional access policy to enforce MFA.
We wanted to check each users to see if they had setup MFA and had a method configured. We also wanted to get information on licensing status and assigned licenses.
The only pre-req for using the script is that the MSOnline Powershell module is installed.
To install the MSOline module open and admin PowerShell windows and run
Install-Module -Name MSOnline
To confirm the module is installed run the below command.
Get-Module -ListAvailable MSOnline
First we need to connect to MS Online to do this run
Connect-MsolService
Once connected to check the MFA status I will be using the StrongAuthenticationMethods properties as if MFA is configured for the user there will be a default method set.
For users that haven’t configured MFA no StrongAuthenticationMethods is set.
Recently a request came in from our security team to audit recently create, deleted AD object, accounts due to expire (this is for third party users) and modified / created group policy objects so that they would be able to trace the changes happening in Active Directory.
I decided to write a PowerShell script that will export the required information and then send a the csv export to the user that require the information.
This could also be used to import the data to a dashboard by either using the CSV files or if the dashboard can use direct PowerShell script like PowerBI.
First there are some mandatory parameters. Exportpath and domain.
To allow the script to be run without emailing the csv I have left the smtpserver, to and from address as not mandatory parameters.
The script used two different modules
Group Policy:
ActiveDirectory:
To install these go on a Windows server go to add roles and features and select Group policy Management
and under RSAT enabled the Active Directory module.
Once all the features are enable we can run the script.
I have set the default time to last 7 days but if you want to go back further then update the date value.
To run the script so that it just export local without email the reports use the below.
