Checking LAPS Password Cross-Forest Using PowerShell

We have been using LAPS for the last year or so and for the most part it is a great tool for managing local admins password.

In our environment we have multiple forest with trusts relationships and the one main issue that we have is that the LAPS UI client can’t check the local admin password of devices in other domains, unless we logon as a user in that domain and run the LAPS UI as that user.

We can use PowerShell but this requires either the commands to be run from a devices in the domain with the LAPS PowerShell module or can be done using remote ps session but this is not something everyone on the service desk would know how to do or will have rights to do.

Enter-PSSession dc.domain.local
Get-AdmPwdPassword -ComputerName "computer" | Select-Object Password,ExpirationTimestamp

We could also get the AD attributes by using get-adcomputer comandlet but this requires the AD PowerShell module.

Get-ADComputer -Identity "computer" -Server "DC" -properties ms-mcs-admpwd,ms-mcs-admpwdexpirationtime | select-object ms-mcs-admpwd,ms-mcs-admpwdexpirationtime

I wanted to try create a script that would allow them to check the other domains without the needing to know how to do PS remoting or having modules installed.

To get around using the AD module we will be using ADSI search instead as this a default part of PowerShell. I used the below blog to get started.

Use the PowerShell [adsiSearcher] Type Accelerator to Search Active Directory | Scripting Blog (microsoft.com)

Hey, Scripting Guy! How Can I Search Active Directory from Within Windows PowerShell? | Scripting Blog (microsoft.com)

To query the remote doamin using adsisearch we first need to have a way of converting the domain name to the format that adsisearch requires.

This can be done using DirectoryContect class

System.DirectoryServices.ActiveDirectory.DirectoryContext

System.DirectoryServices.ActiveDirectory.Domain

DirectoryContext Class (System.DirectoryServices.ActiveDirectory) | Microsoft Docs

The first part of the script gets the domain details and returns the objects back to the search variable.

Next part we will filter the results based on the client variable to return only the required computer details.

Once the script is working, I just needed to create some mandatory parameters and create a function called Check-LAPS so that it’s easier for people to use without having to hardcode anything in the script and has some help and examples.

Below is the link to the full script.

Scripts/ActiveDirectory/LAPS at master · TheSleepyAdmin/Scripts (github.com)

Below shows how the script is run and the results.

I used the above script with WPF to create a GUI that is now complied in to an exe. I though this would be a lot easier to pass on the service desk as all they need to do is run the exe.

This was compiled using ps2exe PowerShell module.

MScholtes/PS2EXE: Module to compile powershell scripts to executables (github.com)

All they need to do is put in the domain name and computer name.

This should then return the LAPS password.

I will do a different post on creating the GUI itself in as this was my first GUI and I would like to create a few other.

Exchange Online PowerShell Basic Authentication Connection Issue

Today I ran in to an issue where I couldn’t connect to Exchange Online PowerShell. I could connect to other service without issue (Office 365, Azure AD….).

When trying to connect to Exchange online I was getting an warning about basic auth.

At first I thought this was to do with Microsoft deprecating basic authentication

Exchange Online deprecating Basic Authentication – Microsoft Lifecycle | Microsoft Docs

but after reading the document it says that this doesn’t affect Exchange ActiveSync (EAS), IMAP, POP, and Remote PowerShell.

I then read the Exchange Online v2 module prerequisites

https://docs.microsoft.com/en-us/powershell/exchange/exchange-online-powershell-v2?view=exchange-ps#prerequisites-for-the-exo-v2-module

In this there is a part that winrm needs to be enabled for basic authentication even though the module uses modern authentication.

This had been working for a long time without issue so I didn’t know what exactly had changed so I decided to check the winrm config

To check the winrm configuration run the below command

winrm get winrm/config/client/auth

In my case it was being set by a GPO. We had recently applied the Windows 10 2004 baseline policy to all our Windows 10 devices so I decided to check the setting in that GPO.

Security baseline (FINAL): Windows 10 and Windows Server, version 2004 – Microsoft Tech Community

There is a spreadsheet in the document folder that has all the policy settings. In the computer policy we can see that Allow basic authentication is set to disabled.

Since this is a security risk (there is no encryption with basic authentication) and only a few device really need to connect, We decided to create a new policy that enabled this for only admins that will need to use Exchange Online PowerShell and use security filtering to only apply to a security group with the required devices.

Once basic authentication was re-enabled I was able to connect without issue.

If you have this issue and apply either the CIS or the Windows 10 baseline policy this could be the cause.

VMware PowerCLI Integrated Authentication Issues

Recently we have been having an issue with VMware PowerCLI not passing through the users credentials when running Connect-viserver to connect to our vCenter servers.

This has been causing problems when trying to use scheduled task to automated reports and run remediation task like removing old snapshots or reporting on VMware Tools versions as it is prompting for credentials.

For integrated authentication to work, the vCenter servers needs to be setup to allow single sign on for the domain that you will be connecting from, so confirm that your Active Directory Identity source is added and that SSO works from the web client. If not, complete this first before trying to use PowerCLI with integrated authentication.

I have SSO configured and tested so this wasn’t my issue.

When using the older version of PowerCli version 6.5 and below we had no issue with integrated authentication and would connecting to vCenter server without prompting me for credentials.

We could continue to use the old version of PowerCLI but we would be missing out on improvement and new commandlets so I wanted to try and get the newer version working for automated task.

First we install the VMware.PowerCLI module using Install-Module VMware.PowerCLI, the current version is 12.1

When I tried to connect using this version of PowerCLI I get prompted for a user name and password and this is what is stopping my automated task from running.

There are a few ways to workaround the prompt and this can also be used to confirm if SSO is working correctly. One way would be to add credential store item that can be used for connecting.

To add this use the VICredentialstore commandlet.

New-VICredentialStoreItem -User domain\username -Password Password -Host vc.domain.local -File C:\Temp\vicreds.xml

This outputs the credential to an xml file that can then be imported and called using Connect-VIserver.

$logon = Get-ViCredentialStoreItem -File C:\Temp\vicreds.xml
Connect-VIServer -Server $logon.Host -User $logon.User -Password $logon.Passwor

This works but I don’t really want to have a xml file that has information saved to it and someone might remove the folder or file by mistake.

The other way would be similar but using a txt file with the converted to a secure string password but again this relies on a file which is not ideal and not really all that secure.

The last option and one that I wouldn’t recommend at all is to hard code a username and password in the script.

So now that we have gone through some work arounds I decided to have a look at the actually problem .

The above proves that authenticating against AD is working so I knew it wasn’t an account or SSO issue so it had to be an issue with PowerCLI itself.

I connect using a my user name and passwords and it connect without issue.

I then check the VPX log under /storage/log/vmware/vpxd to see if there are an issue but I didn’t see any issues.

Next I tried to use the -verbose parameter to return more information on what exactly connect-viserver was doing,

This then returned an error for TLS.

I next checked the settings on PowerCLI configuration settings.

To check the PowerCLI configuration use

Get-PowerCLIConfiguration

When checking the configuration the Invalidcerificaeaction was set to unset.

I changed this setting to warn instead of unset

Once this setting was changed I can now connect to PowerCLI with integrated authentication, I do get a long warning message though I could set this to ignore and this returns no warning or error.

There are two ways around this instead of changing the above setting. One is to install the the certificate as a trusted root certificate so that the cert is trusted.

Or if you have an internal certificate authority you to replace the default VMware cert with an internal cert.

Once this is done the connection work without requiring manual intervention and my automated scripts can be run using scheduled tasks again with the latest version of PowerCLI. Hopefully this will be helpful to anyone else having this issue.

Microsoft Graph API Result Size Limit

Recently we have been running some Microsoft Graph API queries and were not getting back all the results expected.

Microsoft Graph seems to have a limit of 1000 results for signin logs, when the limit is reached graph will then start paging the result and adding them to @odata.nextLink property.

There is a Microsoft doc on paging in Graph.

Paging Microsoft Graph data in your app – Microsoft Graph | Microsoft Docs

There are different limits depending on the API.

Below is an extract from the paging doc.

Paging behavior varies across different Microsoft Graph APIs. Consider the following when working with paged data:

  • Different APIs might have different default and maximum page sizes.
  • Different APIs might behave differently if you specify a page size (via the $top query parameter) that exceeds the maximum page size for that API. Depending on the API, the requested page size might be ignored, it might default to the maximum page size for that API, or Microsoft Graph might return an error.
  • Not all resources or relationships support paging. For example, queries against directoryRoles do not support paging. This includes reading role objects themselves as well as role members.

The @odata.nextLink then needs to be called to get the addtional results.

When we run a query and the results are under the limit there will be two objects returned

@odata.context and value

If we run a query that goes over the max results limit of 1000 the value object is changed to @odata.nextLink

If we use count on the $Results variable it returns 996 items.

To query the addtional results we need to loop through each page to extract the results.

To loop through the results we can create a new variable with the @odata.nextLink and use while loop to go through the variable to get results from the pages.

When we do this the $results count go up to 1142

Below is the full script to get the addtional results.

$ApplicationID = ""
$TenatDomainName = ""
$AccessSecret = Read-Host "Enter Secret"


$Body = @{    
Grant_Type    = "client_credentials"
Scope         = "https://graph.microsoft.com/.default"
client_Id     = $ApplicationID
Client_Secret = $AccessSecret
} 

$ConnectGraph = Invoke-RestMethod -Uri "https://login.microsoftonline.com/$TenatDomainName/oauth2/v2.0/token" `
-Method POST -Body $Body

$token = $ConnectGraph.access_token

$LoginUrl = "https://graph.microsoft.com/v1.0/auditLogs/signIns"
$LoginResults = Invoke-RestMethod -Headers @{Authorization = "Bearer $($Token)" } -Uri $LoginUrl -Method Get

$Results = @()
$Results += $LoginResults.value

$Pages = $LoginResults.'@odata.nextLink'
while($null -ne $Pages) {

Write-Warning "Checking Next page"
$Addtional = Invoke-RestMethod -Headers @{Authorization = "Bearer $($Token)" } -Uri $Pages -Method Get

if ($Pages){
$Pages = $Addtional."@odata.nextLink"
}
$Results += $Addtional.value
}

If using the Microsoft Graph SDK the process is much simpler.

Once connect to using Connect-Mgraph we can then run

Get-MgAuditLogSignIn | 
Select-Object UserDisplayName,UserPrincipalName,IPAddress,AppDisplayName,ResourceDisplayName,ClientAppUsed,CreatedDateTime

The $signins results variable returns the same 996 results as we got with the above invoke-restmethod graph api query.

To get all the addtional results all we need to do is add the -All parameter and this should then return all results.

This shows the two different method to get the larger results from GraphAPI queries.

Upgrading ESXI Host From 6.7 to 7.0 Using vSphere Lifecycle Manager

In this post we will go through using vSphere Lifecycle Manager VLM to upgrade an ESXI host from 6.7 to 7.0. Lifecycle Manager replaces vSphere Update Manager in vCenter server 7.0 the process is pretty much the same as in VUM.

Before upgrading to a new version of ESXI first step should be to check VMware compatibility, this can be done by either checking the VMware compatibility matrix.

VMware Compatibility Guide – System Search

Or in VLM we can now use Hardware compatibility, This syncs a list of compatible hardware and this can then be checked directly on the host to verify if the ESXI host hardware is supported.

To check using VLM, Open the vSphere web client > Menu > Lifecycle Manager

We then need to sync the hardware compatibility list. Click Actions > Sync HCL

Once synced we can run a compatibility check from the Host > Updates > Hardware Compatibility

After the compatibility is all confirmed and no issues are found, we can go ahead with the upgrade.

Next we need to download the ISO image for ESXI 7.0 that will be imported to VLM and used in the upgrade baseline.

Download VMware vSphere – My VMware

To import the ISO go back to VLM > Imported ISOs > Import ISO

Click browse and select the ESXI ISO that was downloaded earlier.

The ISO will then start to import.

The ISO should now show under Imported ISOs

Next we need to create a new upgrade baseline, this can either be done under baseline or by selecting the image and clicking on New Baseline.

Give the baseline a name

Select the ISO to be used

Click next and finsh off the baseline

Check baselines to confirm the creation has completed.

The baseline can be assigned to the host individually or the cluster. I am going to apply to the cluster. Go to the Cluster and select Updates > Attach.

Select the upgrade baseline.

We can check the compliance for the baseline for all host in the cluster.

To remediate you can either do this from the cluster or on the individual host. I will remediate from the host itself.

Go to the host, Select Updates > Baseline > Upgrade Baseline > Remediate

Accept the end user agreement.

The upgrade will do a remediation pre check before allowing the upgrade.

Once you click Remediate, the upgrade task will start

The host should now reboot and start the upgrade.

When we check the baseline the host should now show as compliant and running ESXI 7.0.

Upgrading a host using VLM is a straight forward process and makes it easy to keep your ESXI host at the latest release version.

Windows Query Remote Group Membership Using PowerShell

We needed to do an audit on privileged group membership on workstations. There are many way to do this but using PowerShell to query WMI remotely was the method that we choose as we had to do a few thousand and I wanted to do the checks in groups. .

I decided to write a script that will take the list of devices to be checked from a txt file and then use parameter for the export path and groups to be search.

In this script we will use the class Win32_Group below is a link to the Microsoft Docs

Win32_Group class – Win32 apps | Microsoft Docs

When querying WMI we can use filters to limit the results. If you run query without filtering the command will return all groups even those in AD. See below command and the returned results.

Get-WmiObject  -Class Win32_GroupUser | Select-Object GroupComponent,PartComponent,PSComputerName

If we use a filter we can then reduce these by using domain which will be the local machine name and the local group name.

Get-WmiObject -Class Win32_GroupUser -Filter "GroupComponent=""Win32_Group.Domain='LAB-Host01',Name='Administrators'""" | Select-Object GroupComponent,PartComponent,PSComputerName

Now that we have a filter we can use variables to specific the host name from the text file with computer names and group names.

In the script these will be set using the complist and groups variables.

Below are two examples of how the script can be run against one group or multiple groups

.\Get-RemoteGroupMembers -CompList c:\Temp\Comps.txt c:\Temp\Results -groups “Administrators”

For multiple groups just add a , between the names and double or single quotes if there are spaces in the group name.

.\Get-RemoteGroupMembers -CompList .\Comps.txt -exportPath .\ -groups “Administrators”,”Remote Desktop Users”

The full script is on my github page. As always any scripts should be tested before run in production.

Scripts/Get-RemoteGroupMembers at master · TheSleepyAdmin/Scripts (github.com)

How to Update vCenter 7.0 Virtual Appliance

The process of patching vCenter appliance has been become a lot easier in recent years. Keeping vCenter fully up to date is important for stability and security.

In this post we will go through the process of patching for vCenter 7.0 to the latest version using the GUI connecting to the internet. You can also update using command line or by downloading and mounting the ISO image.

First we need to logon to the admin management console.

https://vcenter.domain.local:5480

Use the root logon that was configure when setting up the appliance.

First steps is to confirm there is a valid backup of the appliance.

Click backup now.

There is an issue with vCenter 6.7U2 and above where it fails on SMB with SMB location is invalid if SMBv1 is disabled. So if you get that error you can just enabled SMBv1 temporarily or enabled OpenSSH on Windows to allow SSH connection which is what I would do in production.

Once completed the backup should kick off.

I also usually take a snapshot as that is the quickest recover option.

Once we have a back up, we can now continue with the updating the appliance. The current version of the appliance is 7.0.0.10100.

Go to Update and click check updates

Once the check is completed select the latest patch. Select either stage only or stage and install if you want the update to be installed straight away. The version we will be updating to is 7.0.1.00200.

Accept the end user agreement.

This will run a pre-check on vCenter before the upgrade will continue. Once no issue are found put in the administrator’s password.

Tick the box to confirm that a backup has been completed.

The install will now start and can take a hour or so to complete.

During the upgrade there will be outages to vCenter while services restart.

a

vCenter should now be update to the latest version.

VMware vRealize Log Insight Adding Windows Servers

In the last post we went through querying logs using the different filter options and how to create a dashboard using the queries in vRealize Log Insight (vRLI).

Part 1: VMware vRealize Log Insight Install and Configure – TheSleepyAdmins

Part 2: VMware vRealize Log Insight AD Authentication and Role Based Access – TheSleepyAdmins

Part 3: VMware vRealize Log Insight Query Logs and Creating Dashboards – TheSleepyAdmins

In this post we will go through adding a Windows server agent and adding the content pack

To add a server we need to download the agent by logging on to vRLI > Administration > Agents and click on download log insight agent.

Select the required agent to be downloaded.

Once downloaded copy the installer to the server and run.

Enter in the FQDN or IP address for the vRLI server is not already there and click install.

To install the agent using command line the below can be used just need to update the path and msi file name.

Path to msi\VMware-Log-Insight-Agent-8.2.0-16776561_*.msi /quiet

You can also add some command line switches to change the default install

Path to msi\VMware-Log-Insight-Agent-8.2.0-16776561_*.msi SERVERHOST=LAB-vRLI.thesleepyadmin.local LIAGENT_AUTOUPDATE=yes /quiet /lxv* vRLI_Agent_install.log

Command-line Options for vRealize Log Insight Agent Installation on Linux (vmware.com)

Once installed, the agent should now show under the agent tab in vRLI.

Next we need to add the Windows content pack to vRLI, Go to Content Packs and search for Microsoft Windows

Click on the content pack and install

Now that the content pack is added, we can copy the Microsoft – Windows to a new group so that its assigned to Windows agents.

Select copy template

Give the Agent group a name and description

Once copied you can change the settings if required or turn off some events if there not required, in this we will be leaving them as default.

Add a filter so that the Windows servers are added to the agent group. This can be done by Hostname, IP, OS or version.

Click save new group to finish.

It can take a little while for the agent configuration to update and for events to start being sent.

Once they do start to send events you should see the counters update.

We can now go to Interactive Analytics and query the events logs.

If there are different application specific events logs that need to be added they can be added to the existing group or a different agent group can be created.

To add addtional event logs to the existing agent group, go back to the agent group.

Go to build and on Windows Event Log click new.

Give the Windows Event Log a name

Copy the event log name from Windows event viewer and put this under Windows Event Log Channel in vRLI.

Click save agent group.

Now once a task is run the events should now show in vRLI.

This concluded the series on vRealize Log Insight, going through this shows that log insight is a good tool for centrally managing and monitoring system logs and events and can be used for VMware, Windows and Linux servers. Hope that this series of post have been helpful.

VMware vRealize Log Insight Query Logs and Creating Dashboards

In the previous post we went through setting up AD Authentication and role based access in vRealize Log Insight (vRLI).

Part 1: VMware vRealize Log Insight Install and Configure – TheSleepyAdmins

Part 2: VMware vRealize Log Insight AD Authentication and Role Based Access – TheSleepyAdmins

In this post we will be going through querying logs and creating a Dashboard.

vRealize Log Insight collects, imports, and analyzes logs to help with troubleshooting problems with systems, services, and applications.

You can search and filter log events on the Interactive Analytics tab, Logs can be queried using different filters like text, timestamp, source, fields and Regular Expressions

TypeDescription
TimestampThe time when the event occurred
SourceWhere the event originated. This could be the originator of the syslog messages such as an ESXi host or a forwarder such as a syslog aggregation.
TextThe raw text of the event.
FieldsA name-value pair extracted from the event. Fields are delivered to the server as static fields only when an agent uses the CFAPI protocol.

Below is the user guide for vRLI

Using vRealize Log Insight – vRealize Log Insight 8.2 (vmware.com)

First we will be going through querying logs.

Open vRLI > Interactive Analytics, This will show all logs to query use the Add filter button

In the below we are going to query any hosts that have disconnected over the last 48 hours. Below are the two filters I used for the host disconnections.

vmw_vc_alarm_status contains Red

vmw_vc_alarm_type contains host connection and power state

We create a table view for events. Below is for snapshots taken in the last 48 hours , we can then add additional information by clicking on time series button and adding group by fields.

vmc_vc_task_type contains create virtual machine snapshot and virtualmachine.createsnapshot

I wanted to add the users that created the snapshot, source and VM name.

This image has an empty alt attribute; its file name is image-188.png

There are a lot of different event types that can be queried and it will all depend on what event’s you are looking for.

Next we will create a dashboard with a few different queries. To create a new dashboard go to Dashboards > New Dashboard

Give the Dashboard a name and select if it should be shared with all users.

Create a filter and click Add to Dashboard

Give the filter a name

If there are any existing filters that are currently in any other dashboard they can be cloned to your custom dashboard by clicking on the the gear icon and selecting clone, then select your custom dashboard.

Once all the required queries are added to the dashboard we can go to My Dashboards to view.

In the next post we will go through adding a Windows agent and adding the content pack for Windows.

VMware vRealize Log Insight AD Authentication and Role Based Access

In the previous post we went through installing and the initial configuration for vRealize log Insight (vRLI).

Part 1: VMware vRealize Log Insight Install and Configure – TheSleepyAdmins

In this post we will go through the steps required to enabled Active Directory (AD) Authentication integration and setting up role based access groups.

This will allow for central management by using AD for users to access using there domain accounts and access permission to be based of AD group membership.

First step is to configure the AD authentication integration.

Logon to the log insight management console > Administration

Once under Administration go to integration > Authentication Configuration > Active Directory

Add in the AD details, I would generally create a AD service account for each LDAP service as it allows me to manage what account are being used for what service (The account should only need domain users rights and be set to deny interactive log for security).

I would also use Require SSL to encrypt the LDAP connection if your DC has a SSL cert that can be used, if not use standard port 389.

Next we need to go to Administration > Management > Access Control

To view Roles click on the role tab

The below are the default User roles, The default roles are fine for me so I wont be create any custom roles just assign the current roles using AD groups.

Below is each role and the description

OptionDescription
UserUsers can access the full functionality of vRealize Log Insight. You can view log events, run queries to search and filter logs, import content packs into their own user space, add alert queries, and manage your own user accounts to change a password or email address. Users do not have access to the administration options, cannot share content with other users, cannot modify the accounts of other users, and cannot install a content pack from the Marketplace. However, you can import a content pack into your own user space which is visible only to you.
Dashboard UserDashboard users can only use the Dashboards page of vRealize Log Insight.
View Only AdminView Admin users can view Admin information, have full User access, and can edit Shared content.
Super AdminSuper Admin users can access the full functionality of vRealize Log Insight, can administer vRealize Log Insight, and can manage the accounts of all other users.

Next we need to create groups in AD that will be used to allow access. I have create an Admin, Dashboard and Read only groups.

Last step is to add the group to vRLI, Go back to Administration > Management > Access Control and click on New Group.

Add in the group name and select the role to be assigned.

Once all groups are added we can test by adding a user to the group and confirming there access.

This image has an empty alt attribute; its file name is image-177.png

Open a new browser session and open the vRLI web management address and select Active Directory as the identify source.

The user account should be able to logon but only have a limited view compared to a full Admin.

Dashboards:

Administration:

In the next post we will go through querying logs and creating a dashboards.