I have been working on a daily check report for our VMware environment so that we don’t have to manually check each morning.
The report uses PowerCli to generate information and then output the results to a HTML file.
The report requires a few that either the old PowerCLI snapin is available or preferably the PowerCLi PowerShell module.
The script can either be run directly by a users with rights to query vCenter or by setting up a scheduled task.
The following prerequisite will be needed for the script to run.
PowerShell V4 or V 5
PowerCLI 6.0 or later version
vCenter 6.0 or later version
There will also need to be a mail server or relay server available for the report to be emailed.
This has been tested on PowerCLI version 6.0 and above. The version on the server I will be running from is 12.3.0 which is the latest release at this time.
The report checks
VMware tools check
Snapshot older than the specified snapshots days
vCenter Alerts over the last 12 hours
Datastore under specified % free space
There are mandatory parameter that are required for the script to run and send the report.
VCServer = vCenter Server address
SMTPServer = Mail server address
Toaddress = destination email
Fromaddress = sending address
Report Export = folder report will be exported to
There are some variables at the start of the script that can be set to customize the report to only show the required snapshots days and datastore % free. In my case I wanted 3 days and below 20% free on datastores.
I have embedded the html CSS format in the script so it can be update to change the color, font size or font type.
In the previous seven post’s we went through installing and configuring the vROps virtual appliance, connecting to vCenter server, configuring Window Active directory as an identity source, create custom alerts and notifications, creating dashboards, upgrading the appliance to the latest version and requesting / configuring a custom SSL cert.
In this post we will be going through installing the Windows agent and configuring the management pack to alert on Windows server OS level alerts like performance, services and application. This can be useful for monitoring physical servers running Windows.
Below is a link to the VMware document on vROps agent deployment I used for reference.
Once we have the agent, we can deploy to the servers that need to be monitored.
Copy the file to the server and run installer.
Add in the vROps server when prompted to
Next the installer will look for the thumbprint for the cert that is used for vROps. Logon to https://vrops/admin and click on the cert icon on the top right to view the current cert details.
Enter the user name and password that will be used to connect to vROps.
Set the install location the default is to install in c:\ep-agent this can be change if required.
The agent should now start to install.
We can run ep-agent.bat query from the install folder ep-agent\bin to confirm the agent has installed correctly.
Once completed we can check vROps to confirm the agent is reporting back, to view the agent in vROps logon to the web client > Administration > End Point Operations.
To view details for the server go to Environment > Operating Systems > Operating System World > Windows and select the server to view.
Once the server is added we can now monitor disk, CPU, memory and other metrics.
We can also monitor services.
To add a service to be monitored,
Go to server and click on action > monitor os object > monitor windows service
Give the monitor a name, select the object type and add in the service name (this needs to be the actual name and not the display name)
Set the collection interval. Click ok to to create the monitor.
Click on Environment and we can view the service monitor we just added.
If we stop the service the next time the collection runs the service should show a critical alert.
We can add addtional metrics if needed. In this example we might want to see the logical disk space % free.
First we either need to modify the existing policy or create a new policy.
In this example we will be adding a new policy and inherting from the default policy.
Go to Policies and click add, give the policy a name and select where it will be inherit from. Then click create policy.
Go to the policies and click on the policy we just created and go to edit policy.
We will be adding a metric so we will select metrics and properties and enable the required metrics.
% free is under EP Ops Adapter > Windows >Fileserver Logical Disk > Utilization and % Free space (%).
Set the policy state to enabled.
Next we can apply the policy to either the object or if there are a lot of device it would be easier to create and apply to a custom group.
Now we can go to the server and confirm the policy is applied.
After a few minutes we can check the server object we can see the new metric and the data start to be shown.
Now that we have the metric showing next we can create an alert.
First we will need to create a symptom definition. Go to symptom definitions and click add.
Select the metric that will be used and give the symptom a name and set the threshold.
We can search to for the symptom to confirm it exist.
Next we need to create the alert. Go to alert definitions and click add.
Give the Alert a name and select Windows as the base object type.
Next we need to add the symptom we created.
Add a recommendation if any are applicable or create a recommendation (this is not required but can be usefully)
We need to add to a policy in this case it’s the Windows_Server_Agent and create a notification if this is required.
We can search for the alert to confirm it has been created and to view the details.
Now when the server goes below 10% free disk space the server will alert.
Below is what the email notification will look like, we have configured email notification in a previous post so we wont go back over it here.
There are many metrics and alerts that can be configure this is just an example of one type. We can also create multiple alerts so that we get warning alerts at maybe 20% before getting a critical alert.
In the previous six post’s we went through installing and configuring the vROps virtual appliance, connecting to vCenter server, configuring Window Active directory as an identity source, create custom alerts and notifications, creating dashboards and upgrading the appliance to the latest version.
In this post we will be going through requesting and applying a custom certificate. Configuring a custom cert is good practice from security standpoint and also will stop the security warning when access the vRealize web client.
Adding a certificate requires that there is a internal certificate authority that can be used to issue the certificate or we could use a public CA but there would be a cost to that, in this example we will be using a Windows Server CA.
I used the below VMware kb as reference when creating the cert.
First step is to connect to the vROps appliance using ssh connection and to generate the key file and cert request that will be used to generate the cert.
To enabled ssh go to the admin page and enable the ssh status.
If you have not updated the root password on the appliance already then this require to connect by ssh. To do this open a VM console for the appliance and go to login. The default root password is blank so just hit enter and it will prompt for a new password to be set.
Once the above has been completed, ssh to the vrops server I use putty but any ssh client will work.
After connecting I usually create a folder to keep the key file and cert request to they are simpler to find later if I need them again.
Next we need to generate a key file
openssl genrsa -out key_filename.key 2048
Next run the below command to create the certificate request
Enter in the details for the cert. These can also be pre creating using a .config file but I just typed them in to the ssh console.
There should now be a key file and cert request in the folder.
Copy the .csr file as this will be used to generate the cert from the internal CA.
To generate the certificate logon on to the Microsoft CA web enrollment page.
Click submit and advanced certificate request.
Click submit a certificate request
Open the .csr file in a text editor and copy the content to certificate request box and select the certificate template to be used.
Click submit and the certificate should be generated. The cert needs to be downloaded as base 64.
Save the cert. The root CA cert also needs to be downloaded
Once all the cert files and key file are created, they now need to be combined to a .PEM format as that is the required format for vrops.
To combine the cert using Windows using the type command. The order the of the cert needs to be server cert, then key file, intermediate cert (if there are any in my case I only have the root cert) , root cert and then the PEM output.
type server_cert.cer key_filename.key cacerts.cer > vrops.pem
The .PEM file should now be created and is ready to be applied to vROps.
The last step is to apply the certificate, logon to the vROps admin page and go to the certificate icon in the top right.
Click Install new certificate.
Click browse and select the pem file we created. If there are no issue with verifying the pem file it should show as ready to install.
Click install to complete. The page should now reload and when we check the cert it should now be using the custom cert.
In the next post we will go through installing the Windows vROps agent and configuring the Windows management pack.
In the previous five post’s we went through installing and configuring the vROps virtual appliance, connecting to vCenter server, configuring Window Active directory as an identity source, create custom alerts and notifications and creating dashboards.
In this post we will be going through upgrading the vROps appliance to the latest version 8.3. Keeping appliances up to date is good from a security standpoint and also allows for new features and fixes to be applied.
To confirm if the upgraded version of vROps is supported by the existing vCenter version use the VMware interoperability matrix site.
In the previous four post’s we went through installing and configuring the vROps virtual appliance, connecting to vCenter server, configuring Window Active directory as an identity source and create custom alerts and notifications.
In this post we will be creating a custom dashboard. Dashboard can be used to visual your VMware infrastructure, view performance issue and capacity planning / right sizing. Dashboards are made up of views and widgets.
We can create custom dashboards or clone and existing dashboard and modify that to add addtional view or widgets.
To create a new dashboard logon to the vROps web client > Dashboards > Dashboards > Create Dashboard
Give the dashboard a name and we can toggle between view and widgets using the below button.
Start adding the required views / widgets for the dashboard in this dashboard we are looking for performance related.
First we will add a view list that we will use to create an relationship with the other widgets.
Give the view a name and select if the content should be refreshed or not. Since I want to specify the cluster I will be setting self provider to On. If this is not set to on input data object will be greyed out
Next we need to specify the object under the inventory tree. We will be using vSphere Host and Clusters and the object will be vSphere World.
Under output data we will use cluster utilization.
Click save and output should look like the below.
Once we have the view we can add the addtional widgets and start creating the interactions.
No data will be showing till the Interactions are in place.
After adding in the required widgets and click on show interactions.
Now we just connect the LAB_Cluster view to the other widgets we just added.
Now if we select the cluster view we will have alert volume, health data, scoreboard and object relationship information returned.
If we want to share the dashboard with other users we can select share icon
Click on groups and select the group to be included.
Now share icon should show beside the name of the dashboard.
If there are pre existing dashboard that we want to customize we can clone these by going to Dashboards >> Manage Dashboards, select the dashboard you want to clone, click on the three dots and select clone.
Give the dashboard a name and we can then start to modify.
In the next post we will go through scheduling reports and updating vROps appliance
In this post we will be going through the different alert types and configuring actions bases on certain alerts.
Below are the three types of alert in vROPs: Health Alerts: The health alert list is all the generated alerts that are configured to affect the health of your environment and require immediate attention. You use the health alert list to evaluate, prioritize, and immediately begin resolving the problems.
Risk Alerts: The risk alerts list is all the generated alerts that are configured to indicate risk in your environment. Address risk alerts in the near future, before the triggering symptoms that generated the alert negatively affect the health of your environment.
Efficiency Alerts: The efficiency alerts list is all the generated alerts that are configured to indicate problems with the efficient use of your monitored objects in your environment. Address efficiency alerts to reclaim wasted space or to improve the performance of objects in your environment.
Each alert type has four different severity types, info, warning, immediate and critical. The can all be configured
To create a custom alert logon to vROPs web client > alerts > Alert Definitions
Click Add, give the alert a name
We will be using virtual machine so we will
select base object type and select vCenter Adapter > virtual Machine
We want to alert on Capacity so click on advanced.
Impact = Health
Criticality = Symptom based
Alert Type & Subtype = Virtualization/Hypervisor : Capacity
Next we need to add the a symptom that will be used to trigger the alert. If there is no symptom that matches what we want to alert on we can create a new symptom. For snapshot there is only greater than 2 days so we will create a new symptom.
We will be using Virtual Machine: Disk space > Snapshot > Age (Days) and set the value to greater than 5 days.
We can set a recommendation that already exists to not keep snapshots over 72 hours or create a custom recommendation.
We can also apply a policy we will use the default policy.
Complete the wizard to create the new alert.
Next we will configure an email notification instance to allow alerts to be emailed.
Go to Administration > Outbound Settings > Add
Added in the email servers settings.
Click test to validate the mail flow is working.
Click save to to complete.
Once we have the outbound email instance configured, we can setup alerts to send emails notifications.
I created a new alert for Powered off VM so it would be easier to get a alert to trigger to test the email notification.
Go to Alerts > Notification
Next add in the details and select the email instance we setup earlier. I will be alerting for when LAB-Linux01 is powered off so will use object and specify the VM name and alert definition.
Click save to create the notification.
Now once the VM is powered off we will get an notification.
In the next post we will be going through creating a dashboard.
In the previous post in this series we went through installing vROps virtual appliance and connecting to vCenter. In this post we will go through adding an AD authentication source and configuring access groups.
There are 5 different authentication sources that can be added to vROps.
SSO SAML: An XML-based standard for a web browser single sign-on that enables users to perform single sign-on to multiple applications.
VMware Identity Manager: A platform where you can manage users and groups, manage resources and user authentication, and access policies and entitle users to resources.
Open LDAP: A platform-independent protocol that provides access to an LDAP database on another machine to import user accounts.
Active Directory: Specifies the use on Active directory to be used to import users accounts or groups.
Other: Specifies any other LDAP-based directory services, such as Novel or OpenDJ, used to import user accounts from an LDAP database on a Linux Mac machine.
First we need to logon to the vROps web client > Administration > Authentication Sources
Click Add and select the source type required. We use Microsoft AD so we will be using Active Directory.
Give the identity source a display name I usually use the domain name as this make it simpler when view settings. Use basic as this auto-discovers the DC and DN (Distinguished Name).
Add the user account that will be used to for the LDAP connections to the domain. This account should only need to have domain users rights.
I also always create a specific service account to be uses for each application AD integration. I would also recommend using SSL/TLS where possible as this will encrypt the LDAP requests between the appliance and the domain controller.
Click on details to view the auto discovered host and
Click test verify all settings are correct, if set to use SSL there will be a prompt to accept the certificate.
Once the test is successful we can complete adding the authentication source.
Once completed the AD source should show.
Next we will configure the groups in AD that will be used to assign access roles in vROps.
To add the groups they need to be imported from AD and then assign the required role.
Go to Administrator > Access Control > import
Use the search string to check for the groups.
Select the role that will be assigned to the group
Assign the other required roles and select the object that are required for the group.
To test we can open a new session and select the AD authentication source instead of local user.
We can check the domain controller security event logs to confirm the authentication.
Based on the roles assinged the user will only have limited access.
In the next post we will go through configure alerting and create some capacity planning reports that can be used to plan for future compute requirements.
In the next set of post’s we will be going through installing and configure vRealize Operations manager (vROps). I haven’t had to install or configure vROps in a few years so want to go back over it before we replaced our existing deployment.
vROps is a application from VMware that can be used to monitor, optimize and manage VMware management tools like vCentre, ESXi..
There are 3 different editions of vROps.
Standard: Allows management of vSphere only.
Advanced: Adding VMware cloud (AWS / Azure), Operating system monitoring and dashboards.
Enterprise: Give all the advanced features but also allows for application / database monitoring and third party management packs.
Select the version you are installing and then add in the number of vCenter, host, datastores and VM that will have data collected and this will then give you the recommended sizing for your vROps deployment.
In my case it was extra small deployment.
What I always do before deploying any VMware appliance is create a static DNS record. This makes it easier to connect to the appliance after it’s deployed and for some appliance (like vCenter server its a requirement or the deployment will fail.)
To download the required vROps appliance go to my VMware and select the required version.
To deploy the OVA create a new VM in VMware and select deploy VM from OVF or OVA file.
Give the appliance a Name and either drag and drop the OVA file or browse to the location and select.
Select a datastore
Agree to the end user license agreement.
Select a network, deployment type size, disk provisioning (thin or thick) and if VMware will be powered on automatically. Since this is only a single vCenter setup we will be using a small deployment type.
Set the timezone and network IP, gateway, netmask and domain name
Review the settings and complete.
The VM will start to deploy.
Once the deployment is completed, connect to either the IP or FQDN of the appliance to start the setup.
Select either express or new installation. We will be using the express installation as we only have one vCenter.
Set the admin password.
Complete the install
When the deployment completed the vROps logon page should show.
Logon and completed the installation.
Accept the End user agreement.
Enter your product key or use the evaluation.
You can join customer experience or untick to not take part.
Click finish to complete.
vROps is now installed.
In the next post we will go through connecting to vCenter Server, configure Active directory integration and build out some dashboards.
Recently we have been having an issue with VMware PowerCLI not passing through the users credentials when running Connect-viserver to connect to our vCenter servers.
This has been causing problems when trying to use scheduled task to automated reports and run remediation task like removing old snapshots or reporting on VMware Tools versions as it is prompting for credentials.
For integrated authentication to work, the vCenter servers needs to be setup to allow single sign on for the domain that you will be connecting from, so confirm that your Active Directory Identity source is added and that SSO works from the web client. If not, complete this first before trying to use PowerCLI with integrated authentication.
I have SSO configured and tested so this wasn’t my issue.
When using the older version of PowerCli version 6.5 and below we had no issue with integrated authentication and would connecting to vCenter server without prompting me for credentials.
We could continue to use the old version of PowerCLI but we would be missing out on improvement and new commandlets so I wanted to try and get the newer version working for automated task.
First we install the VMware.PowerCLI module using Install-Module VMware.PowerCLI, the current version is 12.1
When I tried to connect using this version of PowerCLI I get prompted for a user name and password and this is what is stopping my automated task from running.
There are a few ways to workaround the prompt and this can also be used to confirm if SSO is working correctly. One way would be to add credential store item that can be used for connecting.
This works but I don’t really want to have a xml file that has information saved to it and someone might remove the folder or file by mistake.
The other way would be similar but using a txt file with the converted to a secure string password but again this relies on a file which is not ideal and not really all that secure.
The last option and one that I wouldn’t recommend at all is to hard code a username and password in the script.
So now that we have gone through some work arounds I decided to have a look at the actually problem .
The above proves that authenticating against AD is working so I knew it wasn’t an account or SSO issue so it had to be an issue with PowerCLI itself.
I connect using a my user name and passwords and it connect without issue.
I then check the VPX log under /storage/log/vmware/vpxd to see if there are an issue but I didn’t see any issues.
Next I tried to use the -verbose parameter to return more information on what exactly connect-viserver was doing,
This then returned an error for TLS.
I next checked the settings on PowerCLI configuration settings.
To check the PowerCLI configuration use
When checking the configuration the Invalidcerificaeaction was set to unset.
I changed this setting to warn instead of unset
Once this setting was changed I can now connect to PowerCLI with integrated authentication, I do get a long warning message though I could set this to ignore and this returns no warning or error.
There are two ways around this instead of changing the above setting. One is to install the the certificate as a trusted root certificate so that the cert is trusted.
Or if you have an internal certificate authority you to replace the default VMware cert with an internal cert.
Once this is done the connection work without requiring manual intervention and my automated scripts can be run using scheduled tasks again with the latest version of PowerCLI. Hopefully this will be helpful to anyone else having this issue.