VMware vRealize Log Insight Install and Configure

In the next few post we will be going through deploying and configuring vRealize Log Insight (vRLI).

Log Insight is a tool from VMware than can be used for log analytics, log management for both infrastructure / application and help with troubleshooting.

I have been looking at deploying this in production so though I would setup in my lab and see how it works and if it would be usefully to deploy in our production environment.

Below is the link to the minimum requirements for vRLI

Minimum Requirements (vmware.com)

Hardware Requirements

Below is the the required hardware depending the size selected

Preset SizeLog Ingest RateVirtual CPUsMemoryIOPSSyslog Connections (Active TCP Connections)Events per Second
Extra Small6 GB/day24 GB7520400
Small30 GB/day48 GB5001002000
Medium75 GB/day816 GB10002505000
Large225 GB/day1632 GB150075015,000

Network Port Requirements

The following network ports must be externally accessible

PortProtocol
22/TCPSSH
80/TCPHTTP
443/TCPHTTPS
514/UDP, 514/TCPSyslog
1514/TCPSyslog ingestion via SSL only
9000/TCPvRealize Log Insight Ingestion API
9543/TCPvRealize Log Insight Ingestion API (SSL)

Once all the requirement are checked we can start to deploy the appliance.

First step is to download the appliance OVA below is the link to the current

vRealize Log Insight | Log Analysis Tool | VMware

Create a new VM from VMware web client and upload the OVA file.

Select a datastore

This image has an empty alt attribute; its file name is image-1.png

Accept the license agreement

Select a network, deployment type and disk provisioning

Add in the hostname, IP address, gateway, DNS server and root password.

Review the setting to confirm all the settings are correct

The appliance will start to deploy

Once the install has finished it should show the DCUI, with the IP / hostname that was assigned during the deployment

Next we have to configure Log insight. Open a browser and either put in the DNS name if created or IP that was assigned to the appliance.

Click start new deployment

Set Admin credentials and email address.

Next screen will ask to put in a license, I will be using a trial license so this will only last for 60 days

Set the NTP settings.

Configure SMTP settings as required. To test if all settings are correct use the send test mail

Click finish to complete the install.

Next we will configure the vSphere integration.

Add in the vCenter details, click test connection to confirm the details are correct

Save vCenter configuration details.

vCenter server will now show and the collection status should show as collecting

In the next post we will go through configure AD integration and using the log insight dashboards.

Upgrading from vCenter Server Appliance 6.7 to 7.0

In this post we will be going through upgrading from VCSA 6.7 to 7.0. Keeping your appliance at the latest version will give access to new features, feature improvements and security fixes.

When upgrading a VCSA there will be a new appliance VM created. The database and configuration from the existing appliance will be copied during the upgrade process.

Currently in my LAB vCenter Appliance is running 6.7 Update 3l.

Before upgrading to any newer version of VMware it important to check that all products that connect to vCenter (Backup, Reporting or Monitoring tools) all support the latest release.

To check for VMware products you can use the VMware interoperability matrix link below.

VMware Product Interoperability Matrices

This can also be used to view the upgrade path as if you are running 6.0 version of vCenter there is not a direct upgrade to 7.0 and will require a two step upgrade.

There is also a sequence on which products should be upgrade. See below link.

Update sequence for vSphere 7.0 and its compatible VMware products (78221)

For third party (non-VMware) products you will need to check the product support page to verify if the versions are supports with vCenter 7.0

Once everything is confirmed as supported we can go ahead with the upgrade.

First download the 7.0 ISO file from VMware.

Download VMware vSphere – My VMware

Before attempting an upgrade make sure that there is a backup of the appliance that can be used to restore incase of any issue during the upgrade. The source appliance shouldn’t be changed but I would recommend a backup just to be extra safe.

Once download mount the ISO to Windows and go to the \vcsa-ui-installer\win32\ and double click on installer.exe

Select Upgrade

Follow the upgrade wizard

Accept the end user license agreement

Next step needs the vCenter details and the ESXi host or vCenter logons details where the appliance is running. (The hostname of the vCenter was case sensitive and I was getting the below error when using all lower case.)

If a certificate warning appears accept to continue

Give the appliance a name (if you want to re use the existing name the change the current appliances name to something different or there will be an error show when clicking next to continue with the upgrade.)

Select the deployment size

select the datastore

Assign a temporary IP address that will be used while copying data

Review the settings and if all looks to be correct click finish to start the upgrade.

The appliance should now start to deploy

After the deployment of the appliance completes, the next step is to run the configuration stage. This will copy the configuration and data from the existing appliance. Once completed the old appliance will be shutdown.

Click continue to start stage 2 of the upgrade

Click next

There will be a pre-req check done before the upgrade can continue. If there are any errors the upgrade wont be able to procced.

Select the data that will be copied

select if you want to join the customer experience improvement program

Review the settings and tick I have backed up the source vCenter server

Click finish and there will be a warning that the source vCenter will be shutdown

The copy and importing of data can take more than an hour to complete.

Once the upgrade has completed the old appliance should be shutdown and the new appliance running vCenter 7.0 should have all data and configuration copied.

When we logon to the vCenter appliance it will now running 7.0.0

Last step is to verify that all management tools work as excepted after the upgrade.

Configure UnityVSA on VMware 6.7

In the previous post we went thorough configuring and setting up the UnityVSA.

Deploying EMC UnityVSA on VMware ESXi 6.7 – TheSleepyAdmins

In this post we will configure the iSCSI initiators, connecting vCenter to the UnityVSA and configure iSCSI LUN that will be used as shared storage between a LAB virtual ESXi hosts.

First step was to create the iSCSI VMKernal adapter on the ESXi host to allow connection to the Unity.

Logon to vCenter, go to the required host > configure > VMKernel adapters and click Add Networking

Select VMKernel Network Adapter

This image has an empty alt attribute; its file name is image-89.png

Follow the wizard to create the new VMKernal adapter ( I am using the same IP range as my host as this is just a LAB setup but in production this should be a separate physical network)

Next we need to configure the storage adapter, go back to configure > Storage Adapters and click Add Software Adapter

Next we need to configure the adapter settings to point to the SAN in this case the UnityVSA IP.

Once added the adapter will recommend a rescan, we haven’t configure any LUN’s yet so I will do the rescan after that.

Next we need to configure the VMware access and LUN’s on the Unity.

Logon to the web management console > go to VMware > vCenters and click the + button to add the vCenter server

Add in vCenter details, click find and select the required hosts

I wont be creating a VASA provider so I left that un-ticked and completed the wizard

The hosts and vCenter should now show in the Unity console

This image has an empty alt attribute; its file name is image-107.png

Once the host and initiators are added the last step is to configure the LUN’s. This can either be done using block or can be done using VMware storage integration.

If using the VMware integration it will automatically add the datastore to VMware using VMware API’s.

Creating a block LUN is basically the same steps only it doesn’t automate the creation of the Datastore in VMware so in this post we will be using the VMware integration for provisioning LUN’s.

Go to Storage > VMware > Datastores and Click the + button to add

Select block and the VMFS version

Give the datastore a name (This will also be the name that is assigned to the Datastore in VMware)

Select the pool to be used and size of the LUN

Add the hosts that will be configured to access the LUN

I wont be taking snapshots or replication the LUN so I left them un-ticked and complete the wizard

The LUN should now start to be created

There will also be task in VMware showing the Unity starting the rescan of the host HBA’s

If we now check the datastores in VMware the new datastore should show.

VMware and the UnityVSA are now configured and additional LUN can be added to allow for vMotion testing and to setup a cluster as if it where using shared storage.

Deploying EMC UnityVSA on VMware ESXi 6.7

I have been using virtual SAN appliance in my labs for a few years as it is the easiest way to configure shared storage for my LAB cluster and as I don’t have a NAS or another external storage device available.

EMC have a version based on there Unity array and this is the same version we us in production so I wanted to configure this as my virtual SAN.

In the next few posts we will go through configuring a UnityVSA and setting up so that it can present iSCSI LUN’s to VMware.

To download the OVA use the below link, the current version 5.0.3

Data Storage Management Software Downloads | Dell Technologies US

Create a new VM and select deploy using an ova

Select the required datastore

Select the appropriate network

Assign a static IP

This image has an empty alt attribute; its file name is image-66.png

Complete the VM deployment wizard.

The VM will now start to deploy.

Once the appliance was deployed there was no network connectivity and I had to run the intial configuration command.

To connect I used the VMware console. The default logon for the admin account is Password123# but this didn’t work for the console logon.

I had to use service for both the usersname and password.

To configure the IP run the below command.

svc_initial_config -4 “IPAddress Subnet defualtGateway”

I was then able to connect to the Unity Management console. To logon use the default admin/Password123#

Next follow the initial configuration wizard

set the DNS and NTP

Once at the licenses page, we will need the system UUID to be able to register from a trial licenses. The get license didnt work for me so i use

https://www.dellemc.com/en-us/auth/elmeval.htm

Next we configure the storage pools, I added some additional drivers to the VSA to be used as storage pools

Follow the wizard to create the storage pool

This image has an empty alt attribute; its file name is image-98.png

I wont be configure alerts or proxy servers so I leave them as default values.

Next we configure the iSCSI interfaces, these will be used to connect to the ESXi host later

I wont be configuring a NAS server either so will leave that as default also

Finish the configuration

The UnityVSA should now be configured and show simlar to the below.

In the next post we will go through configuring the LUN’s and connecting to vCenter.

Set Custom SSL Certificate on VMware vCenter 6.7 Appliance using Windows CA

In this post we will go through generating and applying a custom SSL cert for VMware vCenter 6.7 Appliance.

When VCSA is installed it generates a self singed cert which cause users to get a cert error when connecting. I used the below VMware doc when setting up the cert.

Replace Machine SSL Certificate with Custom Certificate (vmware.com)

I have already configure a Windows CA so this or another CA will be a requirement before creating the cert.

Before attempting this in production make a backup or take a snapshot to allow quick recovery in case of any issues.

To generate the cert signing request (CSR) , we will use the certificate-manager CLI

To connect I enable SSH on the appliance.

Then used putty client to connect.

First I created a folder that will be used to export the CSR and private key.

Once created we can then run certificate manager using the below.

/usr/lib/vmware-vmca/bin/certificate-manager

We want to replace the Machine certificate with custom cert so select option 1.

Enter the users name and password and select option 1 to generate CSR.

put in the require information below are the fields and values

FieldValue
Countryuse your own country 
NamevCenter FQDN 
OrganizationUse your own company name
OrgUnituse your own 
Stateown state
IPAddressvCenter IP Address 
HostnamevCenter FQDN 
VMCA NamevCenter FQDN 

Once the CSR has been created,

use a client like WinSCP to connect to vCenter and go to the specified folder above in my case this was the cert folder and copy the CSR file.

Now go to your CA to submit the CSR, I will be using the web enrolment.

Click on request a certificate

Click on submit an advanced certificate request.

Click submit a certificate request.

Open the CSR file using notepad and copy the txt

Paste the txt in to the saved request box and select the template I created a custom template for web servers.

Once submitted you should get the option to download the cert selected base 64 encoded as this is required by vCenter.

Once downloaded, upload the cert back to vCenter using WinSCP

Launch certificate manager again and select option 1 but then option 2 to import the custom cert.

/usr/lib/vmware-vmca/bin/certificate-manager

Enter in the details cert file that will be used for vCenter, the private key that was issued with the CSR request and the CA cert file.

You should now the service being updated.

Once completed vCenter should now be using the custom SSL cert.

VMware 6.7 External PSC to Embedded PSC migration

VMware have recently deprecated using external platform service controller (PSC) with vCenter 6.5 Update2 and 6.7

Deprecation of the external Platform Services Controller deployment model (60229) (vmware.com)

Since we where upgrading from Windows vCenter and PSC to appliance’s we had to migrate to new external PSC and then convert the external PSC after the migration.

In this post we will be going through the process of converting and decommissioning an external PSC.

In vCenter 6.7 there is a conversion tool built in to the vCenter web console that allow the PSC to be converted from an external to embedded PSC.

vCenter Server Converge Tool Enhancements in vSphere 6.7 Update 2 – VMware vSphere Blog

The conversion tool is available under Administration > deployments > system configuration

Select the vCenter server that has the external PSC and click converge to embedded

Enter in the logon details for the existing PSC SSO domain and select Join AD if you want to also join the vCenter appliance to AD (use this if the existing PSC has domain joined).

This image has an empty alt attribute; its file name is image-39.png

The converge tools will start and and it will require vCenter servers services to restart, this will cause an outage to vCenter while the service restart.

This image has an empty alt attribute; its file name is image-40.png

Once all the vCenter services have restarted, vCenter should now show as an having an embedded PSC.

The last step is to decommission the old PSC.

Select the old PSC from in the System configuration and use the decommission PSC action.

Click on porceed.

Enter the SSO logon details, view thumbprint and accept the acknowledgment tickbox.

Click decommission and the PSC will start to be removed.

Once completed vCenter will now be running an embedded PSC and the external PSC will be powered off and can be deleted.

If there are other vCenter in enhanced linked mode follow the same process for each additional vCenter

VMware 6.7 PSC Decommission: Failed to get the PSC thumbprint

As part of our recent VMware 6.7 upgrade we where migrating from external PSC to embedded PSC.

All went fine until we tried to decommission the old PSC’s, when trying to view the thumbprint we got the below error:

Failed to get the PSC thumbprint. Ensure PSC port is correct.

With out getting the thumbprint you can’t continue.

We tested accessing the PSC using web browser,

using openssl from the vCenter appliance

openssl s_client -connect PSC01.domain.local:443

We also tested access using telnet to confirm if port 443 was open

curl -v telnet://PSC01.domain.local:443

All test came back fine and there was no ports being blocked.

In the end the issue was caused by using the short name for the vCenter servers https://VCSA/UI

When we changed to the full FQDN, https://VCSA.domain.local/UI this fixed the issue and we where able to complete the decommission. The issue looks to be related to the SAN name on the cert only having the FQDN.

PSC2

VMware ESXi 6.7 Upgrade: Missing dependencies VIBs Error

Recently we have been upgrading some VMware host from ESXi 6.0 to ESXi 6.7, We where applying the image using VMware update manager and a HPE custom ESXi image.

When applying the image we where getting incompatible warring and where not able to apply the image to upgrade ESXi on some hosts.

The issues was related to VIBS but they where not showing in the html 5 client. vibs1

To find the missing VIBS we ended up having to mount the ISO through HPE ILO and try a manual upgrade which did show the conflicting VIBS. vibs2

In our case the VIBS causing issue was the below.

Mellanox_bootbank_net-mlx4-core_1.9.9.8-10EM.510.0.0.799733
Mellanox_bootbank_net-mlx4-en_1.9.9.0-10EM.510.0.0.799733
Emulex_bootbank_scsi-lpfc820_10.5.55.0-10EM.500.0.0.472560
Mellanox_bootbank_net-mst_2.0.0.0-10EM.500.0.0.472560

The issue seem to be related to older hosts that where previously upgraded from ESXi 5.5.

VMware recommends doing a fresh install if possible but in this case that was not possible, but if you get this error and can do a fresh install that would be the better option. If you can’t do a fresh install below is how I checked to see if the VIBS where in use and how to remove them. 

Next we needed to find out if the VIBS where in use by either the storage or network adapters,

below is the VMware KB that explain how to do this.

https://kb.vmware.com/s/article/1027206

To get the list of storage and network adapter use the esxcli commands

esxcli storage core adapter list

esxcli network nic list

To get check the VIBS version we can use

esxcli software vib list | grep Mel

esxcli software vib list | grep scsi-lpfc820

vibs3

Once we know the version numbers of the VIBS, we just need to confirm they are not used and if not used remove them.

If they where in use we would need to look at creating a custom image or wipe and reload the ESXi host.

We use esxcli to view if the drivers are in use and what version each is

esxcli system module list | grep lpfc820, esxcli network nic get -n vmnic0

vibs4

Once we confirm that none of the VIBS are required the last step is to remove each one. Below is the KB from VMware on removing VIBS.

https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.upgrade.doc/GUID-7FFEBD91-5D82-4E32-93AB-F10D8BFFECAA.html

There might be some VIBS that have dependency’s on others in our case the net-mlx4-core needed to be removed after the net-mlx4-en as it was dependent on it.

To remove we use esxcli software vib remove -n ‘vibname’

vibs5

Reboot the ESXi host if required,

After the reboot, scan the host again from updates tab in VMware vSphere web client and it should now show as non-complaint and not incompatible. vibs6

The host should now upgrade as normal.

HPE Gen 10 SUM Integrated Smart Update Tools VMware

I was installing some new ESXi host using HPE Gen 10 blade servers and was getting a warning when trying to update the firmware using HPE SPP (Service pack for proliant). The issue is due to HPE changing from installing updates directly to the ESXi OS level and instead using HPE ILO.

HPE03

If iSUT is not installed the update will be staged on the host but wont install. To install, download iSUT from the HPE Support site. Below is the link to version 2.3.6 which was the version I used. ISUT_Tool

HPE15Once download and extracted we need to copy the file to the ESXi host, the easiest way to copy is by enabling SSH on the host and using WinSCP.     HPE04I created a folder called hpe_isut on the ESXi host to copy the file to. HPE05Once the files are copied over, use either SSH or ESXi Shell to install the tools. I used SSH with putty as it was easier.

To install use esxcli command, this is the command I used.

esxcli software vib install -d /hpe_sut/sut-esxi6.0-bundle-2.3.6.0-16.zipHPE06Once the install has completed, a restart is required to finish the install. After the reboot the next step is to set the iSUT mode there are 4 <OnDemand/AutoStage/AutoDeploy/AutodeployReboot>

I chose AutoDeploy, to set the mode use the below command.

sut –set mode=AutoDeployHPE09

Once this has completed, run the inventory again from the SPP and the warning should now be gone and the firmware and driver updates should now apply. HPE10

HPE14

Install and Configure VMware NSX

Recently we have been looking to implement zero trust networking. One way to achieve this was to use physical firewall and multiple VLAN’s to break out traffic and restrict access to each VLAN this would take a long time to complete and is quite difficult to manage.

It would require adding between 30 to 60 additional VLAN to our physical servers and VMware and re assinging IP to each server which would cause a lot of downtime.

As an alternative to this I have been looking at VMware NSX to try achieve this same segmentation without the need to redesign the entire VMware networks.

NSX consists of multiple components under different planes like management, control, and data plane’s below is an image of the different plane’s. 

In the next set of posts I am going to go thorough install and configuring a basic NSX deployment. I will be setting this up in a Lab environment and will use nested ESXi and appliances.

It is recommended to have NSX installed on its own management cluster along with vCenter.

First step is to download the OVA for NSX current version is 6.4.4

https://my.vmware.com/web/vmware/details?productId=417&downloadGroup=NSXV_644

below are the system requirments to deploy NSX

NSX Component Hard Drive Memory vCpu
NSX Manager 60 16 4
NSX Controller 20 4 4

NSX 6.4.4 is not supported on vSphere 5.5 below are the supported and recommed verison of vSphere to run NSX 6.4.4:

  • For vSphere 6.0:
    Supported: 6.0 Update 2, 6.0 Update 3
    Recommended: 6.0 Update 3. vSphere 6.0 Update 3 resolves the issue of duplicate VTEPs in ESXi hosts after rebooting vCenter server. SeeVMware Knowledge Base article 2144605 for more information.
  • For vSphere 6.5:
    Supported: 6.5a, 6.5 Update 1
    Recommended: 6.5 Update 1. vSphere 6.5 Update 1 resolves the issue of EAM failing with OutOfMemory. See VMware Knowledge Base Article 2135378 for more information.

Once the OVA is downloaded logon to vCenter right-click on datacenter and deploy OVF Template.

NSX6_1

Select the location of OVANSX2

Give the appliance a nameNSX3

Select the Cluster that will run the applianceNSX4

Click next NSX5

Accept the licence agreement and click continueNSX6

Chose Thick ProvisionNSX7

Select the network that will be used for the management networkNSX8

The next screen is where all the customization will be setup

Appliance Password:

HostName:

Network settings: management IP, subnet, gateway, DNS and NTP. Leave blank if  you want to use DHCP but its recommend to use static addressesNSX9NSX10

Once all setting are configured click next and confirm all settings on the last screen. Once finished the OVA should start to deploy. (Note that this failed the first time for me as I selected a host and there seems to be an issue with this in vCenter 6.7, once I selected the cluster the OVA deployed without issue)NSX11

Once the OVA had been deployed I decided to edit the memory size as I was running low on memory so I change it from 16Gb to 8Gb but for production this should be left at 16Gb.

After this you can connect using DNS name configured above or through the management IPNSX12

The last step in this post is to connect NSX to vCenter

Logon using  admin and the password specified in the config of the OVA

Click on Manage vCenter Registration NSX13

both the lookup and vCenter server connection will need to be configuredNSX15

Add vCenter server and user name / passwordNSX16NSX18

There will be a prompt to trust the vCenter certificate click yes to continueNSX17

Once configured both status should show as connectedNSX19

Open the vCenter web client and once logged on there should now be an addtional tab for Networking & Security. (At the time of this post this option is only available in the Flash version of the Web client not the HTML 5 version) 

NSX20NSX21

In the next post we will start to configure the NSX and controllers.