HP NMI Error with ESXi - Gen8

In the environment many of our ESXi hosts (HP DL380 Gen8's) have generated PSOD's with NMI generated events.  If you are experiencing this issue please UPGRADE your ilo4 firmware to version 1.51.  The issue described is under HP article c04332584.  You can download the latest firmware from HP.



Deploying OVA fails with "Failed to Deploy OVF/OVA package: The operation is not supported on the object"

We recently had an issue deploying either an OVA/OVF file that was created from one of the virtual machines in our environment.  Deploying this OVA failed instantly (after entering cluster locations / datastore) with the following error: "The Operation is not supported on the object".  After much investigation and examining the /var/log/hostd.log file on the ESXi host we were trying to deploy too, we found the line:  "Video Ram size edit is not supported when auto-detect is True."  Editing the Virtual Machine hardware settings and editing the Video Card and changing the setting from Auto-detect to Specify custom settings (This can be anything) and then exporting the virtual machine as an OVA allowed the deployment to work! Very interesting that this setting caused a problem!  Previously we have also encountered this when the Virtual Machine has an ISO mounted when to converting to OVA.

PowerCLI script to check VIBs on a host

I work in an environment where ESXi hosts require a lot of additional software to be installed.  Examples include a specific Cisco VEM module to be installed, the Trend DSA Filter driver etc.  A simple way to check what version or even if the software has been installed on the host is as follows.  This example will check if the Trend DSA Filter is installed.

(get-esxcli -vmhost hostname).software.vib.list() | where {$_.Name -eq "dvfilter-dsa"}

This will return information if the module is installed and give important information such as Versions etc.

VMware vSphere Storage Appliance (VSA) Installation and Removal Tricks

I've recently had the pleasure of playing around with VMware's vSphere Storage Appliance.  I would like to quickly point out some of the troubles I had with the installation so that if you guys come across the same problems they can quickly be sorted.  The installation I was using at the time was "VMware-vsa-all-5.1.3.0-1090545.iso"

 

During the installation if you receive an Error 2896 during the VSA Manager installation it is because of a script problem that doesn't like servers with multiple drives, specifically if you are also trying to install the product on for example D:\ rather than C:\. 

 

Browse to %temp%\sva\VSAManager/installTool and edit the runtool.bat file.  Edit the 9th line down and change it from cd %BUILDDIR% to cd /d %BUILDDIR% this will ensure that the installation will occur.

 

If you receive the error during setup through VSA Wizard (from within the VI Client) "Cannot create VSA cluster:  Adding hosts to HA Cluster failed."  Looking at the VSA Configuration log you will see that EVC mode failed to get configured.  I was running a pair of AMD hosts, it was trying to set the EVC mode to amd-rev-e.  If i manually created a cluster, there was no issue.  This hack allows the install to complete, you can then turn on EVC manually on the cluster afterwards.   Browse to the installation directory of vCenter on which you installed the VSA manager, e.g. "C:\program files\vmware\infrastructure\tomcat\webapps\VSAManager\WEB-INF\classes\" and wtih an administrative notepad, edit "dev.properties" file.  Change the evc.config=true  line to evc.config=false  NOTE that this option will REQUIRE a restart of the "VMware VirtualCenter Server" and "VMware VirtualCenter Management Webservices" services.  You should then be able to run through the VSA setup wizard without a hitch.

 

Remember, your hosts must have 4 NIC's in order for the setup to complete, if you have only a single quad port card, you will have to do a "Brownfield" network setup, otherwise you will recieve the following error:  Cannot create VSA cluster:  Failed to complete network configuration for specified host:  <hostname> , hence revered all other hosts, Could not find 2 NICs on the system.

The setup can be done from this documentation:  VSA Brownfield Setup

 

One final thing to note, is that if you are trying to run the cleanup.bat file (when removing the VSA) edit the file so that any line that reads cd %BUILDDIR% to cd /d %BUILDDIR% and the system I was on had no JAVA_HOME variable set.  So I set it to the following "set JAVA_HOME=D:\vmware\Infrastructure\jre" and that allowed a nice clean up.

 

PowerCLI to enable LockDown Mode on all hosts

Here is a quick and easy way to make sure that LockDown Mode is enabled on all your hosts using PowerCLI (if your environment dictates it)

(get-vmhost * | get-view) | foreach-object -process {$._EnterLockdownMode()}

That will go through every host and enable LockDown mode.  If you see the following error while running the script:

Exception calling "EnterLockdownMode" with "0" argument(s):  "The administrator permission has already been dsiabled on the host (except for the vim user)"

 It just means that the host already has that setting enabled and it can be ignored.

vCenter VPXD.exe process using 100% memory

We recently had an issue in an environment that I am looking after that caused the VPXD.exe (vcenter server service) to use 100% of the memory allocated to a machine, plus all the disk space on the server (as the page file was set to system managed and just kept growing) until the server stopped responding.  After much investigation, the following lines were noted in the vCenter server log (appearing a lot).  The error is as follows:

[05444 error 'Default'] SSL StreamImpl::DoServerHandshake (0000000028a34600) SSL_accept failed with Unexpected EOF
[05444 warning 'ProxySvc'] SSL Handshake failed for stream TCPStreamWin32(socket=TCP(fd=5470) local=<vcenter IP>, peer=<remote client connection>), error=SSL Exception: Unexpected EOF

That error message basically translates to a VI Client instance connecting in to vCenter, when this occurs, each connection consumes additional memory which isn't released.  In our case, we had an incident where the VM Tools instance was ripped out of a management server, this management server used the VMXNET 3 driver.  In its unusual state it was polling the vCenter server a large number of times a minute (like it was flapping). Once VMware Tools was reinstalled on the management server, vCenter calmed down and returned back to a normal amount of memory usage.

VMware VRM Server not able to connect to local vCenter Server

If you install the VRM Server as part of vSphere Replication (this problem is from version 5.0.2) you may see this error in the hms log file.  The HMS logs are located on the VRM itself under /opt/vmware/hms/logs.  You can see how to get these logs from KB2013091.

The error is as follows:

2013-04-12 11:01:14.526 WARN hms.configuration [mai]
(..hms.cfg.VcExtensionChecker) | Extension login failed:java.lang.RuntimeException:

java.util.concurrent.ExecutionException: com.vmware.vim.binding.vim.fault.InvalidLogin:
Inheritied from com.vmware.vim.binding.vim.fault.VimFault:
Inheritied from com.vwmare.vim.fault.InvalidLogin:  Cannot complete
login due to an incorrect user name password.

Now, the problem is that your VRM Server needs to not only communicate to the vCenter on HTTP (TCP 80) it also needs to do the initial connection when the services start on HTTPS (TCP 443).  This was the case in the environment I was working in due to firewalls between the two.  If the VRM Server can only talk to vCenter on 80, it will fail to fully communicate to vCenter.  Once you open up HTTPS between vCenter and the VRM Server, login to the admin page https://<ip of your vrm server>:5480 to restart the services.  It should then connect.  If this is the case for your environment, you will need to follow KB2019120 in order for vCenter to show the health status correctly on the health page for the VRM.  You will also note that KB1009562 which is the Port Number reference for SRM and its' components do not state that this port is required.  I could not get communication functional between the same site vCenter Server and VRM Server until both TCP 80 & 443 were allowed through the firewall to the vCenter server.  I would also allow the VRM server to talk to the remote site vCenter server on HTTP and HTTPS (the kb article only states TCP 80).