Thursday 19 September 2013

PowerCLI script to check VIBs on a host

I work in an environment where ESXi hosts require a lot of additional software to be installed.  Examples include a specific Cisco VEM module to be installed, the Trend DSA Filter driver etc.  A simple way to check what version or even if the software has been installed on the host is as follows.  This example will check if the Trend DSA Filter is installed.

(get-esxcli -vmhost hostname).software.vib.list() | where {$_.Name -eq "dvfilter-dsa"}

This will return information if the module is installed and give important information such as Versions etc.

VMware vSphere Storage Appliance (VSA) Installation and Removal Tricks

I've recently had the pleasure of playing around with VMware's vSphere Storage Appliance.  I would like to quickly point out some of the troubles I had with the installation so that if you guys come across the same problems they can quickly be sorted.  The installation I was using at the time was "VMware-vsa-all-"
During the installation if you receive an Error 2896 during the VSA Manager installation it is because of a script problem that doesn't like servers with multiple drives, specifically if you are also trying to install the product on for example D:\ rather than C:\
Browse to %temp%\sva\VSAManager/installTool and edit the runtool.bat file.  Edit the 9th line down and change it from cd %BUILDDIR% to cd /d %BUILDDIR% this will ensure that the installation will occur.
If you receive the error during setup through VSA Wizard (from within the VI Client) "Cannot create VSA cluster:  Adding hosts to HA Cluster failed."  Looking at the VSA Configuration log you will see that EVC mode failed to get configured.  I was running a pair of AMD hosts, it was trying to set the EVC mode to amd-rev-e.  If i manually created a cluster, there was no issue.  This hack allows the install to complete, you can then turn on EVC manually on the cluster afterwards.   Browse to the installation directory of vCenter on which you installed the VSA manager, e.g. "C:\program files\vmware\infrastructure\tomcat\webapps\VSAManager\WEB-INF\classes\" and wtih an administrative notepad, edit "" file.  Change the evc.config=true  line to evc.config=false  NOTE that this option will REQUIRE a restart of the "VMware VirtualCenter Server" and "VMware VirtualCenter Management Webservices" services.  You should then be able to run through the VSA setup wizard without a hitch.
Remember, your hosts must have 4 NIC's in order for the setup to complete, if you have only a single quad port card, you will have to do a "Brownfield" network setup, otherwise you will recieve the following error:  Cannot create VSA cluster:  Failed to complete network configuration for specified host:  <hostname> , hence revered all other hosts, Could not find 2 NICs on the system.
The setup can be done from this documentation:  VSA Brownfield Setup
One final thing to note, is that if you are trying to run the cleanup.bat file (when removing the VSA) edit the file so that any line that reads cd %BUILDDIR% to cd /d %BUILDDIR% and the system I was on had no JAVA_HOME variable set.  So I set it to the following "set JAVA_HOME=D:\vmware\Infrastructure\jre" and that allowed a nice clean up.

Sunday 5 May 2013

PowerCLI to enable LockDown Mode on all hosts

Here is a quick and easy way to make sure that LockDown Mode is enabled on all your hosts using PowerCLI (if your environment dictates it)

(get-vmhost * | get-view) | foreach-object -process {$._EnterLockdownMode()}

That will go through every host and enable LockDown mode.  If you see the following error while running the script:

Exception calling "EnterLockdownMode" with "0" argument(s):  "The administrator permission has already been dsiabled on the host (except for the vim user)"

 It just means that the host already has that setting enabled and it can be ignored.

vCenter VPXD.exe process using 100% memory

We recently had an issue in an environment that I am looking after that caused the VPXD.exe (vcenter server service) to use 100% of the memory allocated to a machine, plus all the disk space on the server (as the page file was set to system managed and just kept growing) until the server stopped responding.  After much investigation, the following lines were noted in the vCenter server log (appearing a lot).  The error is as follows:

[05444 error 'Default'] SSL StreamImpl::DoServerHandshake (0000000028a34600) SSL_accept failed with Unexpected EOF
[05444 warning 'ProxySvc'] SSL Handshake failed for stream TCPStreamWin32(socket=TCP(fd=5470) local=<vcenter IP>, peer=<remote client connection>), error=SSL Exception: Unexpected EOF

That error message basically translates to a VI Client instance connecting in to vCenter, when this occurs, each connection consumes additional memory which isn't released.  In our case, we had an incident where the VM Tools instance was ripped out of a management server, this management server used the VMXNET 3 driver.  In its unusual state it was polling the vCenter server a large number of times a minute (like it was flapping). Once VMware Tools was reinstalled on the management server, vCenter calmed down and returned back to a normal amount of memory usage.

Tuesday 23 April 2013

VMware VRM Server not able to connect to local vCenter Server

If you install the VRM Server as part of vSphere Replication (this problem is from version 5.0.2) you may see this error in the hms log file.  The HMS logs are located on the VRM itself under /opt/vmware/hms/logs.  You can see how to get these logs from KB2013091.

The error is as follows:

2013-04-12 11:01:14.526 WARN hms.configuration [mai]
(..hms.cfg.VcExtensionChecker) | Extension login failed:java.lang.RuntimeException:

java.util.concurrent.ExecutionException: com.vmware.vim.binding.vim.fault.InvalidLogin:
Inheritied from com.vmware.vim.binding.vim.
Inheritied from com.vwmare.vim.fault.
InvalidLogin:  Cannot complete
login due to an incorrect user name password.

Now, the problem is that your VRM Server needs to not only communicate to the vCenter on HTTP (TCP 80) it also needs to do the initial connection when the services start on HTTPS (TCP 443).  This was the case in the environment I was working in due to firewalls between the two.  If the VRM Server can only talk to vCenter on 80, it will fail to fully communicate to vCenter.  Once you open up HTTPS between vCenter and the VRM Server, login to the admin page https://<ip of your vrm server>:5480 to restart the services.  It should then connect.  If this is the case for your environment, you will need to follow KB2019120 in order for vCenter to show the health status correctly on the health page for the VRM.  You will also note that KB1009562 which is the Port Number reference for SRM and its' components do not state that this port is required.  I could not get communication functional between the same site vCenter Server and VRM Server until both TCP 80 & 443 were allowed through the firewall to the vCenter server.  I would also allow the VRM server to talk to the remote site vCenter server on HTTP and HTTPS (the kb article only states TCP 80).

Thursday 4 April 2013

Resetting a Local Administrator Account Password in Windows 2008 R2

Had trouble using the NT Password utility or any other recover tool to reset a local administrator account for Windows 2008 R2?   Well here's a simple way to do that which uses the Vista Backdoor trick.  You should get yourself a copy of Hiren's Boot CD to do this.  One thing to note is that if you are doing this on a virtual machine the VMware Paravirtual SCSI device will not show any disks when you boot off this tool.  The other thing to note is that the "Offline NT/XP/2000/Vista/ Windows 7 Password Changer" is a use at your own risk tool, I've also not had it work on fully patched Windows 2008 R2 machines:

  • Burn the ISO to CD (if no iLO/RMC or the machine is not virtual)
  • Boot off the Hiren's Boot CD
  • Select MiniXP Environment

  • Once the environment has booted, fire up Explorer

  • Browse to the Drive (if you have multiple, check each one) with the Windows installation on it

  • Browse to <Drive>\Windows\System32\

  • Rename "Magnify.exe" to "Magnify.exe.old"

  • Copy "CMD.exe" and paste it in the same directory (you should end up with a file called 'CMD_2.exe' at the very bottom of the explorer list of files) rename the copy to "Magnify.exe"

  • Reboot the Server (with Hiren's now unmounted)
  • At the login screen, down the bottom left click the "Ease of Access" icon (circled in the image)

  • Select "Make items on the screen larger (Magnifier)" then click OK
  • You'll now have a Command Prompt running (if you type whoami, you'll see it is running as NT System)

  • Type "net user" and hit enter (you'll see a list of local users)
  • Type "net user <username> <password>" and the user of your choice's password is now changed!!!!  Note * change the password to a password that would match the local/domain security policy of your machine

  • You can also use standard net commands to add a user to the local administrators group, create a new user, etc! Close down the Magnify (well command prompt window) for the tidy up.

  • Once done, delete the "Magnify.exe' and replace it by renaming "Magnify.exe.old" back to "Magnify.exe"  Remember to take ownership of Magnify.exe.old first to Administrators and give full access before you do this!

  • The last thing to do is to make sure that Magnify doesn't launch every time you log in to the server.  Go to "Ease of Access" from control panel

  • Click "Optimize Visual Display"

  • Uncheck "Turn on Magnifier"
  • Click "Apply" then "OK"

There you have it, a simple and "SAFE" way of changing a local administrators password on a Windows Server 2008 R2 machine

Tuesday 19 March 2013

Replacing a failed disk in ZFS

As you can see below running “zpool status” shows me that disk c6d0 has faulted (before you replace the disk you can run "format" to check the serial numbers to make sure you pull out the right disk!!!!)

I replaced the faulty disk, then ran “zpool replace zfs c6d0” another check of “zpool status” and you can see the drive is rebuilding (resilvering as it’s called in the ZFS world).

Run a “zpool status” in 30 mins or so, this will give you an accurate finish time, in my case I can see its going to take about 6 and a half hours.

VMware vCenter Heartbeat Licensing

VMware Heartbeat Licensing?  It's a tricky question and one I have the answer to.  1 single license (yes that's just ONE) will entitle you to run Heartbeat on a vCenter Server AND a separate MS SQL server.  The caveat being that you are only supposed to protect VMware related databases (this could mean running VMware databases on a separate instance).  The other fact to note that if you have this type of setup, you will have 4 instances of the Heartbeat software running, 1 on the primary vCenter server, 1 on the secondary vCenter server, 1 on the primary Database server and 1 on the secondary Database server.

Stuff will be coming

I'm just setting myself up.  This is my first attempt at "blogging" (and I call myself a techo!).  Interesting stuff to follow!