Critical ESXi 5.5 Patch Released

Hot off the presses is a critical patch for ESXi 5.5, dated March 7, 2015. This addresses two VSAN patches:

  • Upgrading from VMware ESXi 5.5, Patch Release ESXi550-201502001 (Build 2456374) or VMware ESXi 5.5, Patch Release ESXi550-201501001 (Build 2403361) to vSphere 6.0 might result in VMware Virtual SAN (VSAN) Data Unavailability. See VMware Knowledge base article 2113024 for more details.
  • The Virtual SAN traces are not generated after you upgrade to VMware ESXi 5.5, Patch Release ESXi550-201502001 (Build 2456374) or VMware ESXi 5.5, Patch Release ESXi550-201501001 (Build 2403361).

You can read the full KB here. You can download the patch here.

Note: If you are Nutanix customer, this issue does not affect you so can safely ignore this update.


vCenter 5.5 U2 does not upport SQL AlwaysOn AGs afterall

VMware has just published a KB article (KB 2086946) [Temporally down] on using Microsoft failover clustering services to support high availability of vCenter 5.5 U2. My assumption for this support is the demise of the vCenter Heartbeat product. I appreciate VMware giving customers a new choice for vCenter high availability.

Slightly buried in this gem of a KB article, in the original version, was the new support for using SQL 2012 Always-On Availability Groups for the vCenter database! However, the KB was since pulled and I’m told by VMware it was published in error. In fact, AAGs have not been qualified and are not supported for the vCenter database. If you are a customer that wants vCenter DB SQL AAG support, then escalate to your TAM and make your opinion heard. In case you missed it, vCenter 5.5 does support the ‘traditional’ shared storage SQL cluster configuration for vCenter.

SQL AAGs are still supported on vSphere in the general sense. This new KB article was for the specific instance of the vCenter database. If you want to see the full support matrix for Microsoft high availability solutions, check out this article.

Prior to the KB being pulled it stated this:


vSphere 5.5 Update 2 Released

Hot off the digital presses is vSphere 5.5 Update 2. This is a minor update, but with some important supported database updates. It’s great to see SQL Server 2014 now supported. I’m in fact surprised they supported SQL 2014 so fast, so kudos to VMware. Now if we can only get SQL AlwaysOn availability groups supported..maybe someday. You should also take note that SRM 5.8 requires vCenter 5.5 Update 2, so whenever SRM 5.8 comes out be sure to upgrade your vCenter prior to deployment. vSphere 5.5 update 2 also allows the “legacy” C# vSphere client to modify some properties (RAM, Change network port group, Remove devices, vCPU, Mount ISO, Increase disk space, reservations, Edit advanced settings) of HW v10 VMs. Thanks VMware! Full vCenter release notes are here.

What’s New

vCenter Server database support: vCenter Server now supports the following external databases:

  • Oracle 12c. Important: For pre-requisite requirements, see KB 2079443.
  • Microsoft SQL Server 2012 Service Pack 1
  • Microsoft SQL Server 2014

vCloud Hybrid Service: The vCloud Hybrid Service (vCHS) introduces a new container, Hybrid Cloud Service, on the vSphere Web Client home page. The Hybrid Cloud Service container contains the vCHS installer and the new vCloud Connector installer.

Customer Experience Improvement Program: The vSphere customer experience improvement program is introduced to collect configuration data for vSphere and transmit weekly to VMware for analysis in understanding the usage and improving the product.

And as expected, ESXi 5.5 Update 2 also came out today. No surprise here. The biggest new feature here, and which was revealed at VMworld, is the support for 6TB of RAM in an ESXi host. Not too many people will be running hosts that big, but nice to know VMware has fully tested such monster hosts. For the full ESXi release notes go here.

Each of the release notes has a very long list of resolved issues. So if you are experiencing a particular bug, be sure to see if it has been resolved in 5.5 Update 2. You may just get lucky. These updates include security patches, so better start testing them in your pre-production environments and planning prod updates in the near future. As always, test, test, test before pushing this to production hosts.

Major vSphere Toolkit Update – v2.0 Live

While I was at VMworld 2014 in San Francisco last week I got a lot of very positive feedback about my 20 part vSphere 5.5 install series and my vSphere toolkit script. I’m glad its helped so many people make their vSphere SSL life easier. Up until now my Toolkit script assumed the ‘simple’ vCenter install with all services located on a single VM. This worked for most deployments, but clearly doesn’t cover all use cases. Some large organization may separate out the roles, such as SSO.

So to that end I’ve made some significant changes my to vSphere Toolkit script in v2.0, which requests the user to input the FQDN for all certificates. I’ve also added a prompt for the vCenter IP address, in case you want that in your certificate as well. Previously this was setup as a static variable in the script. To still keep things as easy as possible, the script will still read the hostname of the computer that it is running on and default to that for all the prompts. So if you have a simple install, you just need to press ENTER about a dozen times and don’t need to type a single hostname.

As always, the latest version of the script can be downloaded at The only other feature that has been requested is a triple stack of Certificate authorities, versus the root and subordinate architecture that I support today. I’m not sure there’s enough demand to make those changes, but that could be an enhancement in the future.

If you do decide to implement distributed vCenter components, then you will need to manually copy the certificate directory structure to each server and use the VMware SSL automation tool script in the proper sequence on each component. Below is a screenshot of the FQDN requests for each of the certificates.


vSphere 5.5 Toolkit Updated

This weekend I did a minor update to my VMware vSphere 5.5 SSL Toolkit script. It’s now at v1.59. I updated the OpenSSL download to use 0.9.8.zb, and also added a primitive PowerShell 3.0 check. PowerShell 3.0 and higher has always been required, but now I try and check for it. If you are running PS 3.0 and still get an error, then please leave a comment in this post. The logic isn’t all that intelligent, so may need tweaking.

If you aren’t familiar with my vSphere 5.5 toolkit script, then you can check out Part 8 of my 19 part vSphere 5.5 installation series. As always, you can download the latest version from

Join the over 10,000 downloads of my Toolkit script and make your SSL life a lot easier.

vCenter Server 5.5 Update 1c Out

A few days ago VMware released a minor update to vCenter to bring it up to vCenter Server 5.5 Update 1c. The update applies to both the Windows and vCenter Server appliance deployment models. If you are using vCloud Automation Center (vCAC), then you will want this update. Update 1c resolves issues around SSO and vCAC. Updates in this release include:

  • Attempts to perform vCloud Automation Center tenant administration operation fail with an error
    When you attempt to perform any vCloud Automation Center tenant administration operations such as removing an administrator from the default tenant (vsphere.local), the operation fails with a System Exception error.
  • Attempts to log in to vCloud Automation Center fail if the SAMAccountName contains extra trailing spaces
    When you attempt to log in to vCloud Automation Center, the login attempt fails if the SAMAccountName attribute contains extra spaces trailing at the end of the name.
  • Attempts to log in to vCloud Automation Center fail if the password contains the colon (:) character
    While attempting to log in to vCloud Automation Center, if you use a password that contains the colon (:) character, the login attempt fails.
  • Attempts to use the Windows Session Authentication feature might fail when you log in to vCloud Automation Center by using Windows Session Authentication on browsers such as Internet Explorer, Google Chrome, and Mozilla Firefox might fail due to an error in the VMware Client Integration Plug-in.
  • Windows Session Authentication login has failed as a result of an error caused by the VMware Client Integration Plugin
  • Attempts to log in to vCloud Automation Center fail if a custom UPN suffix is configured in the alias field for AD over LDAP
  • When you attempt to log in to the vCloud Automation Center where the custom UPN suffix is configured in the alias field for Active Directory (AD) over Lightweight Directory Access Protocol (LDAP), the login attempt fails.
  • Attempts to log in to vCloud Automation Center using vSphere Single Sign-On 5.5.0b might fail with an error

You can read the full release notes here. The Windows download is available here.

VMware vSphere 5.5 Toolkit v1.58 Live

As many of you know, one of my passions throughout my IT career has been security. Having worked in the Federal Government space for most of my career, making sure solutions are secure is always a top priority. Securing your VMware infrastructure is very important, and one of the primary tasks is using trusted SSL certificates. So last year I wrote the vSphere 5.5 Toolkit PowerShell script, which has had over 9,000 downloads! I had no idea it would be so popular. Here’s a screenshot of the main menu:

vsphere 5.5 toolkit

Features of the SSL toolkit script include:

  • Downloads and installs the proper version of OpenSSL ( if it’s not already installed
  • Creates 2048 bit RSA private keys in the proper format
  • Creates a directory for each service bundle of SSL certificates
  • Generates seven OpenSSL configuration files, one for each certificate, in the appropriate directory
  • Downloads both root and subordinate root public certificates
  • Submits the CSRs to the online CA and downloads the certificates
  • Creates the needed service PEM files for the vCenter certificate automation tool
  • Creates the required root/subordinate PEM files
  • Handles the special SSO 5.5 certificate requirements
  • Does NOT require PowerCLI
  • Assumes all vCenter components are on one server
  • Automatically uses the hostname of the server you run the script on for all certificates
  • Creates a pre-filled vCenter Certificate Automation environment script – Just run!
  • Works with offline CAs
  • Creates SSO 5.5 certificate replacement files – Only used if manual replacing certs
  • Creates customized SQL vCenter and VUM database creation script
  • Creates SQL ODBC DSNs for vCenter and VUM
  • Automatically downloads and installs SQL 2008 R2 or SQL 2012 client package
  • Linux vCenter Server Appliance support for online minting and offline CSR creation
  • Creates certificates for Auto Deploy, Dump Collector, Syslog collector, Authentication Proxy
  • Support Microsoft CAs that require manual certificate approval

I’ve now updated the script with some minor modifications for v1.58, dated July 12, 2014:

  • Updated OpenSSL download to 0.9.8za
  • Removed SQL 2012 SP1 client download (link broken)
  • Fixed Database creation script bug
  • Added additional error handling and Powershell-ized more commands
  • Changed the file to use sts in the URI per KB2058519

These are incremental updates, and the base functionality has remained the same. I am hoping for vSphere v.Next that VMware will streamline the whole process and give SSL replacement a makeover. I have no idea if this is in the works or not.

As always, you can download the latest version of the toolkit script from: If you are using an older version I suggest you grab the latest copy. If you want full SSL lifecycle management and a paid solution, I recommend you check out the VSS Labs vCert Manager, which you can find out about here.

Also remember to check out my 20 part vSphere 5.5 series, which covers the usage of the toolkit script and a whole lot more. You can find that series at:

Nesting Hyper-V 2012 R2 on ESXi 5.5

imagesSince joining Nutanix I’ve had the opportunity to get exposed to Microsoft Hyper-V 2012 R2, as our platform supports the three most common hypervisors: VMware vSphere, Hyper-V, and KVM. I’m now embarking on writing some Hyper-V guides for Nutanix, and wanted a way to leverage my existing ESXi 5.5 Nutanix block to learn about Hyper-V networking. While I’m very familiar with VMware networking, this project presented itself as a great learning opportunity for Hyper-V. This article will show you how to nest Hyper-V 2012 R2 on ESXi 5.5.

My first challenge in getting a proper Hyper-V test bed setup was to deploy Windows Server 2012 R2 on my ESXi 5.5 express patch 4 host, then get the Hyper-V role installed. Now what I’m about to do is very unsupported, and I’m only doing it for my personal learning and quickly deploy a Hyper-V “learning” lab. After some extensive Binging and trial and error, I’ve narrowed down the unsupported tweaks needed to successfully run Hyper-V 2012 R2 on VMware ESXi 5.5.

Let’s get to it!

1. Deploy your standard Windows Server 2012 R2 template. Mine happened to be fully patched, and included the spring “update” which gave us back a semi-functional start button. I also used customizations specifications to automatically rename the VM, install license key, change the SID, etc. Nothing earth shattering here. I also used vHW v8, versus the newer v10 VM.

2. Power off your freshly deployed WS2012 R2 VM, and unregister it from vCenter.

3. Download the corresponding .VMX file to your computer and open it in Wordpad.

4. Somewhere in the VMX file add the two following lines:

vhv.enable = “TRUE”

hypervisor.cpuid.v0 = “FALSE”


5. If you have upgraded your VM to vHW 10 then you can follow William Lam’s tip and set the guestOS to use to be “windowsHyperVGuest”. If you are using vHW v8 then I just left it to the default “windows8svr-64”.


6. Save the VMX file and re-upload it to the datastore, overwriting the old file.

7. Right click on the VMX file and register the VM.

8. Now I didn’t need to do this, but saw some other users that had to configure this setting. In vCenter open the properties of the VM and change the CPU/MMU virtualization option. Select the bottom option.



9. Power on your VM, then login to Windows.

10. Install the Hyper-V role, and you shouldn’t get any warnings. Reboot after the roll is installed, and now you are ready to rock and roll with Hyper-V 2012 R2.


Veeam Best Practices for VMware on Nutanix

Note: This article has been significantly updated on 4/18/14 with new information, in a great collaborative effort with Luca Dell’Oca (@dellock6) from Veeam. The official whitepaper can be downloaded here.

The goal of the joint whitepaper between Veeam and Nutanix is to help customers deploy Veeam Backup & Replication v7 on Nutanix, when used with VMware vSphere 5.x. This post will highlight some of the major points and how customers can head off some potential issues. The whitepaper covers all the applicable technologies such as VMware’s VADP, CBT, and Microsoft VSS. It also includes and easy to follow checklist of all the recommendations.

Veeam is modern data protection for virtual environments, and are also a great sponsor of my blog. The web-scale Nutanix solution and its data locality technology are complimented by the distributed and scale-out architecture of Veeam Backup & Replication v7. The combined Veeam and Nutanix solutions leverage the strengths of both products to provide network efficient backups to enable meeting recovery point objective (RPO) and recovery time objective (RTO) requirements.

The architecture is flexible enough to enable the use of either 100% virtualized Veeam components or a combination of virtual and physical components, depending on customer requirements and available hardware. You could also use existing dedicated backup appliances. In short, our joint solution is flexible enough to meet your requirements and efficiently use your physical assets. For example, if you have requirements for tape-out, then you will need at least one physical server in the mix to connect your library to since tape Fibre Channel/SAS pass-thru is not available in ESXi 5.x.


When virtualizing  solution the last thing you want is your backup data stored in the same location as the data you are trying to protect. So the first best practice for a 100% virtualized solution is to use a secondary Nutanix cluster. The cluster would be comprised of at least three Nutanix nodes. This is where the virtualized Veeam Backup & Replication server (along with the data repository), would reside. Should you have a problem with the production Nutanix cluster, your secondary cluster is unaffected. Depending on the amount of data you are backing up and your retention policies, you may or may not want the same Nutanix hardware models as your production cluster. For example, you may want to consider the 6000 series hardware which are ‘storage heavy’ for your secondary cluster. The following figure depicts a virtualized Veeam backup solution.


In case you aren’t familiar with Nutanix, on each node (server) there is a controller VM which services all I/Os for the VMs running on that host. Performance scales out as you add nodes, since you are adding more controllers. You are not bottlenecked like with legacy SANs which typically only have two controllers. You can see this in the diagram above, where there are three controller VMs, one per node. Two of the controllers (CVMs) are in the production cluster and one in the secondary cluster. A Nutanix cluster requires a minimum of three nodes, so for two clusters a total of six nodes is required.

Since the first version of this post, Veeam and Nutanix have done more testing and gathered feedback from the field. As a result, the second best practice is now recommending to use “Network mode” backups and not Hot-add (also known as Virtual Appliance mode). Why? For medium to large scale deployments this results in a higher backup reliability.  When used with the Nutanix 10Gb NICs, it still has great performance. The primary goal of this joint paper is to provide a solid solution that customers can use, and this highlights our collaborative efforts.

Network mode connects to each ESXi host through the VMkernel management interface. So the third best practice is to make sure your ESXi management interfaces are using the 10Gb NICs and not the 1Gb NICs. The following screenshot shows one of the many possible NIC configurations. Here I’m showing the 10Gb NICs as active adapters, with our 1Gb NICs in standby. This is not a required configuration, but just an example. If you have ESXi enterprise plus, this could be a great time to look at Load Based teaming, if you aren’t already using it.


The fourth best practice is for the Veeam repository server, where I recommend adding dedicated VMDK(s) that use the PVSCSI controller. The PVSCSI controller is more CPU efficient under high IOPS load as my colleague Michael Webster blogged about here. I’d also recommend using vSphere 5.5, where a single VMDK can exceed 2TB. That enables larger backup repositories, which you may need in medium to large environments.

Finally, backing up your data has little value if you can’t restore it. When using Veeam Backup and Replication with Nutanix, I’m please to say that the full spectrum of restore options are at your fingertips with no special procedures required. For example, you can use Veeam’s vPower NFS technology, instant VM recovery, file-level restores, and U-AIR. Nutanix also fully supports all the application consistency options that Veeam offers their customers. So you can fully backup your Exchange, SQL, SharePoint, Active Directory, and other applications in a logically consistent manner.

The forthcoming whitepaper has a lot more detail, and other recommendations regarding backup types, operating systems, and version of Nutanix OS that we recommend. Once the full best practices guide is published I’ll add a link to this post. This has been a great collaborative effort with Luca Dell’Oca from Veeam, and you can grab your copy here.

VMware vSphere 5.5 Update 1 Released

Hot off the presses is VMware vSphere 5.5 Update 1. Unlike prior “update” releases which are mostly bug fixes, this update packs more of a punch. Full vCenter release notes can be found here. ESXi 5.5 Update 1 release notes are here.

What’s New:

  • vCloud Hybrid Services vSphere client plug-in now works in the vSphere web client
  • You can install vCenter on Windows Server 2012 R2 (Whoohoo!)
  • Support for VMware VSAN (the “punch” in this update)

Notable Bug Fixes:

  • vCenter fails to install on Windows Server 2012 R2
  • A couple of Windows ‘workgroup’ related SSO bug fixes
  • Performance chart bugs
  • Trusted Platform Module attestation information reporting bug
  • SSO Integrated Windows Authentication provider bug fixes
  • Local OS group authentication failures
  • Custom UPN SSO logon failure bug fix
  • SSO appears to support more complex password characters such as ; ^ and “
  • Updated JRE to 1.7.0 Update 45 to address security issues
  • VCSA root password expires after 90 days and locks out
  • NTP security bug fix
  • Unable to disable weak ciphers on CIM port 5989
  • VMXNET3 driver resets in Windows Server 2008 R2 when using RSS
  • E1000 and E1000E purple screen issues
  • Glib-c update to address security issues

Even if you don’t have a need for any of the ‘new’ features, the security fixes alone should prod you into doing due diligence testing then rolling it out into production. I see the GA of VSAN as a validation of the huge hyperconverged market which vendors like Nutanix have pioneered. As more vendors ‘see the light’ about web scale infrastructure, more and more customers will start to realize the centralized storage arrays are morphing into dodo birds faster than you think.

Download links:

ESXi 5.5 Update 1
vCenter 5.5 Update 1
vSphere CLI 5.5 Update 1
Other Software