Archives for June 2014

VSS Labs vCert Manager Part 1

Last August I wrote a blog post about this great new VMware SSL tool by VSS Labs called vCert Manager, which replaces many of your VMware SSL certificates all from the comfort of a nice GUI. It’s a full certificate lifecycle management tool for VMware vSphere and related components. For the full feature list and comparison with the free VMware tools, check out my post here. I’ll wait for as you read through that long article.

Ok with that out of the way, this time up I’ll actually walk you through the installation process, and see how easy it is to replace your vCenter 5.5 and ESXi host SSL certificates, all from a nice GUI. vCert manager is a licensed tool, but as you will see, has a lot of great features that no other tool that I’m aware of has.

If you are cash strapped and can only afford a free tool, then please remember my vSphere Toolkit script + the VMware certificate replacement tool combination. While the combination is less painful than the manual procedures, they fall very much short of the vCert Manager tool in terms of both functionality and ease of use.

VSS Labs has a special vExpert program, where you can get a NFR license that supports two vCenter servers and up to 10 ESXi hosts, for non-production usage. Their standard eval license supports 1 vCenter and 5 ESXi hosts.

What does it Update?

The list below is all of the VMware components that vCert Manager v1.2 can update:

  • vCenter 5.0: vCenter, Update Manager, Inventory Service, web client
  • vCenter 5.1: SSO, Inventory Service, vCenter, Web Client, Update Manager, Orchestrator
  • vCenter 5.5 (single location): SSO, Inventory Service, vCenter, Web Client, Update Manager, orchestrator
  • ESX/ESXi: 4.0 through 5.5

My Environment

In my environment I have Microsoft vCenter 5.5 server, and three ESXi 5.5 hosts. All of the vCenter components are installed on one server. I also have a two tiered Microsoft CA hierarchy. The root is offline, and I have an online issuing CA. Both are Windows Server 2012 R2. I have also setup a certificate template for VMware products, which you can read about here. You must have a certificate template, so if you don’t, go configure one following my guide.


1. I provisioned a Windows Server 2012 R2 VM from my standard template, which has no additional roles or features beyond .NET Framework 3.5 and .NET Framework 4.5. The tool is IIS and ASP.NET based, so we need to install IIS and ASP.NET to get started. Launch Server Manager and select the Web Server (IIS) role.


2. On the Features page expand .NET Framework 4.5 and select ASP.NET 4.5. Click through the rest of the wizard and wait for the components to install.


3. Go back to server manager and under Web Server enable ASP.NET 4.5, ISAPI Extensions and ISAPI Filters. Wait for the installation to complete.



4. Download the VSS Labs vCert Manager binary and start the installer. Click through the wizard until you get to the following screen. I chose the defaults, which are shown below. If you wanted, you could create a custom service account which the application pool would run under.

VSS Labs

5. Next up you can specify an account which the service uses to logon. Again, I left the default here. By using this default the computer machine account will need SQL database permissions, which I cover later.


6. Now we need to specify the SQL database connectivity, so that it can create and use a SQL database. I simply entered the database server, database name, and port. A great feature is supporting SQL encryption. According to the install guide a future version will support a system ODBC connection. The account I was logged in as has SQL sysadmin rights, so the Test Connection passed.


7. Finally you can change the installation directory. I choose the default here. The installation took less than a minute after I completed the wizard.


SQL Permissions

1. Go to your SQL server and add a new security login. Use the computer name (followed by a $) that vCert manager is installed on and change the database to the vCert manager database that was created during the installation process.






The installer will create a non-trusted SSL certificate, which I recommend we replace with one from your trusted CA. In my case I have an autoenrollment policy so that my computer already has a machine certificate that I can bind with in IIS. If your machine does not already have a machine certificate, then issue one and install it in the machine’s certificate store.

1. Open IIS and navigate to the vCert Manager site in the left pane. In the right pane click on Bindings then modify the bindings for the 8056 port entry. Select the thumbprint for your trusted machine certificate.



2. Open a web browser and navigate to https://FQDN:8056 and you should get a nice looking login page. They suggest using Chrome v26, Firefox 20 or IE 10. I used IE 11 and didn’t notice any issues. Verify there are no SSL warnings.

vCert Manager

And there you go..vCert manager is now fully installed and secured via a trusted SSL certificate. In the next installment we will dive into the management interface and start replacing certificates.

Nesting Hyper-V 2012 R2 on ESXi 5.5

imagesSince joining Nutanix I’ve had the opportunity to get exposed to Microsoft Hyper-V 2012 R2, as our platform supports the three most common hypervisors: VMware vSphere, Hyper-V, and KVM. I’m now embarking on writing some Hyper-V guides for Nutanix, and wanted a way to leverage my existing ESXi 5.5 Nutanix block to learn about Hyper-V networking. While I’m very familiar with VMware networking, this project presented itself as a great learning opportunity for Hyper-V. This article will show you how to nest Hyper-V 2012 R2 on ESXi 5.5.

My first challenge in getting a proper Hyper-V test bed setup was to deploy Windows Server 2012 R2 on my ESXi 5.5 express patch 4 host, then get the Hyper-V role installed. Now what I’m about to do is very unsupported, and I’m only doing it for my personal learning and quickly deploy a Hyper-V “learning” lab. After some extensive Binging and trial and error, I’ve narrowed down the unsupported tweaks needed to successfully run Hyper-V 2012 R2 on VMware ESXi 5.5.

Let’s get to it!

1. Deploy your standard Windows Server 2012 R2 template. Mine happened to be fully patched, and included the spring “update” which gave us back a semi-functional start button. I also used customizations specifications to automatically rename the VM, install license key, change the SID, etc. Nothing earth shattering here. I also used vHW v8, versus the newer v10 VM.

2. Power off your freshly deployed WS2012 R2 VM, and unregister it from vCenter.

3. Download the corresponding .VMX file to your computer and open it in Wordpad.

4. Somewhere in the VMX file add the two following lines:

vhv.enable = “TRUE”

hypervisor.cpuid.v0 = “FALSE”


5. If you have upgraded your VM to vHW 10 then you can follow William Lam’s tip and set the guestOS to use to be “windowsHyperVGuest”. If you are using vHW v8 then I just left it to the default “windows8svr-64”.


6. Save the VMX file and re-upload it to the datastore, overwriting the old file.

7. Right click on the VMX file and register the VM.

8. Now I didn’t need to do this, but saw some other users that had to configure this setting. In vCenter open the properties of the VM and change the CPU/MMU virtualization option. Select the bottom option.



9. Power on your VM, then login to Windows.

10. Install the Hyper-V role, and you shouldn’t get any warnings. Reboot after the roll is installed, and now you are ready to rock and roll with Hyper-V 2012 R2.


vCenter 5.5 Update 1b Now Out

2014-06-13_8-54-42Earlier this year the Heartbleed OpenSSL vulnerability came to light, and many products from many different vendors were affected. Heartbleed is a very serious vulnerability, and you should conduct a thorough audit to validate all of your software and hardware components are not affected. My bet is that many of your product are affected, and VMware was far from alone in having to issue software updates. Fresh off the presses is vCenter 5.5 Update 1b, which addresses CVE-2014-0224, Heartbleed. They have updated their OpenSSL libraries to 0.9.8za, 1.0.0m and 1.0.1h. I find it interesting that all three versions are used within the product.

As usual, this minor update includes some other bug fixes a well and not major new features. Other resolved issues include:

Host profile answer file might not be applied when you reboot the ESXi host
After you restart the vCenter services, if you use Auto Deploy to reboot a stateless ESXi host, the host profile answer file might not be applied to the ESXi host and the host configuration might fail. This issue occurs if the reference host is not available.

Under certain conditions, Virtual SAN storage providers might not be created automatically after you enable Virtual SAN on a cluster
When you enable Virtual SAN on a cluster, Virtual SAN might fail to automatically configure and register storage providers for the hosts in the cluster, even after you perform a resynchronization operation.

As is typical with VMware release notes there is a handy dandy list of known issues, which you should thoroughly read through before deploying this update. A few known issues that jumped out at me include:

  • Upgrade from vCenter Single Sign-On 5.1 Update x to 5.5 Update 1a fails if the password set for administrator@vsphere.local contains double quotation mark (“)
  • After you upgrade vCenter Server 4.x to 5.5 Update 1a, vCenter Server is inaccessible through vSphere Web Client (Due to SSL issues)
  • Attempts to upgrade from earlier versions of vCenter Single Sign-On 5.5 to 5.5 Update 1a might fail if the Windows Error Reporting Service is disabled
  • Installation using Simple Install fails on Windows Server 2008 R2 and Windows 2012 hosts


Given the extreme severity of the Heartbleed issue, this is one security update that I recommend you immediately start testing in your lab environment and roll out into product once you are satisfied that it hasn’t broken anything in your environment. You can find the full release notes here. You can download the updated ISO here.

vSphere 5.5 U1 NFS APD Fix out

A couple of months ago when upgrading one of my Nutanix clusters to vSphere 5.5 U1 I started to see what appeared to be random loss of connection to my NFS datastores. All VMs on the datastores would become inaccessible, then a few minutes later, access would be restored. As it turns out, this All Path Down (APD) bug introduced in vSphere 5.5 U1 and affected most any storage vendor using NFS. VMware wrote a KB article about the problem, which you can read here.

I am now pleased to announce that with vSphere 5.5 express patch 04, VMware has said they resolved the issue. You can download the patch here. I haven’t yet tried out the patch, but I’m very glad it is now out. For those of you still on vSphere 5.5 GA and using NFS, I would encourage you in a *lab* environment to thoroughly test express patch 04 and validate for your environment the issue has been resolved.

A big thanks to VMware for staying on top of this issue and coming out with a patch. I’ll be testing this in my lab very shortly.