vCenter Server 5.5 Update 1c Out

A few days ago VMware released a minor update to vCenter to bring it up to vCenter Server 5.5 Update 1c. The update applies to both the Windows and vCenter Server appliance deployment models. If you are using vCloud Automation Center (vCAC), then you will want this update. Update 1c resolves issues around SSO and vCAC. Updates in this release include:

  • Attempts to perform vCloud Automation Center tenant administration operation fail with an error
    When you attempt to perform any vCloud Automation Center tenant administration operations such as removing an administrator from the default tenant (vsphere.local), the operation fails with a System Exception error.
  • Attempts to log in to vCloud Automation Center fail if the SAMAccountName contains extra trailing spaces
    When you attempt to log in to vCloud Automation Center, the login attempt fails if the SAMAccountName attribute contains extra spaces trailing at the end of the name.
  • Attempts to log in to vCloud Automation Center fail if the password contains the colon (:) character
    While attempting to log in to vCloud Automation Center, if you use a password that contains the colon (:) character, the login attempt fails.
  • Attempts to use the Windows Session Authentication feature might fail when you log in to vCloud Automation Center by using Windows Session Authentication on browsers such as Internet Explorer, Google Chrome, and Mozilla Firefox might fail due to an error in the VMware Client Integration Plug-in.
  • Windows Session Authentication login has failed as a result of an error caused by the VMware Client Integration Plugin
  • Attempts to log in to vCloud Automation Center fail if a custom UPN suffix is configured in the alias field for AD over LDAP
  • When you attempt to log in to the vCloud Automation Center where the custom UPN suffix is configured in the alias field for Active Directory (AD) over Lightweight Directory Access Protocol (LDAP), the login attempt fails.
  • Attempts to log in to vCloud Automation Center using vSphere Single Sign-On 5.5.0b might fail with an error

You can read the full release notes here. The Windows download is available here.

vCenter 5.5 SSL Certificate and SQL Toolkit Updated

11-17-2013 7-03-32 PMFresh off the press is an updated version of my vCenter 5.5 SSL certificate Toolkit script. Last year when I did my popular vCenter 5.1 install series the posts contain a series of scripts and CLI commands to replace the SSL certificates. While that process worked for many people, it still was not as easy as it should be.

So for vCenter 5.5 I wrote a PowerShell script that did all the SSL certificate creation ‘magic’ in one place. In the intervening weeks since the first version went up, I’ve made a number of changes based on user feedback (and code submission) and my own development effort. I want to develop it further, but that will have to wait for a number of weeks while I complete a big project I’m working on. But for those that did download the first version and haven’t seen my Tweets about updates, I wanted a dedicated post to highlight the full feature set of v1.41 (November 10th).

The script is designed to be used in conjunction with the VMware vCenter certificate automation tool, NOT replace it. While that tool will create CSRs, I find it a bit cumbersome and does not help you in minting the certs. Regardless of what kind of CA you have, the script will help. The degree of automation varies, as the script is targeted for an online Microsoft CA. Once you use my tool to mint all of your certificates, then it’s a straight forward matter of using the VMware certificate tool to replace the self-signed certificates with your freshly minted ones.

As you will see in the feature list, the script goes beyond just SSL assistance and can also aid in your SQL database and DSN creation.

The script has the following features:

  • Downloads and installs the proper version of OpenSSL (0.9.8.Y) if it’s not already installed
  • Creates 2048 bit RSA private keys in the proper format
  • Creates a directory for each service bundle of SSL certificates
  • Generates ten OpenSSL configuration files, one for each certificate, in the appropriate directory
  • Creates certificates for AutoDeploy, Dump Collector and Syslog collector
  • Downloads both root and subordinate root public certificates
  • Submits the CSRs to the online CA and downloads the certificates
  • Creates the needed service PEM files for the vCenter certificate automation tool
  • Creates the required root/subordinate PEM files
  • Handles the special SSO 5.5 certificate requirements
  • Assumes all vCenter components are on one server
  • Automatically uses the hostname of the server you run the script on for all certificates
  • Creates a pre-filled vCenter Certificate Automation environment script – Just run!
  • Works with offline CAs
  • Creates SSO 5.5 certificate replacement files – Only used if manual replacing certs
  • Creates customized SQL vCenter and VUM database creation script
  • Creates SQL ODBC DSNs for vCenter and VUM
  • Automatically downloads and installs SQL 2008 R2 or SQL 2012 client package
  • Linux vCenter Server Appliance support for online minting and offline CSR creation
  • Creates certificates for Auto Deploy, Dump Collector and Syslog collector
  • Support Microsoft CAs that require manual certificate approval

On the potential roadmap is replacing the ESXi 5.x host certificates, and a bit more robust Linux VCSA support. A screenshot of the main menu is shown below.

As always you can download the latest version from: It’s gotten over 1,500 downloads in the few weeks that its been available, which is great. Hopefully it is helping people install vCenter 5.5 and more easily configure trusted certificates. For instructions on how to use the tool and a change log, start in Part 8 of my vCenter 5.5 install series.

11-10-2013 5-29-56 PM

vSphere 5.5 Install Pt. 15: Install vCenter

10-12-2013 8-30-50 PMThe previous 14 installments have all been leading up to this, installing vCenter. Yes, we are finally here. In this post we install vCenter, the windows vSphere client, fix profile driven storage, and configure vCenter to support a clustered SQL database. This post is not the end of the road, as we still need to secure vCenter with trusted SSL certificates and secure our ESXi servers.

Blog Series

SQL 2012 AlwaysOn Failover Cluster for vCenter
vSphere 5.5 Install Pt. 1: Introduction 
vSphere 5.5 Install Pt. 2: SSO 5.5 Reborn 

vSphere 5.5 Install Pt. 3: vCenter Upgrade Best Practices and Tips
vSphere 5.5 Install Pt. 4: ESXi 5.5 Upgrade Best Practices and Tips 
vSphere 5.5 Install Pt. 5: SSL Deep Dive
vSphere 5.5 Install Pt. 6: SSL Certificate Template
vSphere 5.5 Install Pt. 7: Install SSO
vSphere 5.5 Install Pt. 8: Online SSL Minting
vSphere 5.5 Install Pt. 9: Offline SSL Minting 
vSphere 5.5 Install Pt. 10: Update SSO Certificate
vSphere 5.5 Install Pt. 11: Install Web Client
vSphere 5.5 Install Pt. 12: Configure SSO
vSphere 5.5 Install Pt. 13: Install Inventory Service
vSphere 5.5 Install Pt. 14: Create Databases
vSphere 5.5 Install Pt. 15: Install vCenter
vSphere 5.5 Install Pt. 16: vCenter SSL
vSphere 5.5 Install Pt. 17: Install VUM
vSphere 5.5 Install Pt. 18: VUM SSL
vSphere 5.5 Install Pt. 19: ESXi SSL Certificate

Permalink to this series:
Permalink to the Toolkit script:

Install vCenter

1. If you are continuing from the last installment, then you should be logged into your vCenter server as the vCenter service account. If not, login as the vCenter service account. This is very important!

2. Launch the vSphere 5.5 installer and select vCenter Server.

10-12-2013 8-34-20 PM3. Go through the wizard until you get to the license key window. Enter a valid vCenter 5.x license key. Or, you can skip that screen for evaluation mode.

10-12-2013 8-35-55 PM

4. On the database option screen change the option to use an existing database. Your DSN should be listed from the pull down menu.

10-12-2013 8-37-23 PM

5. Since we are logged in with out service account and using Windows authentication we can’t change any options here.

10-12-2013 8-38-43 PM

6. You may get a warning about the recover model for your SQL database. If you use Full Recovery mode then you need to do regular backups to clear the logs. If you are in a lab or home environment you may want to change it to simple. Consult your DBA for best practices in your production environment.

10-12-2013 8-39-43 PM7. Enter the service account password.

10-12-2013 8-42-13 PM

8. Choose whether you want a standalone vCenter instance or linked mode. Remember Linked Mode can only interoperate with vCenters at the same release level.

10-12-2013 8-44-27 PM

9. Review the port numbers, but I would not change any of them.

10-12-2013 8-45-52 PM

10. Choose the inventory size based on your environment.

10-12-2013 8-46-47 PM

11. Enter the SSO password that you used during the SSO configuration.

10-12-2013 8-47-43 PM

12. Again, a thumbprint of the SSO certificate is shown. You should have memorized it by now and can verify it without referring back to the certificate.

10-12-2013 8-50-56 PM

13. I recommend leaving the administrator@vsphere.local default. Later on we will configure a delegate group for vCenter access.

10-12-2013 8-51-57 PM

14. Confirm the Inventory Service settings.

10-12-2013 8-53-29 PM

15. Confirm the installation directory then click Install.

10-12-2013 8-54-44 PM

16. After several minutes vCenter should successfully install.

Install vSphere Client

Although VMware is really limiting what you can do with the Windows vSphere client, it is still needed for some functionality such as VUM remediation, SRM, and connecting to ESXi hosts. So go back to the vSphere 5.5 installer and install the vSphere Client.

10-12-2013 9-55-49 PM

After you install and launch the client you will see a big warning on the login window. Clearly, the Windows VI is going to suffer a mob hit in the near future and end up in an unmarked grave. So learn the web client, and remember HW v10 VMs can only be modified via the web client.

10-21-2013 9-03-26 PM

Profile Driven Storage

If you are installing vCenter under a Windows service account, then we need to make a tweak to the Profile Driven Storage service. The installer configures it to run under Local System privileges, but that doesn’t work to well.

10-12-2013 10-05-25 PM

Open the service properties and change the Log On to use your vCenter service account. Restart the service.

Database Clustering

If you are clustering your SQL database, then we need to make a manual configuration change to vCenter. I’m assuming since supporting clustering was a last minute addition, they didn’t have time to add GUI option to the installer. If you are using a standalone SQL server, skip this section.

1. Navigate to C:\ProgramData\VMware\VMware VirtualCenter and make a backup of the vpxd.cfg file.

2. Stop the VMware VirtualCenter Server service. It make take a few minutes for it to stop.

3. Open the vpxd.cfg file in Wordpad (NOT Notepad). Scroll down and find the <vpxd> tag. Insert the three lines which I have highlighted below.

10-21-2013 8-51-16 PM

4. Save the file (without any text formatting), then restart the VMware VirtualCenter Server and VMware VirtualCenter Management Webserver services.

5. Log into the vSphere Web Client and verify that you can see your vCenter server and inventory.


In this post we installed  vCenter, fixed a permission bug with the profile driven storage service, and enabled SQL clustering support. What’s left to do? Secure vCenter with trusted SSL certificates, install VUM, and secure our ESXi hosts. Check out vCenter SSL in Part 16.

vSphere 5.5 Install Pt. 5: SSL Deep Dive

SSL certificateNow that you’ve read up on upgrade best practices and SSO enhancements in the first four installments, we can get to the fun stuff! Understanding vCenter 5.5 SSL certificates. Ok maybe not fun for all, but fun to me. Last year in my 5.1 install series I just plowed right into the minting process, but didn’t really explain the big SSL picture. Since SSL is probably the biggest point of confusion in vCenter 5.1 and later, this post will serve to illuminate one of my favorite subjects. After all I have to keep up my “SSL guy” nickname. Do not fear, this series will eventually get to actually mounting the vCenter ISO and doing a software install. But I hope these foundational pre-install articles enable you to better understand the product.

Blog Series

SQL 2012 AlwaysOn Failover Cluster for vCenter
vSphere 5.5 Install Pt. 1: Introduction
vSphere 5.5 Install Pt. 2: SSO 5.5 Reborn 
vSphere 5.5 Install Pt. 3: vCenter Upgrade Best Practices and Tips
vSphere 5.5 Install Pt. 4: ESXi 5.5 Upgrade Best Practices and Tips
vSphere 5.5 Install Pt. 5: SSL Deep Dive
vSphere 5.5 Install Pt. 6: SSL Certificate Template
vSphere 5.5 Install Pt. 7: Install SSO
vSphere 5.5 Install Pt. 8: Online SSL Minting
vSphere 5.5 Install Pt. 9: Offline SSL Minting
vSphere 5.5 Install Pt. 10: Update SSO Certificate
vSphere 5.5 Install Pt. 11: Install Web Client 
vSphere 5.5 Install Pt. 12: Configure SSO
vSphere 5.5 Install Pt. 13: Install Inventory Service
vSphere 5.5 Install Pt. 14: Create Databases
vSphere 5.5 Install Pt. 15: Install vCenter
vSphere 5.5 Install Pt. 16: vCenter SSL
vSphere 5.5 Install Pt. 17: Install VUM
vSphere 5.5 Install Pt. 18: VUM SSL
vSphere 5.5 Install Pt. 19: ESXi SSL Certificate

Permalink to this series:
Permalink to the Toolkit script:

Who cares about SSL?

Why should you go through the headaches of replacing all the VMware self-signed certificates? What’s the risk of using untrusted certificates? What can happen if the SSL connection is compromised?

Hypervisors are likely the underpinnings of your business critical apps and intellectual property. If your hypervisor is compromised then it’s just a few short commands to access your critical business data. Unless you like your infrastructure being p0wn3d, then you don’t want your VMware infrastructure compromised. If you don’t use trusted certificates, and just click through all the VI client SSL warnings (you have clicked Ignore and trust this certificate many times…haven’t you?) then you won’t know that a man-in-the-middle attack has taken place.

A man-in-the-middle attack is where a third party intercepts your “secure” communications and relays data between you, the attacker’s device, and the end host (like an ESXi server). This can be accomplished by ARP spoofing, or other means. If it’s second nature to ignore and click through all VI certificate warnings, you will have no idea your credentials have been intercepted….in clear text. No fancy brute force decryption required. Just sit back, grab a coke, and enjoy cleartext flowing across your screen. An interesting article on attacking VMware is here.

The diagram below shows the man-in-the-middle attack, and the following graphic is the clear text credentials from the VI client. With those credentials the bad guys can now own your infrastructure.

10-2-2013 9-04-19 PM

10-2-2013 9-23-20 PM

If you use trusted SSL certificates and untrain your staff to ignore SSL warnings, then it’s significantly harder to spoof the communications.  Think of this this way: Would you enter your credit card details or social security number on a site that threw up constant SSL warnings? No. So why trust potentially billions of dollars of IP to untrusted certificates that your grandma could spoof. There’s a reason why financial institutions, Government and other high value targets require real certificates. Don’t think that just because you run Nana’s Cookie shop that the bad guys aren’t after you…they are.

To be clear, man-in-the-middle attacks and SSL certificate spoofing are not unique to VMware or because they did something wrong. It’s a problem across the industry with self-signed certificates. I hope you are sufficiently scared to take certificates seriously and replace them. Even if you bake cookies in your garage and sell them on the corner stand, I’d update your SSL certs.

Certificate Requirements

Last year with vCenter 5.1 there was some confusion around certificate requirements. Ok, A LOT of confusion. How many certs do you need? What CA template options? What is a certificate OU?  I was stumbling around in the dark last October since official guidance at that time was a wee bit skimpy. Thankfully that situation has been largely remedied by VMware. The list below is a combination of the vCenter Certificate automation tool requirements and of the certificates themselves:

  • Platform Support: Windows Server 2008 R2 SP1 or later, including Windows Server 2012 (not R2)
  • Private Key Algorithm: RSA with 1024 bits or higher (I use 2048). No fancy elliptical curve support.
  • Recommended Signature Algorithms: SHA256, SHA384, SHA512
  • NOT Recommended algorithms: MD2, MD5, SHA1
  • Path or certificate filename can’t contain: ^ (caret), % (percent), & (ampersand), ; (semicolon), ) (closing parenthesis)
  • Use PEM certificate format, with a header of —–BEGIN CERTIFICATE——
  • Must use a complete certificate chain
  • Use OpenSSL v0.9.8y (Do NOT use OpenSSL 1.x !!). Recommend 32-bit version.
  • Does NOT support wildcard certificates
  • Have the subject alternative name (SAN) field included in them
  • Have unique OrganizationalUnitNames for the components (which in turn creates a unique Distinguished Name (DN))
  • Key Usage: Include digitalSignature, keyEncipherment, dataEncipherment
  • Seven unique certificates: InventoryService, SSO, UpdateManager, vCenter, WebClient, LogBrowser, Orchestrator

As you can see, there are a number of very specific requirements to meet in order to have success with replacing your vCenter SSL certificates. These are essentially unchanged from vSphere 5.1, but are now documented. Thank you VMware! Official sources of the requirements are here and here, if you have a burning desire to read more…before you fall asleep.

One final requirement that can trip people up is the format of the RSA private key file (rui.key). VMware states that only a key file with the —–BEGIN RSA PRIVATE KEY—- string will be accepted. Newer versions of OpenSSL may format it differently, creating problems down the road.

RSA private key

Certificate Subject Properties

One of the areas of confusion is exactly what certificate properties are important for a successful trusted certificate implementation. If we look at a SSL certificate which we will be generating in an upcoming post, we see a number of properties that may be Greek to most people.

In the example below we see properties common to all our SSL certificates. For example, we can see the signature hash algorithm is the recommended SHA256. You can also see validity dates, which are important. The primary point of confusion, and what took a lot of detective work when 5.1 went GA, is what needed to be in the Subject field. The SSO service in 5.1 and carried through to 5.5, requires a unique DN (distinguished name) for each certificate. The most common way to make the DN unique is to modify the OU value for each certificate. You could modify another field, but I would not recommend it. For all practical purposes the OU must be unique.

As you can see in the first certificate the OU value is vCenterSSO. In the second certificate the OU is VMwareUpdateManager.  While you don’t have to use these exact strings, they must be unique within the SSO certificate store. I’m opting go with the ‘official’ OU descriptions. Yes, even if you do the simple install with all components on a single VM, every component on that server must have a different certificate. You can NOT use wildcard certificates, either.

Commercial CAs can also have problems minting these properties. Although I’m not sure why you’d want to spend the money on a public CA certificate for an internal service. But some people try it…and are usually disappointed with the results.

vCenter SSO certificate VMware Update manager certificate

Certificate Key Usage

There also has been some lively discussion around other certificate properties, and what is and is not required. Certificate key usage defines how the OS or application can use the certificate. For example, can you only use it to digitally sign an email? Or only for code signing? Or just to secure RDP traffic? Those ‘features’ are enabled by Key Usage and Enhanced Key Usage fields.

For the purposes of this series I’m using KB 2037432 as a reference. As you can see in the graphic below, which is a VMware provided certificate request template, you see various fields. Highlighted are the KeyUsage and extendedKeyUsage fields. Of some controversy are whether dataEncipherment and clientAuth options are really required. Those are not properties normally allowed if you use the Microsoft “web server” template. For the purposes of this series I’ll assume they are required and will show you how to create a custom CA template to mint them. YMMV, and possibly the “web server” template will work fine for you. VMware must have a good reason to include them, so I’m not going to argue. My cooler is stocked with bigger VMware fish that need to be fried.

VMware certificate request

If you look at the properties of a properly formatted certificate you will see I used the correct template which enabled the minting of a certificate with said key usage requirements.

SSL data encipherment SSL authentication

Subject Names

Another important certificate property is the subject name and the subject alternative name (SAN). The subject name (commonName) is the FQDN of your vCenter server (assuming all components are on a single VM). Even if you are using the simplified deployment model all seven certificates will have the same commonName. The subjectAltName are alternative names that the host is known by. For those of you familiar with Microsoft Lync or Exchange deployments, you will be familiar with SANs.

There’s a little debate on what needs go to into the SAN field. Specifically, do you include the IP address or not. In the official VMware certificate template request they include the IP (shown below). However, at VMworld they said not to include the IP. Authentication uses Kerberos, which requires hostnames (can’t use IPs). Since the certificate replacement process is challenging, having to re-issue certs if the IP changes is not remotely appealing. My stance is to NOT include IPs in the certificate.

An interesting note I found in KB2037432 was that the FQDN in the SAN field was moved to the last value to support SRM. Curious that the order of SAN fields would matter, but apparently it does for SRM. Sounds like a bug to me.

SSL subject alternative name

Certificate Signing Request (CSR)

In order for a CA to mint you a certificate you must submit what is called a CSR, or certificate signing request, to your CA. This request contains all of the properties in the request template that you see above. It also contains data which is tied to the private RSA key. In our case we will use OpenSSL to process configuration files for each of the seven certificates, then submit the CSR to the CA, where we can then download our freshly minted certificate. The CSR will be in Base64 format, meaning it’s a neatly formatted text file you can open with Notepad and peek at. Although you will need to be Neo to decode it on the fly.

Certificate Authority Hierarchy

In many environments you will have more than one certificate authority. Typically an enterprise would have an offline root CA that is tightly controlled and secured. You then have one or more online issuing CAs which are joined to the domain and issue your certificates. Or, you may have a simplistic model where a single online CA does everything. If you have two or more tiers of CA, then you have some extra work cutout for you. Why? vSphere needs information about all tiers so that it can properly validate the chain of trust all the way to the root. Subsequent installments will show you exactly what extra steps are needed.

Certificate Creation Process

The certificate creation process is fairly simple once you understand the process. First, OpenSSL creates a private RSA key, usually 2048 bits long. That is used in conjunction with the certificate configuration file by OpenSSL to generate a CSR. The CSR is submitted to the CA, and out pops an SSL certificate at the other end. Nifty!

SSL certificate process

PEM Certificates

The vCenter Certificate automation tool requires certificate files to be in a very specific format. If they are not in that format (PEM), then you will have show-stopping problems and wonder why everything is a smoking pile of ashes. PEM encoded files are a series of Base64 certificates in a specific order. First up the tool needs the public certificates for your CA(s). If you have a subordinate CA, then this file (chain.cer) needs to have both the root and subordinate CA BASE64 certificates concatenated together. Do not add any spaces or lines, as that will not work. An example is shown below.

PEM certificate

In addition to the root CAs, each of the seven services needs a chain.pem file that is a combination of your root CA(s) and the certificate for your service. Again, don’t add spaces or lines in the file or you will run into problems.

PEM certificate

Don’t worry about creating the files now or where they go. Coming up is a PoweShell script that does all of the work for you. But if you are troubleshooting, then understanding the file format should help. I know I got tripped up on it last year and embarrassingly opened a support ticket. But I probably wasn’t the only one. Now you won’t have to be one!


Now that you hopefully understand a bit more about the certificate requirements, you will be in a better position to mint all of our SSL certificates. SSL certificates are complicated, and even in vSphere 5.5 they remain largely unchanged. Paths where VMware stores the certs are somewhat different, but the big overhaul that was needed did not happen. There are some cool ‘under the covers’ changes to certificates that I haven’t seen publically talked about yet which I’ll cover in the future. The teaser is that it sets the foundation for better certificate management in the future.

Next up in Part 6 is how to configure your Microsoft CA to issue SSL certificates with all the required properties.

VMware SSL pain? vCert Manager to the Rescue

VSS LabsOne of the most critical aspects to securing your VMware vSphere environment is replacing the self-signed certificates with trusted certificates. VSS Labs is releasing a new product, called vCert Manager, which will vastly improve the SSL certificate management experience for VMware customers.

Over the years I’ve written several articles detailing the process, which got exponentially harder with vSphere 5.1. I often hear users complain about the SSL certificate replacement process. Feedback on my blog has been quite verbose and colorful…some posts had to be censored. If you are struggling with the vSphere 5.1 installation, you can check out my vSphere 5.1 resources here.

While VMware did release the vCenter Certificate Automation tool earlier this year for usage with vCenter 5.1, it is command line/menu driven and does not fully automate the entire process. Given this tool was the only game in town I recommended my readers use the tool. You can find my multi-part usage series for the tool here.

The real game changer is the enterprise-scale VSS Labs vCert Manager. This goes way beyond the basic vCenter Certificate Automation tool, and provides full certificate lifecycle support for securing your vSphere environment with trusted SSL certificates. By full lifecycle I mean from automated vSphere infrastructure discovery, CSR generation, certificate minting, certificate installation, email expiration notices, automatic expired certificate updating, auditing, reporting, plus a lot more.

The entire process is managed via a web interface (securable with SSL), and requires no knowledge of OpenSSL, CSRs, PEM files, or leaf certificates. Plus it supports vSphere 4.x and 5.x environments, including ESXi hosts. The full support matrix is below.

If you are attending VMworld 2013 in San Francisco, be sure to stop by the New Innovator Pavilion and find the VSS Labs booth. Once you see the tool in action, you will be chomping at the bit to get a copy.

vCert Manager Features

Since the product has a very long feature list I’ve put together three tables that compare and contrast the features of the VSS Labs vCert Manager product and the VMware Certificate Automation Tool v1.01. The first table covers the VMware product/feature support of each tool. The second table covers certificate support, and the third table compares the tool core feature set and licensing.

VMware Product Support

VSS Labs VMware Support

As you can see from the table above, vCert Manager supports a broad range of VMware infrastructure. The VMware vCenter Certificate Automation tool is limited to just replacing vCenter 5.1 certificates. Version 1.01 has no support for prior versions of vCenter or ESXi hosts. Good tool for a specific use case, don’t get me wrong.

In contrast the vCert Manager product supports vSphere 4.0 through 5.1, including their respective ESX/ESXi host versions. I expect both companies to release updated certificate management products as new vSphere releases hit the streets. Neither tool has yet expanded to vCOPS, vCloud Director, or Horizon View. VSS Labs is targeting vCloud Director and Horizon View support in the near term, likely around the end of September.

Another great feature is that vCert Manager can manage dozens of vCenter instances and thousands of ESXi hosts from a single server and interface. No need to install the tool on each vCenter server. Credentials are server and host specific, so you can manage them all even if they are in different business units with different credentials.

Certificate Support

VSS Labs Certificate Support

Next to comprehensive VMware product support, broad certificate authority support and a high degree of automation is extremely important. This is where the vCert Manager product really shines. The VMware vCenter Certificate Automation tool requires a bit of manual pre-requisite work and using intermediate CAs can be tricky. Version 1.0.1 added CSR generation capability, so I certainly appreciate the added ease of use.

In contrast, vCert Manager automates the ENTIRE process, and you don’t have to know anything about certificate properties, SANs, how many certificates you need to mint, or how to submit and download certs from your CA. It can be configured to interface with an online Microsoft CA, so everything happens behind the scenes. Offline CAs are supported to, and as much of the process as possible is automated given the human intervention needed to mint the certificates.

In short, if you have an online CA the entire certificate request creation, minting, and deployment is executed through a GUI interface in an automated and orchestrated fashion. This makes certificate replacement almost foolproof.

Tool Features

VSS Labs vCert Manager Features

The most striking difference to me between the two tools is the user interface. vCert Manager sports a fully webserver driven GUI interface for all operations. If you haven’t used the VMware vCenter Certificate Automation tool you may not know that it’s completely command window based with text menus that instruct you in what order to perform the various steps. The VMware tool also needs to be installed on each server that has a vCenter component, unlike the vCert Manager which is installed on a single server and can manage thousands of vSphere components.

When updating ESXi hosts it requires DRS be enabled on the cluster so that it can put the host in maintenance mode, update the certificate, and return the host into production. This is all automated and when certificate is about to expire, it will automatically replace the certificates in the same manner.

Other great enterprise features of vCert Manager include syslog, full audit trail, basic reporting, and role-based access controls. Unlike the VMware tool, vCert Manager DOES require a Microsoft SQL database. So that’s a little extra configuration, but the installer does most of that work too. It supports Windows Authentication to the SQL database. SQL 2008, 2008 R2, and 2012 are supported.

Licensing and Support

There of course has to be a discussion about licensing and support. The VMware tool is free and if you have active SnS with VMware, they will support the tool for production usage. Clearly that’s a great value, and does ease much of the pain with vSphere 5.1 certificates. I certainly appreciate VMware releasing the tool, and the improvements in v1.01. I expect more improvements in the future.

vCert Manager is a licensed product, and support is provided by VSS Labs. The licensing model is per vCenter server, and per actively managed component (such as an ESXi host). For pricing details you will need to contact VSS Labs.

How to get a Free License

VSS Labs has a free NFR license available for home lab usage, with a special bonus for current fellow VMware vExperts. The tool will discover all of your components, even if they exceed your license, but you can only actively manage the number up to your licensed threshold.

8-19-2013 8-50-46 PM


vCert Manager should go GA by the end of August, if not sooner. When it does go live I’ll post a link to the download. I’ve been using the beta versions, and once the code goes GA I will post a multi-part blog post series on using the tool. So stay tuned!

For a version 1.0 product, I’m very excited to see the depth of support vCert Manager has packed into their first release. Throughout the beta process the company took feedback very seriously. After spending countless hours writing blog articles on VMware SSL certificates over the years, you have no idea how glad I am to see such a comprehensive tool. Maybe less blog articles for me to publish, but better for the community. Less hair pulling, and frustration for all!

The small home lab NFR licenses will give anyone interested in the product a good feel for how it works. I really appreciate the vExpert “bonus”, which will help the community test it against various vCenter versions more easily.

Sample Screenshots

The screenshots below are from a beta version, but will give you a little taste of the user interface and features. My installation series will contain a boatload of screenshots, once I get the GA code.

vCert Manager

vCert Manager Summary

vCert Manager Configuration

vCert Manager certificate authority

vCert Manager managed components

VMware ESXi 5.1 Patches Released

VMwareHot off the presses are some ESXi 5.1 patches. This build of ESXi 5.1 (1157734) fixes several bugs and more importantly addresses some security issues. As always in any environment, please test out the patches thoroughly before putting them into production. Each environment is unique, and issues may surface that could cause you some headaches. These bug fixes aren’t earth shattering, so I would not suggest rushing them out to production systems.

ESXi 5.1 Build 1157734

Highlights of the patch bundle included in this release are:

  • Black frames might appear around text boxes in an application running on Virtual Machine Hardware Version 8 or later. This issue occurs on virtual machines with Windows 7 guest operating system and View 5.0 PCoIP.
  • For two ESXi hosts with different host names, identical machine names are generated in the domain controller under certain conditions. As a result, the ctive Directory functionality is lost for one of the two ESXi hosts.
  • After you upgrade to ESXi 5.1 from an earlier version, attempts to power on a virtual machine with static MAC address outside the allowed range (00:50:56:[00-3f] or 00:50:56:[80-BF]) fail with the following error message: The MAC address entered is not in the valid range.
  • If a physical NIC is named using non-standard naming conventions (other than vmnic#) and is added to a vSwitch, host profile creation fails with the following error message: Invalid value chosen for active NICs.
  • ESXi 5.1 hosts might get disconnected randomly from the vCenter Server system. This issue might occur if the heartbeat thread in the vpxa agent does not receive a response from the futex_wait system call. As a result, the heartbeat thread stops responding, and the vCenter Server does not receive heartbeat messages from the ESXi hosts for several hours.
  • Upon reboot, ESXi 5.1 hosts configured to obtain DNS configuration and host name from a DHCP server displays its host name as localhost in syslog rather than displaying the host name obtained from the DHCP server. As a result, for a remote syslog collector, all ESXi hosts appear to be the same, with the same host name.
  • To prevent buffer overflow, the HPSA proc node truncates LUN details on an ESXi host.
  • This patch updates the esx-base VIB to resolve a stability issue.

As always, you can down the ESXi patches from here. The full KB article for the patch bundle is here.

vCenter Certificate Automation Tool: Part 1 (Pre-reqs and Config)

For those of you that have installed vSphere 5.1 and tried to use your own trusted SSL certificates, you will probably find the experience extremely tedious, cumbersome, error prone, and vastly harder than any product you’ve used before. Now out is the VMware vCenter Certificate Automation tool v1.0, to make the process somewhat easier.

My 15-part vSphere 5.1 series goes into all of the gory details of the manual replacement process, and closely follows the associated VMware KB articles but in a more understandable format that people seem to appreciate. But, now the process can be made easier and less painful.

So what’s new on the vSphere 5.1 SSL front? Late last week VMware released their first stab at easing the SSL certificate replacement torture in vSphere 5.1 with a basic command line tool which helps automate the process. I covered the announcement here. Since I’m pretty familiar with the pain and suffering vSphere 5.1 certificates has caused me, I wanted to see if this tool would make life easier.

Since the process is a bit long and only semi-automated, I’ve broken down the process into a series of posts:

Part 2 (SSO and Inventory)
Part 3 (vCenter and Orchestrator)
Part 4 (Web client and Log Browser)

UPDATE 5/18/2013: I’ve included some certificate and chain.pem validation steps. The certificate tool is very picky about the formatting and contents of the chain.pem files. If you get the certs in the wrong order or have too many certs, the tool produces cryptic error messages. I strongly urge you validate your certificates and chain.pem files or you may be opening a support case with VMware.

Before you begin this process, you MUST read through all of the limitations (“Known Issues”) of the v1.0 tool. The list is pretty extensive, and there was one biggie that jumped out at me. If during the vCenter VUM installer you elect to use the server FQDN instead of the IP address (which I would argue is a best practice), then you can’t use this tool to replace the VUM certificate. Really? Ouch. The tool also doesn’t help you generate any of the certificates, or a new requirement of certificate chain files for each of the seven services. So you still have a lot of pre-work to get to the point of even trying to use the tool. This is not a fully automated end-to-end tool that is wizard driven, which we desperately need.

Going into this with my eyes wide open, and somewhat tempered expectations of what it will do for me, I decided to give it a whirl. VMware’s KB article on the tool guides you through the process, but as usual, I think the process can use a bit more elaboration and screenshots.

In my case I have a Windows Server 2012 CA, and all of the vCenter services are installed on a single Windows Server 2008 R2 VM. My vCenter databases are on an external SQL 2008 R2 VM.


Since this tool doesn’t help you create the certificate request files, generate the certificates, or the new PEM chain files we must do that prior to using the tool. I’ve updated my vCenter 5.1 U1 Installation: Part 2 (Create vCenter SSL Certificates) to address the new requirements and the addition of the Orchestrator certificate. So open up that post and follow through the entire certificate generation process, except for creating the JKS keystore. The script at the end of that post has been updated to create the newly required chained PEM files for each service. So if you’ve used that script before, grab the updated version and run it.

If you did not use my script which creates the chain.pem files, don’t worry. I’ve included steps later in this article on how to manually create them. Now that you’ve run through that long post to create all the certificate files, you should have a directory structure that looks like the screenshot below. I have these folders residing under D:\certs.

Inside each of the seven folders you should have the same set of files, as shown below with the appropriate configuration file. Again, if your chain.pem file is not present, I’ve included steps below on how to create and validate them.

You will also need the following accounts and passwords handy to complete the process:

  • SSO administrator and password
  • vCenter administrator and password
  • Original vCenter database password

Configuring the Tool

1. Download the SSL Certificate Automation Tool from My VMware.

2. Copy it to your vCenter server and unzip it to a safe place, such as D:.

3. Open the ssl-environment.bat  file and fill in all of the missing paths. In my case I set the following:

set sso_cert_chain=D:\certs\sso\chain.pem
set sso_private_key=D:\certs\sso\rui.key
set sso_node_type=single
set sso_admin_is_behind_lb=
set sso_lb_certificate= set sso_lb_hostname=

set is_cert_chain=D:\certs\inventory\chain.pem
set is_private_key_new=D:\certs\inventory\rui.key

set vc_cert_chain=D:\certs\vCenter\chain.pem
set vc_private_key=D:\certs\vCenter\rui.key

set ngc_cert_chain=D:\certs\WebClient\chain.pem
set ngc_private_key=D:\certs\WebClient\rui.key

set logbrowser_cert_chain=D:\certs\LogBrowser\chain.pem
set logbrowser_private_key=D:\certs\LogBrowser\rui.key

set vco_cert_chain=D:\certs\Orchestrator\chain.pem
set vco_private_key=D:\certs\Orchestrator\rui.key

set vum_cert_chain=D:\certs\UpdateManager\chain.pem
set vum_private_key=D:\certs\UpdateManager\rui.key

set sso_admin_user=admin@system-domain
set vc_username=contoso\svc-vctr02-001

set last_error=
set LOGS_FOLDER=%~dp0logs

4. Open an elevated command prompt and run the ssl-environment.bat script.

Creating PEM Files

If you used my script to automate the minting of your certificates then you already have the required chain.pem files. If you decided to go the more manual route, then you probably don’t have the required chain.pem files. They are easy to create, but also easy to screw up as the tool is very picky. Inside each service directory you need to have a chain.pem file. That file is comprised of the service’s certificate, intermediate CA (if you have one) and root CA certificates.

The VMware vCenter Certificate automation tool is very picky about whether your certificates are minted from a root CA or a subordinate CA. To find out which situation applies to you, open one of the certificates you’ve minted. If you only see one CA and the name of your vCenter server then then you don’t have a subordinate CA. If you see two or more CAs, then the one at the top is the root and the second one is the subordinate.

5-18-2013 3-49-48 PM

From inside each service directory you can use the following command to create the chain.pem file (assuming a subordinate CA):

copy /B rui.crt + D:\certs\root64-2.cer + D:\certs\root64-1.cer chain.pem

Repeat this process for every service directory. If you only have a root CA then the command would look like:

copy /B rui.crt + D:\certs\root64.cer chain.pem

Validate Chain.pem Files

You really should validate that the chain.pem files are properly configured. Having the certificates in the wrong order, or including an intermediate CA when your certificates are issued from a root CA will all cause the tool to #fail. Below is a truncated example of my root CA certificate file. I like to look at the last few characters of the cert, since they are very likely unique. Yours will of course be different.

Root64-1.cer (Root)

5-18-2013 4-26-36 PM

Root64-2.cer (Intermediate)

5-18-2013 4-28-46 PM

Opening one of the chain.pem files, if you are using an intermediate CA, shows that the service certificate is at the top, intermediate is in the middle, and the root is at the bottom. This is the correct order, and there must not be any blank lines before or after any certificate.

The certificate headers should also look exactly the same as below, with no extra information present. Even if your environment has an intermediate CA, if your certificates are issued from the root, do NOT include the intermediate CA. In that case you’d only have the service certificate at the top and the root at the bottom. Only include the CAs shown in your minted certificate, no more, no less.

5-18-2013 4-21-16 PM

If you jack up the order of the certs in the file you will get:

ERROR: One or more required parameters are not set or have invalid values: Certificate chain is incomplete: the root authority certificate is not present and cannot be detected automatically. The presence of the root certificate is required so the other service can establish trust to this service. Try adding the authority certificate manually.

Or if you include an intermediate CA but your certs are issued from the root:

ERROR: One or more required parameters are not set or have invalid values: The certificate chain file does not contain a valid certification path. PKIX path validation failed with: Could not validate certificate signature. (at certificate #1)

So please double check that your files are properly formatted, as the tool’s error messages are not that enlightening and don’t deal well with extra CA certificates.

Now that we have generated all of the required certificate files and set the environmental variables, we can walk through the planner and then actually replace the certificates. Replacing the SSO and Inventory SSL certificates are covered in Part 2.

Create Windows CA VMware Certificate Template

In any enterprise environment, small or large, you should always used trusted SSL certificates for your VMware components. Very commonly people want to use a Microsoft Certificate Authority (CA). But, VMware requires certain properties be present in the SSL certificate to properly function. So you need to create a custom VMware certificate template in your CA to accommodate the key requirements.

You will need to modify the default Microsoft CA Web Server template settings to meet published VMware certificate requirements. vSphere 5.0 and earlier had an additional certificate requirement (nonrepudiation) that is not required in vSphere 5.1. This article will show you how to create a Microsoft CA template with all the past and present requirements, so that your bases are covered.

The default Web Server certificate template it does NOT have Data Encipherment, nonrepudiation, or client authentication enabled. Depending on the version of vSphere you are using, one or more of these properties may be required. So while the CA will happily issue you a certificate if you request these features, it will silently ignore the unsupported key usage specified in your CSR, which may cause you problems.

These instructions are based on Windows Server 2012, but all the options are available in prior Enterprise versions of the OS, such as Windows Server 2008 R2. You may have problems with “standard” edition CAs prior to Windows Server 2012, as they lack some certificate features found in Enterprise or higher editions. Windows Server 2012 standard edition has the full compliment of certificate options, so datacenter edition is not required (there is no enterprise edition).

If you are interested in the full 15-part vCenter 5.1 installation series with trusted SSL certificates, click here.

VMware Certificate Template Creation

1. Open the Certificate Authority tool. Locate the top Certificate Templates, right click, and select Manage. 

certificate authority
2. Locate the web server template and duplicate it.

3. Don’t change any of the compatibility settings. Leave it on Windows Server 2003.

4. Since this template will be used for VMware SSL certificates I named the new template appropriately. I also changed the validity period to three years, but the period the certificate is actually issued with depends on other CA properties so it may not be the full period you specify here.
 5. Open the Extensions tab, click on Key Usage, then select Signature is proof of origin and Allow Encryption of User data. Note: ESXi 5.1 does not require nonrepudiation or dataencipherment (encryption of user data). But I’ve enabled them here for backwards compatibility.
6. In the Extensions tab click Application Policies then click Edit.  Add the Client Authentication policy. Note: The vCenter 5.1 services do not require the Client Authentication option, but I’ve included it here for backwards compatibility with vCenter 5.0 and earlier. It appears ESXi 5.1 still wants client authentication.
7. On the Subject Name tab make sure Supply in the request is selected (it is by default). This will let us issue certificates with a SAN (subject alternative name).
4-22-2013 9-00-46 PM
8. After the template is made, you now have to permit certificates to be minted using that template. Right click on the Certificate Templates node as shown below, select New, then Certificate Template to Issue.

9. Select the VMware SSL template, or whatever name you used.

10. If everything went as planned you will have a new certificate template type when submitting a CSR. If you don’t see your new template, you may not have appropriate CA rights to issue the certificate.
11. To validate everything is working as planned, submit a CSR that has the Data Encipherment, nonrepudiation, and client authentication key requirements, then open the properties of the certificate. As you can see in the screenshots below, our minted certificate has all the required properties. If you have no idea how to create a CSR with these extra usage options, don’t fear, just read my blog post here. You are now ready to issue the proper SSL certificates for all of your vSphere environments.
Congrats! You now have a VMware certificate template that you can use with all modern versions of vSphere without fear of ignoring an important key usage attribute.

Schedule VMware UMDS Downloads with Windows Task Scheduler

VMware UMDS (Update Manager Download Service) is a product which ships with vCenter that allows you to download patches for VUM, then use them in air-gapped networks where VUM can’t directly download updates from the internet. UMDS 5.0 has some nice enhancements from 4.x, which helps limit the amount of unnecessary patches it downloads.

If you work in an environment where you need to utilize VMware UMDS you probably want the download process to be as automated as possible. One of the missing features in UMDS is a scheduler, so that’s where you can leverage the Windows Task Scheduler to make life easier. I’m assuming you are using Windows Server 2008 R2 and have already installed UMDS on a server. Creating the task is pretty quick and simple.

1. Launch the Task Scheduler. To keep the tasks more organized, I created a folder at the root level under Microsoft called VMware.

2. Right click on VMware and select Create Task. On the General tab you configure the basic task information. I recommend you change the task to use a privileged account, such as SYSTEM, and check the box to run the task with highest privileges. You could configure service account with only the required rights and use that instead, if running a task with SYSTEM rights doesn’t sit well with you.

3. Click on the Triggers tab and add a new trigger. I like to run the task once a day, at 4AM.

4. On the Actions tab create a new action and configure it as shown below. Change the path as needed to the location of the vmware-umds.exe file. Be sure to add the argument of -D or nothing will happen with the task runs.
5. On the Settings tab I changed the parameter that limits how long the task can run, but this is optional, and you can use any value you want.

6. At this point the task is configured and you can click OK. To test it out you can right click on the task and Run it. Monitor the download directory you configured and make sure it is being populated. For ESXi 4.1.0 and ESXi 5.0 patches, the UMDS repository at this time of this article was about 1.5GB.

Once the download task completes, you then need to “export” the repository into a directory, copy to removable media, and upload to the air-gapped instance of VUM. You can find all of the gory details in the Installing and Administering VMware vSphere Update Manager 5.0 Guide.

Configure custom Default Profile in VMware VM Templates

Over the past year I’ve been developing Windows Server 2008 R2 and Windows 7 VM templates for my VMware environment. However, the process has not been without its challenges. One of the features I wanted was a customized default user profile so that things like WMP, IE, and other settings were configured to our standards.

However, using the VMware customization specifications you can’t easily change the XML file it feeds sysprep to perform a profile copy operation. So how do you get a customized default profile with VMware vCenter?

Here’s the process that I use which works like a charm:

1. Install Windows into a VM using whatever method you wish (autounattend or manual).
2. Customize the Administrator’s profile however you wish (desktop icons, launch WMP, modify the toolbar, etc.).
3. Install any additional software or other tweaks you want to make to your image.
4. Install User Profile Manager then use the Copy To feature to copy the settings to the default profile, as shown below.

5. De-install User Profile Manager.
6. Shutdown your VM and turn it into a VM template.
7. Use vCenter to provision a new VM from the template, and use the customization specification to modify the VM such as hostname, administrator password, etc.

When vCenter provisions your VM and you log into it for the first time the default profile contains your customizations, and is configured how you wanted it to be. Easy as pie!