vSphere 5.5 Install Pt. 5: SSL Deep Dive

SSL certificate Now that you’ve read up on upgrade best practices and SSO enhancements in the first four installments, we can get to the fun stuff! Understanding vCenter 5.5 SSL certificates. Ok maybe not fun for all, but fun to me. Last year in my 5.1 install series I just plowed right into the minting process, but didn’t really explain the big SSL picture. Since SSL is probably the biggest point of confusion in vCenter 5.1 and later, this post will serve to illuminate one of my favorite subjects. After all I have to keep up my “SSL guy” nickname. Do not fear, this series will eventually get to actually mounting the vCenter ISO and doing a software install. But I hope these foundational pre-install articles enable you to better understand the product.

Blog Series

SQL 2012 AlwaysOn Failover Cluster for vCenter
vSphere 5.5 Install Pt. 1: Introduction
vSphere 5.5 Install Pt. 2: SSO 5.5 Reborn 
vSphere 5.5 Install Pt. 3: vCenter Upgrade Best Practices and Tips
vSphere 5.5 Install Pt. 4: ESXi 5.5 Upgrade Best Practices and Tips
vSphere 5.5 Install Pt. 5: SSL Deep Dive
vSphere 5.5 Install Pt. 6: SSL Certificate Template
vSphere 5.5 Install Pt. 7: Install SSO
vSphere 5.5 Install Pt. 8: Online SSL Minting
vSphere 5.5 Install Pt. 9: Offline SSL Minting
vSphere 5.5 Install Pt. 10: Update SSO Certificate
vSphere 5.5 Install Pt. 11: Install Web Client 
vSphere 5.5 Install Pt. 12: Configure SSO
vSphere 5.5 Install Pt. 13: Install Inventory Service
vSphere 5.5 Install Pt. 14: Create Databases
vSphere 5.5 Install Pt. 15: Install vCenter
vSphere 5.5 Install Pt. 16: vCenter SSL
vSphere 5.5 Install Pt. 17: Install VUM
vSphere 5.5 Install Pt. 18: VUM SSL
vSphere 5.5 Install Pt. 19: ESXi SSL Certificate

Permalink to this series: vexpert.me/Derek55
Permalink to the Toolkit script: vexpert.me/toolkit55

Who cares about SSL?

Why should you go through the headaches of replacing all the VMware self-signed certificates? What’s the risk of using untrusted certificates? What can happen if the SSL connection is compromised?

Hypervisors are likely the underpinnings of your business critical apps and intellectual property. If your hypervisor is compromised then it’s just a few short commands to access your critical business data. Unless you like your infrastructure being p0wn3d, then you don’t want your VMware infrastructure compromised. If you don’t use trusted certificates, and just click through all the VI client SSL warnings (you have clicked Ignore and trust this certificate many times…haven’t you?) then you won’t know that a man-in-the-middle attack has taken place.

A man-in-the-middle attack is where a third party intercepts your “secure” communications and relays data between you, the attacker’s device, and the end host (like an ESXi server). This can be accomplished by ARP spoofing, or other means. If it’s second nature to ignore and click through all VI certificate warnings, you will have no idea your credentials have been intercepted….in clear text. No fancy brute force decryption required. Just sit back, grab a coke, and enjoy cleartext flowing across your screen. An interesting article on attacking VMware is here.

The diagram below shows the man-in-the-middle attack, and the following graphic is the clear text credentials from the VI client. With those credentials the bad guys can now own your infrastructure.

10-2-2013 9-04-19 PM

10-2-2013 9-23-20 PM

If you use trusted SSL certificates and untrain your staff to ignore SSL warnings, then it’s significantly harder to spoof the communications.  Think of this this way: Would you enter your credit card details or social security number on a site that threw up constant SSL warnings? No. So why trust potentially billions of dollars of IP to untrusted certificates that your grandma could spoof. There’s a reason why financial institutions, Government and other high value targets require real certificates. Don’t think that just because you run Nana’s Cookie shop that the bad guys aren’t after you…they are.

To be clear, man-in-the-middle attacks and SSL certificate spoofing are not unique to VMware or because they did something wrong. It’s a problem across the industry with self-signed certificates. I hope you are sufficiently scared to take certificates seriously and replace them. Even if you bake cookies in your garage and sell them on the corner stand, I’d update your SSL certs.

Certificate Requirements

Last year with vCenter 5.1 there was some confusion around certificate requirements. Ok, A LOT of confusion. How many certs do you need? What CA template options? What is a certificate OU?  I was stumbling around in the dark last October since official guidance at that time was a wee bit skimpy. Thankfully that situation has been largely remedied by VMware. The list below is a combination of the vCenter Certificate automation tool requirements and of the certificates themselves:

  • Platform Support: Windows Server 2008 R2 SP1 or later, including Windows Server 2012 (not R2)
  • Private Key Algorithm: RSA with 1024 bits or higher (I use 2048). No fancy elliptical curve support.
  • Recommended Signature Algorithms: SHA256, SHA384, SHA512
  • NOT Recommended algorithms: MD2, MD5, SHA1
  • Path or certificate filename can’t contain: ^ (caret), % (percent), & (ampersand), ; (semicolon), ) (closing parenthesis)
  • Use PEM certificate format, with a header of —–BEGIN CERTIFICATE——
  • Must use a complete certificate chain
  • Use OpenSSL v0.9.8y (Do NOT use OpenSSL 1.x !!). Recommend 32-bit version.
  • Does NOT support wildcard certificates
  • Have the subject alternative name (SAN) field included in them
  • Have unique OrganizationalUnitNames for the components (which in turn creates a unique Distinguished Name (DN))
  • Key Usage: Include digitalSignature, keyEncipherment, dataEncipherment
  • Seven unique certificates: InventoryService, SSO, UpdateManager, vCenter, WebClient, LogBrowser, Orchestrator

As you can see, there are a number of very specific requirements to meet in order to have success with replacing your vCenter SSL certificates. These are essentially unchanged from vSphere 5.1, but are now documented. Thank you VMware! Official sources of the requirements are here and here, if you have a burning desire to read more…before you fall asleep.

One final requirement that can trip people up is the format of the RSA private key file (rui.key). VMware states that only a key file with the —–BEGIN RSA PRIVATE KEY—- string will be accepted. Newer versions of OpenSSL may format it differently, creating problems down the road.

RSA private key

Certificate Subject Properties

One of the areas of confusion is exactly what certificate properties are important for a successful trusted certificate implementation. If we look at a SSL certificate which we will be generating in an upcoming post, we see a number of properties that may be Greek to most people.

In the example below we see properties common to all our SSL certificates. For example, we can see the signature hash algorithm is the recommended SHA256. You can also see validity dates, which are important. The primary point of confusion, and what took a lot of detective work when 5.1 went GA, is what needed to be in the Subject field. The SSO service in 5.1 and carried through to 5.5, requires a unique DN (distinguished name) for each certificate. The most common way to make the DN unique is to modify the OU value for each certificate. You could modify another field, but I would not recommend it. For all practical purposes the OU must be unique.

As you can see in the first certificate the OU value is vCenterSSO. In the second certificate the OU is VMwareUpdateManager.  While you don’t have to use these exact strings, they must be unique within the SSO certificate store. I’m opting go with the ‘official’ OU descriptions. Yes, even if you do the simple install with all components on a single VM, every component on that server must have a different certificate. You can NOT use wildcard certificates, either.

Commercial CAs can also have problems minting these properties. Although I’m not sure why you’d want to spend the money on a public CA certificate for an internal service. But some people try it…and are usually disappointed with the results.

vCenter SSO certificate VMware Update manager certificate

Certificate Key Usage

There also has been some lively discussion around other certificate properties, and what is and is not required. Certificate key usage defines how the OS or application can use the certificate. For example, can you only use it to digitally sign an email? Or only for code signing? Or just to secure RDP traffic? Those ‘features’ are enabled by Key Usage and Enhanced Key Usage fields.

For the purposes of this series I’m using KB 2037432 as a reference. As you can see in the graphic below, which is a VMware provided certificate request template, you see various fields. Highlighted are the KeyUsage and extendedKeyUsage fields. Of some controversy are whether dataEncipherment and clientAuth options are really required. Those are not properties normally allowed if you use the Microsoft “web server” template. For the purposes of this series I’ll assume they are required and will show you how to create a custom CA template to mint them. YMMV, and possibly the “web server” template will work fine for you. VMware must have a good reason to include them, so I’m not going to argue. My cooler is stocked with bigger VMware fish that need to be fried.

VMware certificate request

If you look at the properties of a properly formatted certificate you will see I used the correct template which enabled the minting of a certificate with said key usage requirements.

SSL data encipherment SSL authentication

Subject Names

Another important certificate property is the subject name and the subject alternative name (SAN). The subject name (commonName) is the FQDN of your vCenter server (assuming all components are on a single VM). Even if you are using the simplified deployment model all seven certificates will have the same commonName. The subjectAltName are alternative names that the host is known by. For those of you familiar with Microsoft Lync or Exchange deployments, you will be familiar with SANs.

There’s a little debate on what needs go to into the SAN field. Specifically, do you include the IP address or not. In the official VMware certificate template request they include the IP (shown below). However, at VMworld they said not to include the IP. Authentication uses Kerberos, which requires hostnames (can’t use IPs). Since the certificate replacement process is challenging, having to re-issue certs if the IP changes is not remotely appealing. My stance is to NOT include IPs in the certificate.

An interesting note I found in KB2037432 was that the FQDN in the SAN field was moved to the last value to support SRM. Curious that the order of SAN fields would matter, but apparently it does for SRM. Sounds like a bug to me.

SSL subject alternative name

Certificate Signing Request (CSR)

In order for a CA to mint you a certificate you must submit what is called a CSR, or certificate signing request, to your CA. This request contains all of the properties in the request template that you see above. It also contains data which is tied to the private RSA key. In our case we will use OpenSSL to process configuration files for each of the seven certificates, then submit the CSR to the CA, where we can then download our freshly minted certificate. The CSR will be in Base64 format, meaning it’s a neatly formatted text file you can open with Notepad and peek at. Although you will need to be Neo to decode it on the fly.

Certificate Authority Hierarchy

In many environments you will have more than one certificate authority. Typically an enterprise would have an offline root CA that is tightly controlled and secured. You then have one or more online issuing CAs which are joined to the domain and issue your certificates. Or, you may have a simplistic model where a single online CA does everything. If you have two or more tiers of CA, then you have some extra work cutout for you. Why? vSphere needs information about all tiers so that it can properly validate the chain of trust all the way to the root. Subsequent installments will show you exactly what extra steps are needed.

Certificate Creation Process

The certificate creation process is fairly simple once you understand the process. First, OpenSSL creates a private RSA key, usually 2048 bits long. That is used in conjunction with the certificate configuration file by OpenSSL to generate a CSR. The CSR is submitted to the CA, and out pops an SSL certificate at the other end. Nifty!

SSL certificate process

PEM Certificates

The vCenter Certificate automation tool requires certificate files to be in a very specific format. If they are not in that format (PEM), then you will have show-stopping problems and wonder why everything is a smoking pile of ashes. PEM encoded files are a series of Base64 certificates in a specific order. First up the tool needs the public certificates for your CA(s). If you have a subordinate CA, then this file (chain.cer) needs to have both the root and subordinate CA BASE64 certificates concatenated together. Do not add any spaces or lines, as that will not work. An example is shown below.

PEM certificate

In addition to the root CAs, each of the seven services needs a chain.pem file that is a combination of your root CA(s) and the certificate for your service. Again, don’t add spaces or lines in the file or you will run into problems.

PEM certificate

Don’t worry about creating the files now or where they go. Coming up is a PoweShell script that does all of the work for you. But if you are troubleshooting, then understanding the file format should help. I know I got tripped up on it last year and embarrassingly opened a support ticket. But I probably wasn’t the only one. Now you won’t have to be one!


Now that you hopefully understand a bit more about the certificate requirements, you will be in a better position to mint all of our SSL certificates. SSL certificates are complicated, and even in vSphere 5.5 they remain largely unchanged. Paths where VMware stores the certs are somewhat different, but the big overhaul that was needed did not happen. There are some cool ‘under the covers’ changes to certificates that I haven’t seen publically talked about yet which I’ll cover in the future. The teaser is that it sets the foundation for better certificate management in the future.

Next up in Part 6 is how to configure your Microsoft CA to issue SSL certificates with all the required properties.

Print Friendly, PDF & Email

Related Posts

Notify of
Newest Most Voted
Inline Feedbacks
View all comments
Doug McIntyre
October 10, 2013 2:27 pm

With regard to having IP addresses in the SAN field of certs. It seems to matter if you run vCenter windows in an Active-Directory environment, vs. standalone environment. In an AD environment, hostnames work all the time, you can have have FQDNs, and when VMware components sometimes shorten things to short names, the FQDN is used. OOTH, with standalone environments, when vmware components sometimes shorten names and assume FQDN to fill them out, it fails, and you have to refer to items via IP addresses, which is where having the IP address in the SAN field helps out quite a… Read more »

November 14, 2013 2:05 am

Very nice write up on part 5 about the various SSL settings, constraints and challenges. Thank you very much Derek for spreading this great information.

Microsoft has just announced two days ago, the Deprecation of SHA-1 Hashing Algorithm for Microsoft Root Certificate Program for January 1st, 2016. http://technet.microsoft.com/en-us/security/advis
It's recommended to stear away from SHA-1 as soon as possible, and use SHA512 (On x86-64 SHA512 is just as fast as SHA256).

May 12, 2014 10:07 am

Has anyone had luck updating only one component and leaving the others are the default self-signed? For example, if the only component we use that is customer facing is the Web Client, can we just update the Web Client with a CA-signed cert leaving the others as self-signed? I haven't found anything about this in VMware's documentation and would like to know if it's supported.

September 18, 2014 8:12 am

It was stated that sha256 was recommended in the post. I know I can changed this in the cert template. Does the entire chain have to be sha256?

September 18, 2014 11:54 pm
Reply to  Derek Seaman

Same question of Patrick how can i change the template to issue sha256 certificates?

December 15, 2014 11:07 pm

HI Derek, I wrote a script that does many of these pre-req's to eliminate the human error. I've been a big fan of the VCSA since it came out so it very much focused on improving the appliance experience. Feel free to use it and suggest improvements, it is far from perfect. vmwarenotes.blogspot.com

I've not done a Windows-based installation for over a year and like then, I'm back to your site for guidance. Good work!