vCenter Certificate Automation Tool: Part 1 (Pre-reqs and Config)

For those of you that have installed vSphere 5.1 and tried to use your own trusted SSL certificates, you will probably find the experience extremely tedious, cumbersome, error prone, and vastly harder than any product you’ve used before. Now out is the VMware vCenter Certificate Automation tool v1.0, to make the process somewhat easier.

My 15-part vSphere 5.1 series goes into all of the gory details of the manual replacement process, and closely follows the associated VMware KB articles but in a more understandable format that people seem to appreciate. But, now the process can be made easier and less painful.

So what’s new on the vSphere 5.1 SSL front? Late last week VMware released their first stab at easing the SSL certificate replacement torture in vSphere 5.1 with a basic command line tool which helps automate the process. I covered the announcement here. Since I’m pretty familiar with the pain and suffering vSphere 5.1 certificates has caused me, I wanted to see if this tool would make life easier.

Since the process is a bit long and only semi-automated, I’ve broken down the process into a series of posts:

Part 2 (SSO and Inventory)
Part 3 (vCenter and Orchestrator)
Part 4 (Web client and Log Browser)

—

UPDATE 5/18/2013: I’ve included some certificate and chain.pem validation steps. The certificate tool is very picky about the formatting and contents of the chain.pem files. If you get the certs in the wrong order or have too many certs, the tool produces cryptic error messages. I strongly urge you validate your certificates and chain.pem files or you may be opening a support case with VMware.

—

Before you begin this process, you MUST read through all of the limitations (“Known Issues”) of the v1.0 tool. The list is pretty extensive, and there was one biggie that jumped out at me. If during the vCenter VUM installer you elect to use the server FQDN instead of the IP address (which I would argue is a best practice), then you can’t use this tool to replace the VUM certificate. Really? Ouch. The tool also doesn’t help you generate any of the certificates, or a new requirement of certificate chain files for each of the seven services. So you still have a lot of pre-work to get to the point of even trying to use the tool. This is not a fully automated end-to-end tool that is wizard driven, which we desperately need.

Going into this with my eyes wide open, and somewhat tempered expectations of what it will do for me, I decided to give it a whirl. VMware’s KB article on the tool guides you through the process, but as usual, I think the process can use a bit more elaboration and screenshots.

In my case I have a Windows Server 2012 CA, and all of the vCenter services are installed on a single Windows Server 2008 R2 VM. My vCenter databases are on an external SQL 2008 R2 VM.

Prerequisites

Since this tool doesn’t help you create the certificate request files, generate the certificates, or the new PEM chain files we must do that prior to using the tool. I’ve updated my vCenter 5.1 U1 Installation: Part 2 (Create vCenter SSL Certificates) to address the new requirements and the addition of the Orchestrator certificate. So open up that post and follow through the entire certificate generation process, except for creating the JKS keystore. The script at the end of that post has been updated to create the newly required chained PEM files for each service. So if you’ve used that script before, grab the updated version and run it.

If you did not use my script which creates the chain.pem files, don’t worry. I’ve included steps later in this article on how to manually create them. Now that you’ve run through that long post to create all the certificate files, you should have a directory structure that looks like the screenshot below. I have these folders residing under D:\certs.

Inside each of the seven folders you should have the same set of files, as shown below with the appropriate configuration file. Again, if your chain.pem file is not present, I’ve included steps below on how to create and validate them.

You will also need the following accounts and passwords handy to complete the process:

• SSO administrator and password
• vCenter administrator and password
• Original vCenter database password

Configuring the Tool

1. Download the SSL Certificate Automation Tool from My VMware.

2. Copy it to your vCenter server and unzip it to a safe place, such as D:.

3. Open the ssl-environment.bat  file and fill in all of the missing paths. In my case I set the following:

set sso_cert_chain=D:\certs\sso\chain.pem
set sso_private_key=D:\certs\sso\rui.key
set sso_node_type=single
set sso_admin_is_behind_lb=
set sso_lb_certificate= set sso_lb_hostname=

set is_cert_chain=D:\certs\inventory\chain.pem
set is_private_key_new=D:\certs\inventory\rui.key

set vc_cert_chain=D:\certs\vCenter\chain.pem
set vc_private_key=D:\certs\vCenter\rui.key

set ngc_cert_chain=D:\certs\WebClient\chain.pem
set ngc_private_key=D:\certs\WebClient\rui.key

set logbrowser_cert_chain=D:\certs\LogBrowser\chain.pem
set logbrowser_private_key=D:\certs\LogBrowser\rui.key

set vco_cert_chain=D:\certs\Orchestrator\chain.pem
set vco_private_key=D:\certs\Orchestrator\rui.key

set vum_cert_chain=D:\certs\UpdateManager\chain.pem
set vum_private_key=D:\certs\UpdateManager\rui.key

set sso_admin_user=admin@system-domain
set vc_username=contoso\svc-vctr02-001

set last_error=
set ROLLBACK_BACKUP_FOLDER=%~dp0backup
set LOGS_FOLDER=%~dp0logs

4. Open an elevated command prompt and run the ssl-environment.bat script.

Creating PEM Files

If you used my script to automate the minting of your certificates then you already have the required chain.pem files. If you decided to go the more manual route, then you probably don’t have the required chain.pem files. They are easy to create, but also easy to screw up as the tool is very picky. Inside each service directory you need to have a chain.pem file. That file is comprised of the service’s certificate, intermediate CA (if you have one) and root CA certificates.

The VMware vCenter Certificate automation tool is very picky about whether your certificates are minted from a root CA or a subordinate CA. To find out which situation applies to you, open one of the certificates you’ve minted. If you only see one CA and the name of your vCenter server then then you don’t have a subordinate CA. If you see two or more CAs, then the one at the top is the root and the second one is the subordinate.

From inside each service directory you can use the following command to create the chain.pem file (assuming a subordinate CA):

copy /B rui.crt + D:\certs\root64-2.cer + D:\certs\root64-1.cer chain.pem

Repeat this process for every service directory. If you only have a root CA then the command would look like:

copy /B rui.crt + D:\certs\root64.cer chain.pem

Validate Chain.pem Files

You really should validate that the chain.pem files are properly configured. Having the certificates in the wrong order, or including an intermediate CA when your certificates are issued from a root CA will all cause the tool to #fail. Below is a truncated example of my root CA certificate file. I like to look at the last few characters of the cert, since they are very likely unique. Yours will of course be different.

Root64-1.cer (Root)

Root64-2.cer (Intermediate)

Opening one of the chain.pem files, if you are using an intermediate CA, shows that the service certificate is at the top, intermediate is in the middle, and the root is at the bottom. This is the correct order, and there must not be any blank lines before or after any certificate.

The certificate headers should also look exactly the same as below, with no extra information present. Even if your environment has an intermediate CA, if your certificates are issued from the root, do NOT include the intermediate CA. In that case you’d only have the service certificate at the top and the root at the bottom. Only include the CAs shown in your minted certificate, no more, no less.

If you jack up the order of the certs in the file you will get:

ERROR: One or more required parameters are not set or have invalid values: Certificate chain is incomplete: the root authority certificate is not present and cannot be detected automatically. The presence of the root certificate is required so the other service can establish trust to this service. Try adding the authority certificate manually.

Or if you include an intermediate CA but your certs are issued from the root:

ERROR: One or more required parameters are not set or have invalid values: The certificate chain file does not contain a valid certification path. PKIX path validation failed with: Could not validate certificate signature. (at certificate #1)

So please double check that your files are properly formatted, as the tool’s error messages are not that enlightening and don’t deal well with extra CA certificates.

Now that we have generated all of the required certificate files and set the environmental variables, we can walk through the planner and then actually replace the certificates. Replacing the SSO and Inventory SSL certificates are covered in Part 2.