vSphere 6.0 Pt. 12: PSC Machine Certificate

Back in Part 11 of this series we configured the VMCA to be a subordinate CA to our enterprise CA. This ensures that all certificates which get used by vCenter components are automatically trusted. But as previously mentioned, not all organizations can use the VMCA. The US Federal Government would be a prime example, where there’s no way you can stand up your own subordinate CA.

So if you are one of the organizations that can NOT use the VMCA and need to use custom SSL certificate throughout, this post is for you. In this post we will replace the PSC’s machine SSL Certificate with a certificate issued by your enterprise CA, not the VMCA. If you followed Part 11 and are using the VMCA, skip this post.

Just like Part 11, I’ll go through the same process of using a Microsoft online CA, offline CA, and updating the certificates for both Windows and the VCSA. This should cover most scenarios that people have to deal with. If that’s not exactly what your scenario is, you can probably figure out what to do between VMware documentation and my Toolkit posts.

As always, download the latest version of my Toolkit script, as it is rapidly changing as I add more blog posts about SSL and work through issues. The download permalink is below. For this post you will need at least version 0.75 (April 2, 2015) or later to follow along.

Ironically, the VMware supplied certificate tool in it’s GA form has a bug when you replace the machine certificate with multiple intermediate CAs. You can find the KB here. So I’d recommend using my Toolkit script for a Windows PSC, as it does not have the bug and is easier anyway. 🙂 I am told VMware is working on an updated script, but I have no ETA on a release date. If you are using the VCSA you will need to use the workaround, which I cover in my post below.

Blog Series

vSphere 6.0 Install Pt. 1: Introduction
vSphere 6.0 Install Pt. 2: Platform Services Controller
vSphere 6.0 Install Pt. 3: Certificate Management
vSphere 6.0 Install Pt. 4: vCenter Upgrade Best Practices
vSphere 6.0 Install Pt. 5: ESXi Upgrade Best Practices
vSphere 6.0 Install Pt. 6: Install Windows PSC
vSphere 6.0 Install Pt. 7: Config SQL DBs
vSphere 6.0 Install Pt. 8: Toolkit Configuration
vSphere 6.0 Install Pt. 9: SSL Templates
vSphere 6.0 Install Pt. 10: Install VCSA PSC
vSphere 6.0 Install Pt. 11: VMCA as Subordinate
vSphere 6.0 Install Pt. 12: PSC Machine Certificate
vSphere 6.0 Install Pt. 13: Directory Services Certificate
vSphere 6.0 Install Pt. 14: Windows vCenter Install

Permalink to this series: vexpert.me/Derek60
Permalink to my Toolkit script: vexpert.me/toolkit60

Mint Machine Certificate (Online)

You should run the Toolkit script on your Windows external PSC, so you have all the files needed locally and it will also automate the installation. If you are using the VCSA PSC, then run this script from a Windows server that has PowerShell 3.0. Use this online procedure if your Microsoft CA will issue the subordinate certificate either with or without approval.

1. Run the Toolkit PowerShell script on your external PSC or a Windows server VM if using the VCSA PSC. Select the Machine SSL Certificate menu (option 4). Select the option to create a Machine SSL certificate with an online MS CA (option 1).

2. Enter the FQDN of the PSC, or press ENTER, if running from the PSC to accept the name. If no certificate approval is needed, the new machine certificate will be minted and downloaded. If your MS CA is configured to require CA manager approval before issuing a certificate, you will see the following:

Have the CA manager approve the request ID, then re-run my Toolkit script and select the “Resume a pending online request for Machine SSL certificate” (option 4). The script will show you the paths to the chained PEM file and the private key file.

After the request is complete, you will see the following files in the C:\Certs\Machine directory.

You have now minted your Machine SSL certificate, but it is not yet installed. Read on further in this post on how to install it.

Mint Machine Certificate (Offline)

Use this procedure if your issuing certificate authority is NOT a Microsoft online CA. It could be an offline Microsoft CA, or a non-MS CA as well.  This assumes you have the proper templates configured in your CA, per my Part 9 post.

1. Run the Toolkit script and in the Machine SSL menu (option 4) select the option to create a Machine SSL certificate with an offline or non-MS CA (option 2). The script will verify that you have downloaded the root chain certificates.

2. Because I was running this on the external PSC, I just pressed enter for the PSC name.

3. Navigate to C:\Certs\Machine and upload the machine_ssl.csr file to your favorite CA and issue a certificate. Download the issued certificate in the base-64 format and save as new_machine.crt in the same folder. You MUST use this file name and it MUST be base-64 encoded. It should only contain the certificate, not a full chain.

4. Re-run the toolkit and from the menu select the option “Create Machine SSL PEM file from offline or non-Microsoft CA files” (option 5). No input is needed. This will properly create a PEM file with the full certificate chain.

Install Machine SSL Certificate (Windows PSC)

Note: For this procedure I am showing you how to use my Tookit script to install your Machine SSL certificate. VMware provides a Certificate Management tool that can perform the same steps. I show you how to use the VMware tool in the next section, when using the VCSA. The tool is the same on Windows and the VCSA. So if you feel more comfortable using the VMware tool to install the cert, skip down to that section. On Windows you can find the tool at C:\Program Files\VMware\vCenter Server\vmcad\Certificate-manager. My tool uses the manual method as documented in the vSphere 6.0 security guide, so the results are the same.

1. Re-run my Toolkit script and in the Machine SSL menu (option 4) select the option “Install custom machine SSL certificate on this computer” (option 6).

2. Sit back and wait while the script stops services, installs the new certificate, and restarts the services. Keep an eye on the process, as mid way through you will need to confirm the deletion of the existing machine certificate. Simply press Y.

Install Machine Certificate (VCSA PSC)

Note: For this procedure I am showing you how to use the VMware Certificate Manager tool to install the VMCA signing certificate. This assumes you used my Toolkit to generate the certificate files. There’s a bug documented in this VMware KB about the tool failing with multiple intermediate CAs. I’ll include the workaround here, so you have a one stop shop for replacing your certificates.

1. If you haven’t already enabled BASH on your VCSA let’s do that now. Open a console into the VCSA. Press F2 to customize the system. Login. Arrow down to “Troubleshooting Mode Options” then enable BASH shell. Exit the VCSA console.

2. Open a SSH session to the VCSA and type the following:

shell

chsh -s “/bin/bash” root

Make sure you run the ‘chsh’ command from the ‘shell’ prompt and not the VMware restricted shell…it won’t recognize the chsh command. Thanks to William Lam’s blog post here for this step!

2. Download and install your favorite SCP client. I like WinSCP. Connect via SCP using the VCSA credentials.3. Create a folder to put your SSL certificates. I like the ‘/root/ssl’ directory.

4. In WinSCP navigate to the C:\Certs\machine folder. Upload the new_machine.cer and ssl_key.priv files to the SSL directory on the VCSA. The other files in the machine folder are not needed, so don’t upload them. From the C:\certs folder upload the chain.cer AND the root64.cer files to the /root/ssl directory on the VCSA. Note that all the options begin with a double dash, not a single dash. Cut/paste may mangle the dashes and cause the command to fail. Best to manually type the whole command instead of cut/paste.

5. SSH into the VCSA and ensure you get a ‘shell’ prompt. This will be in red, and have the short name of the VCSA. Type the following command.

Windows :

“C:\Program Files\VMware\vCenter Server\vmafdd\dir-cli.exe” trustedcert publish –chain –cert c:\certs\chain.cer

VCSA:

/usr/lib/vmware-vmafd/bin/dir-cli  trustedcert publish –chain –cert /root/ssl/chain.cer

6. In the VCSA shell run the following command:

/usr/lib/vmware-vmca/bin/certificate-manager

6. Choose Option 1 from the main menu. Enter the SSO password as requested.

7. From the new menu select Option 2, Import custom certificates. Input the root certificate file names when prompted. Use /root/ssl/new_machine.cer for the first prompt and /root/ssl/ssl_key.priv for the second. For the third and final prompt enter /root/ssl/root64.cer.

8. After you enter all the certificate paths you will be prompted to continue. The whole replacement process takes less than two minutes.

Inspecting the Machine Certificate

Now that we have installed a new machine SSL certificate, we want to make sure it was issued by our enterprise CA and is trusted. This can easily be done via any browser of your choosing.

1. Launch your favorite browser and go to https://PSC-FQDN/websso/. Open the certificate properties for the SSL site.

2. Click on the Certification Path, and verify that all of your enterprise CAs are listed. If you only see a single entry in this list, and not the full chain, that likely means your Windows computer does NOT trust the full chain. See your CA administrator for getting all of your enterprise CAs published through Active Directory.

Solution Warning

A reader pointed out the SRM and other solutions may fail when replacing the machine certificate on vCenter or the PSC. If you find yourself in this situation, check out this VMware KB article for remediation.

Summary

When you aren’t using the VMware VMCA, you must mint and install a machine SSL certificate for the PSC from your enterprise CA. This certificate is used for all reverse proxy services, such as those accessed by HTTP. You can elect to either use my Toolkit script to install the machine cert, or the VMware tool. Either way, you end up with a trusted machine SSL certificate on your PSC.