vSphere 6.0 Pt. 13: VMware Directory Svc Certificate

One of the lesser known SSL certificates in the vSphere 6.0 product is called the VMware Directory Service certificate. This is used by the built-in LDAP server for authentication and encryption. It’s most an internal use only certificate, and one that some customers may not worry about replacing. In fact, per VMware support, a lot of customers probably won’t replace this certificate. But, I’m a certificate whore, and wanted to be thorough in my coverage of vSphere 6.0. You will also see the certificate when you install vCenter 6.0 with an external PSC, and authenticate to the PSC. Even if you use the VMCA, the directory services certificate is not replaced by a trusted certificate.

In addition, the VMware certificate tool does not have a menu option to replace the VMware Directory Service certificate. But don’t fear, I’ve built it into my Toolkit script. What VMware doesn’t do, I do. So in this installment I will show you how to replace the VMdir certificate with either one trusted by your enterprise CA, or issued by the VMCA. The toolkit script will also automate the installation for you, on a Windows PSC. If you are using the VCSA, I’m sorry, but we have to use a manual process provided by VMware.

Blog Series

vSphere 6.0 Install Pt. 1: Introduction
vSphere 6.0 Install Pt. 2: Platform Services Controller
vSphere 6.0 Install Pt. 3: Certificate Management
vSphere 6.0 Install Pt. 4: vCenter Upgrade Best Practices
vSphere 6.0 Install Pt. 5: ESXi Upgrade Best Practices
vSphere 6.0 Install Pt. 6: Install Windows PSC
vSphere 6.0 Install Pt. 7: Config SQL DBs
vSphere 6.0 Install Pt. 8: Toolkit Configuration
vSphere 6.0 Install Pt. 9: SSL Templates
vSphere 6.0 Install Pt. 10: Install VCSA PSC
vSphere 6.0 Install Pt. 11: VMCA as Subordinate
vSphere 6.0 Install Pt. 12: PSC Machine Certificate
vSphere 6.0 Install Pt. 13: Directory Services Certificate
vSphere 6.0 Install Pt. 14: Windows vCenter Install

Permalink to this series: vexpert.me/Derek60
Permalink to my Toolkit script: vexpert.me/toolkit60

Mint VMDir Certificate (Online)

You should run the Toolkit script on your Windows external PSC, so you have all the files needed locally and it will also automate the installation. If you are using the VCSA PSC, then run this script from a Windows server that has PowerShell 3.0. Use this online procedure if your Microsoft CA will issue the subordinate certificate either with or without approval. This method also supports the VMCA.

If you are using a VMCA as a subordinate, then select menu option 3 to mint your certificates from the VMCA in step 1 below.

1. Run the Toolkit PowerShell script on your external PSC or a Windows server VM if using the VCSA PSC. Select the VMware Directory Service Certificate menu (option 3). Select the option to create a VMDir certificate with an online MS CA (option 1).2015-04-03_8-42-56

2. Enter the FQDN of the PSC, or press ENTER, if running from the PSC to accept the name. If no certificate approval is needed, the new VMDir certificate will be minted and downloaded.2015-04-07_11-03-09

 

If your MS CA is configured to require CA manager approval before issuing a certificate, you will see the following:

2015-04-02_8-06-50

Have the CA manager approve the request ID, then re-run my Toolkit script and select the “Resume a pending online request for VMDir certificate” (option 4). The script will show you the paths to the chained PEM file and the private key file. After the request is complete, all files are located in C:\Certs\VMDir.

Mint Machine Certificate (Offline)

Use this procedure if your issuing certificate authority is NOT a Microsoft online CA. It could be an offline Microsoft CA, or a non-MS CA as well.  This assumes you have the proper templates configured in your CA, per my Part 9 post.

1. Run the Toolkit script and in the VMware Directory Service menu (option 3) select the option to create a VMDir certificate with an offline or non-MS CA (option 2). The script will verify that you have downloaded the root chain certificates.

2. Because I was running this on the external PSC, I just pressed enter for the PSC name.2015-04-07_11-04-37

3. Navigate to C:\Certs\VMDir and upload the VMDir.csr file to your favorite CA and issue a certificate. Download the issued certificate in the base-64 format and save as VMDir.crt in the same folder. You MUST use this file name and it MUST be base-64 encoded. It should only contain the certificate, not a full chain.

4. Re-run the toolkit and from the menu select the option “Create VMDir PEM file from offline or non-Microsoft CA files” (option 5). No input is needed. This will properly create a PEM file with the full certificate chain.2015-04-07_11-06-54

 

Install VMDir Certificate (Windows PSC)

Note: For this procedure I am showing you how to use my Tookit script to install your VMDir certificate. VMware’s certificate tool does NOT support replacing the VMdir certificate, since not all customers feel the need to replace it. I feel the need. VMware did document the process in their vSphere 6.0 documentation, which is what I implemented in the script.

1. Re-run my Toolkit script and in the VMware Directory Service Menu (option 3) select the option “Install custom VMDir certificate on this computer” (option 6). 2015-04-03_8-44-00

2. Wait about 30 seconds, and the process will complete without any user input.

2015-04-02_21-04-48Install VMDir Certificate (VCSA)

By this point I’m assuming you have the BASH shell enabled, and know how to WinSCP and SSH into the VCSA. Those steps have been covered in pervious blog posts, so I’m not repeating them here.

1. Run my Toolkit script and on the main menu select VMware Directory Service Menu (Option 3). On the following menu select option 7 to rename the certificate files.

2. SSH into the VCSA and enter the following command:

/bin/service-control –stop VMWareDirectoryService

3. From the C:\certs\Machine directory copy the vmdircert.pem and vmdirkey.pem files to:

/usr/lib/vmware-vmdir/share/config/

4. Enter the following command:

/bin/service-control –start VMWareDirectoryService

Validate VMDir Certificate

In case you want to verify that the VMDir certificate actually was replaced and is using your trusted certificate, my toolkit can do that too!

1. Launch the Toolkit and from the main menu select VMware Directory Service menu (option 3). From there select Display VMDir Certificate (option 8).

2. OpenSSL is invoked to display in a somewhat unfriendly manner, the SSL certificate used for the LDAP services. Review the properties to ensure they contain those from your trusted CA.

Summary

While not a popular certificate to replace, replacing the VMware Directory Service certificate does its place. Since the Toolkit makes is so easy to do, I’d recommend replacing it as a matter of practice. This will eliminate a somewhat worrisome certificate validation pop-up during the vCenter installation process. Instead of seeing an untrusted certificate, you will see your freshly minted VMDir certificate.

vSphere 6.0 Install Pt. 11: VMCA as Subordinate

As mentioned in my vSphere 6.0 install part 3, VMware has introduced a new certificate option in this release. You can now make a built-in VMware certificate authority (VMCA) a subordinate to your enterprise CA. This is great news, as it will allow some automation of the certificate deployment process. However, there are certainly some regulated industries like the US Federal Government where they’d laugh you out of the office if you asked to stand up your own subordinate CA. So this model won’t be used by everyone.

So if you are one of those organizations that CAN use the built-in VMware VMCA, then follow along in this article to see how you make it a subordinate in your enterprise environment. Later in this series we will cover the manual replacement process so don’t worry…alternatives to the VMCA are covered.

As a reminder, I’ve written the vCenter Toolkit script which helps in the SSL deployment process. In this version of the script I’ve added some VMCA features. See the download link below. In this series I’m assuming a Windows Enterprise CA, but you can use a non-Windows CA with a little bit more work (manual submission and download).

Important: You MUST use Toolkit version 0.7 or later. Please download the latest Toolkit from the URL below.

Blog Series

vSphere 6.0 Install Pt. 1: Introduction
vSphere 6.0 Install Pt. 2: Platform Services Controller
vSphere 6.0 Install Pt. 3: Certificate Management
vSphere 6.0 Install Pt. 4: vCenter Upgrade Best Practices
vSphere 6.0 Install Pt. 5: ESXi Upgrade Best Practices
vSphere 6.0 Install Pt. 6: Install Windows PSC
vSphere 6.0 Install Pt. 7: Config SQL DBs
vSphere 6.0 Install Pt. 8: Toolkit Configuration
vSphere 6.0 Install Pt. 9: SSL Templates
vSphere 6.0 Install Pt. 10: Install VCSA PSC
vSphere 6.0 Install Pt. 11: VMCA as Subordinate
vSphere 6.0 Install Pt. 12: PSC Machine Certificate
vSphere 6.0 Install Pt. 13: Directory Services Certificate
vSphere 6.0 Install Pt. 14: Windows vCenter Install

Permalink to this series: vexpert.me/Derek60
Permalink to my Toolkit script: vexpert.me/toolkit60

Mint VMCA Certificate (Online)

You should run the Toolkit script on your Windows external PSC, so you have all the files needed locally and it will also automate the installation. If you are using the VCSA PSC, then run this script from a Windows server that has PowerShell 3.0. Use this online procedure if your Microsoft CA will issue the subordinate certificate either with or without approval. Before you run the Toolkit, always download the latest version as new versions can come out rapidly. You can get the latest from the permalink above.

1. Run the Toolkit PowerShell script on your external PSC or a Windows server VM if using the VCSA PSC. Select the VMCA menu (option 2). Select the option to create a VMCA signing certificate with an online MS CA (option 1).

2. Press Enter to accept the PSC name, and the script will take care of the rest. After it quickly runs you have a C:\Certs\VMCA directory with all of the needed files.2015-04-01_14-21-25

 

If your MS CA is configured to require CA manager approval before issuing a certificate, you will see the following:

2015-03-29_18-53-10

Have the CA manager approve the request ID, then re-run my Toolkit script and select the “Resume a pending online request for VMCA certificate” (option 3).

2015-03-30_19-07-35

After the request is complete, you will see the following files in the C:\Certs\VMCA directory. As you will notice, I’ve purposely used the same file names as the VMware documentation and tools.

2015-03-30_19-08-41

You have now minted your VMCA certificates, but they are not yet installed. Read on in the blog post to find out how to install them.

Mint VMCA Certificate (Offline)

Use this procedure if your issuing certificate authority is NOT a Microsoft online CA. It could be an offline Microsoft CA, or a non-MS CA as well. Don’t try this procedure with a public CA provider, as they won’t let you issue subordinate CA certificates. Only use this for internal enterprise CAs which you control and can issue subordinate certificate authority certificates with.

1. Run the Toolkit script and in the VMCA menu (option 2) select the option to create a VMCA signing certificate with an offline or non-MS CA (option 2). The script will verify that you have downloaded the root chain certificates.

2. Because I was running this on the external PSC, I just pressed enter for the PSC name.2015-03-29_12-00-023. Navigate to C:\Certs\ and upload the root_signing_cert.csr file to your favorite CA and issue a certificate. Download the issued certificate in the base-64 format and save as root_signing_cert.crt in the same folder. You MUST use this file name and it MUST be base-64 encoded. It should only contain the certificate, not a full chain.

4. Re-run the toolkit and from the menu select the option “Create VMCA PEM files from offline or non-Microsoft CA” (option 4). No input is needed. This will properly create a PEM file with the full certificate chain. 2015-03-30_19-10-59

Now that we have our VMCA certificates, we need to install them. I will cover both the Windows and VCSA installation process. Windows is easier, since I’ve built that functionality into my Toolkit. For the VCSA we will leverage the VMware certificate tool.

Install VMCA Certificate (Windows PSC)

Note: For this procedure I am showing you how to use my Tookit script to install your VMCA signing certificate. VMware provides a Certificate Management tool that can perform the same steps. I show you how to use the VMware tool in the next section, when using the VCSA. The tool is the same on Windows and the VCSA. So if you feel more comfortable using the VMware tool to install the cert, skip down to that section. On Windows you can find the tool at C:\Program Files\VMware\vCenter Server\vmcad\Certificate-manager. My tool uses the manual method as documented in the vSphere 6.0 security guide, so the results are the same.

1. Re-run my Toolkit script and in the VMCA menu (option 2) select the option “Install VMCA signing certificate on Windows PSC” (option 5).

2015-04-01_18-24-46

2. Sit back and wait while the script stops services, installs the new certificate, restarts the services, then lists the certificate properties. Keep an eye on the process, and verify a ‘success’ message half way through. I’ve added a pause and notification when you should see the ‘success’ message. The entire replacement process takes about two minutes. 2015-03-29_12-20-293. Once the script completed, it will show the properties of the new VMCA. Validate that these match the issued certificate.2015-03-29_12-21-47

4. To follow in the footsteps of the VMware Certificate tool that also replaces the Machine SSL certificate by a VMCA issued certificate, re-run the Toolkit and select the Machine SSL certificate menu (option 4). Then select Option 3, Create Machine SSL certificate with VMCA. Press enter to accept the default hostnames (assuming you are running this on the PSC/VMCA).

2015-04-01_18-33-54 5. Re-Run the toolkit and select the Machine SSL certificate menu again. This time select option 6. It will stop the PSC services, install the certificate, and then re-start the services. You will also get prompted mid way through if you want to delete the existing machine certificate. Answer “Y” here to proceed. 2015-04-01_18-39-20

Install VMCA Certificate (VCSA PSC)

Note: For this procedure I am showing you how to use the VMware Certificate Manager tool to install the VMCA signing certificate. This assumes you used my Toolkit to generate the certificate files.

1. If you haven’t already enabled BASH on your VCSA let’s do that now. Open a console into the VCSA. Press F2 to customize the system. Login. Arrow down to “Troubleshooting Mode Options” then enable BASH shell. Exit the VCSA console.

2. Open a SSH session to the VCSA and type the following:

shell.set –enabled true

shell

chsh -s “/bin/bash” root

Make sure you run the ‘chsh’ command from the ‘shell’ prompt and not the VMware restricted shell…it won’t recognize the chsh command. Thanks to William Lam’s blog post here for this step!

2. Download and install your favorite SCP client. I like WinSCP. Connect via SCP using the VCSA credentials.2015-04-01_14-40-003. Create a folder to put your SSL certificates. I like the ‘/root/ssl’ directory.

4. In WinSCP navigate to the C:\Certs\VMCA folder. Upload the root_signing_chain.cer and root_signing_cert.key files to the SSL directory on the VCSA. The other files are not needed, so don’t upload them.

5. SSH into the VCSA and ensure you get a ‘shell’ prompt. This will be in red, and have the short name of the VCSA. Type the following command:

/usr/lib/vmware-vmca/bin/certificate-manager

6. Choose Option 2 from the main menu. Enter the SSO password as requested.

7. From the new menu select Option 2, Import custom certificates. Input the root certificate file names when prompted. Use root_signing_chain.cer for the first prompt and root_signing_cert.key for the second.

2015-04-01_17-35-298. If you haven’t run the tool before, you will be prompted for a series of default values for the certool.cfg. Use the same values here as you did when setting up the Toolkit variables. When prompted, use the FQDN of the PSC for the ‘hostname’.

9. Wait a couple of minutes for the tool to run. After it has completed it has made the VMCA a subordinate to your enterprise CA, and also updated the machine SSL certificate to one issued by the new VMCA.

Inspecting the SubCA Cert

If you wish to look at the properties of the newly minted VMCA subordinate signing certificate which was installed, just double click on the C:\Certs\VMCA\root_signing_chain.cer file. Go to the Certification Path tab, and you can see the full certificate chain. If you do NOT see the full certificate chain, then follow my blog post here to enable your enterprise to trust the VMCA.

2015-03-29_19-10-30

You can also check out the Details tab, and go down to the “Basic Constraints” and “Key Usage” fields. There you can see this certificate can be used as a subordinate CA.

2015-03-29_19-12-08

Inspecting the Machine Certificate

Now that we have installed a new machine SSL certificate, we want to make sure it was issued by the VMCA and is trusted. This can easily be done via any browser of your choosing.

1. Launch your favorite browser and go to https://PSC-FQDN/websso/. Open the certificate properties for the SSL site.

2015-04-01_19-07-392. Click on the Certification Path, and verify that all of your enterprise CAs are listed, as well as the FQDN of your PSC. There should be a certificate listed under the FQDN of your PSC. If you only see a single entry in this list, and not the full chain, that likely means your Windows computer does NOT trust the VMCA subordinate CA. If that’s the case, I would recommend publishing the VMCA public certificate via Active Directory so that your entire domain will trust the VMCA. As you can see in my screenshot below I have three enterprise CAs and the VMCA.

2015-04-01_19-11-42

Solution Warning

A reader pointed out the SRM and other solutions may fail when replacing the machine certificate on vCenter or the PSC. If you find yourself in this situation, check out this VMware KB article for remediation.

Summary

Using my toolkit script, minting and updating the VMCA certificate is a breeze. My toolkit supports online Microsoft CAs, or any offline CA that you wish you use. Remember you can’t use a public CA for issuing your VMCA subordinate certificate. This must be an internal CA which you own, and which has been configured to issue certificates for the subordinate authority template.

Because we are using the VMCA for all of our certificates, we’ve also re-issued the PSC’s machine SSL certificate to one issued by the VMCA. This means any of the web services the PSC is providing are now using a VMCA issued certificate. In the next installment we will assume you can NOT use the VMCA for compliance reasons, and will replace the PSC’s machine SSL certificate with a trusted one. That is covered in Part 12. This will bring our two installations up to the same level. After that, we will replace the Directory Services certificate on the PSC.

vSphere 5.5 Toolkit v1.55 Released

Yes, time to update my vSphere 5.5 Toolkit with a few more features and bug fixes. For those of you that need to replace your vSphere 5.5 SSL certificates, the process can be somewhat cumbersome and time consuming. While VMware has a tool to help you replace the certificates once you create them (SSL certificate automation tool), it has limited functionality in helping you create all the files needed as pre-reqs to running the tool.

Since my vSphere 5.1 installation series was so popular, for vSphere 5.5 I wanted kick it up a few notches. So I wrote the vSphere 5.5 Toolkit script that has a number of features to ease your SSL pain. For a complete list of features, click here. To date it has had over 3,200 downloads. Now live is a minor update, for your deployment pleasure. v1.55 of my Toolkit script is now available for download here.

Derek Seaman vCenter 5.5 Toolkit

What’s new since v1.50?

Root Certificate Validation (New)

This version addresses an issue where sometimes the automatic download of a root or subordinate CA certificate would result in HTML code and not a Base64 certificate. The root cause of this issue is how Microsoft implemented the certificate download feature. Because the root certificates can be renewed, there’s a counter called “renewal” in the download URL to specify which certificate to download.

My script does not have logic to download all certificates and pick out the newest one (maybe in future versions). But what it will do is validate the file contents to ensure a certain string is present which indicates the file contains a Base64 encoded certificate. If the file is invalid an error will appear and the script halts. If that happens, search for “renewal” in the script (two locations) and decrement the number to 0. If it downloads an old certificate that expired, increment the number up by one until it gets the most recent version.

The script also checks manually downloaded base64.cer and interm64.cer certificate files for the same string, to validate they are Base64 encoded. It’s easy to use the wrong file type, which will greatly confuse the VMware certificate replacement tool. All of your certificate files should look like the example below, with —–BEGIN CERTIFICATE—–.

1-11-2014 2-25-42 PM

If your certificates are invalid, then you will get a red warning as shown below.

1-11-2014 2-46-54 PM

Certificate Request Changes (New)

VMware notified me that an upcoming change to a KB article was in the works. According to VMware the Web Client certificate needs the IP address in the SAN field with both DNS and IP extensions (e.g. DNS:10.10.10.10, IP:10.10.10.10). Apparently this is for maximum cross-browser compatibility across IE, Chrome and Firefox. For simplicity all certificate requests have both extensions in this version. If you don’t have any web client issues due to using an IP address vice the FQDN, then you don’t need to re-issue the web client certificate. If you do have issues, then this is probably the reason. You only need to update the web client certificate, not the 250 other vCenter certificates.

ESXi Host Support (If you missed it)

While not new to v1.55, version v1.50 released on December 22, 2013 added fairly robust ESXi host support. I didn’t blog about that version, so some of you may not be aware of it. I did Tweet, so make sure you follow me on Twitter for more timely news. You can manually enter several ESXi hosts to replace the certificates on, or give it an input file of hostnames. SSH is NOT required (uses HTTPS), and should be backwards compatible with vSphere 4.x and later although I have not personally tested it. This supports an Online Microsoft CA, offline CA, or third-party CA.

Summary

Given the positive feedback on the tool, it appears to be doing what I intended: Simplify the vCenter 5.5 installation process and make security easier. If you experience any problems or bugs, please leave a comment. I can’t promise to fix everything, but I’ll try to fit it into my schedule. Again, you can download the latest version from here.

vSphere 5.5 Install Pt. 10: Replace SSO Certs

10-11-2013 6-44-21 PMJust like replacing a hip joint, replacing vCenter SSO SSL certificates can induce some pain, is a bit complex, and the outcome can be questionable. The replacement process in SSO 5.5 is pretty much like that in 5.1, but now we have the VMware certificate automation tool and my vCenter toolkit to make this a safer operation. Outcome is not guaranteed, and there may be side effects.

My approach to replacing SSL certificates in this series is replacement right after the service is installed. Basically you build up a trusted infrastructure from the get-go. VMware would likely say to build it up all untrusted, then go back and replace the certs. Certainly a valid point. If the VMware automation tool was more automated, then I might agree. But by building up trusted layers there are less replacement steps and chance of messing up, IMHO.

I do strongly recommend using the VMware certificate automation tool instead of manual replacement steps. But for the SSO service I’ll show you both ways, each using my vCenter Toolkit script to prepare the needed files.

Blog Series

SQL 2012 AlwaysOn Failover Cluster for vCenter
vSphere 5.5 Install Pt. 1: Introduction 
vSphere 5.5 Install Pt. 2: SSO 5.5 Reborn 

vSphere 5.5 Install Pt. 3: vCenter Upgrade Best Practices and Tips
vSphere 5.5 Install Pt. 4: ESXi 5.5 Upgrade Best Practices and Tips 
vSphere 5.5 Install Pt. 5: SSL Deep Dive
vSphere 5.5 Install Pt. 6: SSL Certificate Template
vSphere 5.5 Install Pt. 7: Install SSO
vSphere 5.5 Install Pt. 8: Online SSL Minting
vSphere 5.5 Install Pt. 9: Offline SSL Minting
vSphere 5.5 Install Pt. 10: Replace SSO Certs
vSphere 5.5 Install Pt. 11: Install web client
vSphere 5.5 Install Pt. 12: Configure SSO
vSphere 5.5 Install Pt. 13: Install Inventory Service
vSphere 5.5 Install Pt. 14: Create Databases
vSphere 5.5 Install Pt. 15: Install vCenter
vSphere 5.5 Install Pt. 16: vCenter SSL
vSphere 5.5 Install Pt. 17: Install VUM
vSphere 5.5 Install Pt. 18: VUM SSL
vSphere 5.5 Install Pt. 19: ESXi SSL Certificate

Permalink to this series: vexpert.me/Derek55
Permalink to the Toolkit script: vexpert.me/toolkit55

Automated Certificate Replacement

1. Download the vCenter Certificate Automation Tool v5.5. You can find it under the Drivers and Tools section of the vSphere 5.5 downloads. Direct link is here. Unzip the contents on your vCenter server.

10-9-2013 6-10-08 PM

2. At the beginning of my toolkit script are two variables that set the SSO and vCenter administrator account names. Now during the SSO install you can’t change these names, so you shouldn’t need to change them here. But just FYI, I wanted to point out they are defined in the toolkit script.

10-9-2013 6-23-42 PM

3. Re-run my vCenter 5.5 Toolkit script, but this time we want to create the automation batch file. The menu number may change, but in today’s version of the script it is option 4.

10-9-2013 6-17-27 PM

4. The script will run in a split second and provide you the path to the batch file. The batch file sets the same variables as the stock VMware ssl-environment.bat file. I stripped out all of the comments and set the paths to where your certificate files are stored, assuming you used my script for part 8 or 9, minting your SSL certs.

5. Copy the ssl-environment.bat file that the toolkit created and overwrite the one in the VMware tool directory. You don’t need to run the batch file as the main updater script will execute it in the background.

10-9-2013 7-14-01 PM

6. Run the VMware ssl-updater.bat file. On the main menu select Option 3.

10-9-2013 6-27-30 PM

7. On the next menu select option 1.

10-9-2013 6-29-58 PM

8. When you select option 1 it will ask you a series of questions (in yellow below). Most of the information should be pre-populated for you. But you do need to input the administrator@vsphere.local password and answer if you are using a load balancer or not (don’t use one, VMware does not recommend it). Cross your fingers and toes, and watch for success messages.

10-9-2013 6-53-45 PM

9. Skip down to the verification section to validate that your certificate was replaced and that the service is in a healthy state. Let’s hope for no side-effects of this delicate operation.

Manual Replacement

If you for some reason you can’t use the VMware certificate tool (I recommend you DO use it, since it provides some certificate checking and is less error prone. But it could have issues that prevent you from using it. If so, then you can follow the steps below. They are directly from KB 2058519, with a couple of corrections (I’ve submitted fixes, so hopefully they will correct them).

Thanks to my vCenter 5.5 toolkit script 90% of the tedious work in that KB article is already done for you. Whoohoo. All we need to do is issue three commands to update the lookup service, then copy over our new certs. That’s it!

1. Open an elevated command prompt (not PowerShell) and enter the following command, replacing the vCenter FQDN, paths, and password as needed.

"C:\Program Files\VMware\Infrastructure\VMware\cis\vmware-sso\ssolscli" updateService -d https://d001vctr01.contoso.net:7444/lookupservice/sdk -u administrator@vsphere.local -p YourPassword -si D:\certs\vCenterSSO\gc_id -ip d:\certs\vCenterSSO\gc.properties

2. Enter the following command:

"C:\Program Files\VMware\Infrastructure\VMware\cis\vmware-sso\ssolscli" updateService -d https://d001vctr01.contoso.net:7444/lookupservice/sdk -u administrator@vsphere.local -p YourPassword -si D:\certs\vCenterSSO\admin_id -ip d:\certs\vCenterSSO\admin.properties

3. Enter the following command:

"C:\Program Files\VMware\Infrastructure\VMware\cis\vmware-sso\ssolscli" updateService -d https://d001vctr01.contoso.net:7444/lookupservice/sdk -u administrator@vsphere.local -p YourPassword -si D:\certs\vCenterSSO\sts_id -ip d:\certs\vCenterSSO\sts.properties

4. If all goes well then your screen should look similar to the one below, with three success messages. If not, you did something wrong. Depending on what’s goofed up, SSO may be in a hosed state and require a re-install.

10-9-2013 8-16-46 PMA

5. Now that the services have been updated, we need to overwrite some certificates. Navigate to the C:\ProgramData\VMware\CIS\runtime\VMwareSTS\conf directory. Backup the ssoserver.crt, ssoserver.key and ssoserver.p12 files. In the vCenterSSO directory that the toolkit script created, copy the ssoserver.crt, ssoserver.key and ssoserver.p12 files and overwrite the old versions.

6. In an elevated command prompt type:

net stop vmwarests
net start vmwarests

Verification

To verify that the SSO service is using the new certificate and didn’t suffer fatal stab wounds, open your favorite browser and go to the lookup service URL. That should be https://vCenterName:7444/lookupservice/sdk It should open without any SSL errors and if you look at the certificate by clicking on the lock icon, it will be issued by your CA. The XML response below is normal (yes even the Unexpected EOF), since we didn’t give the SDK service any input data.

10-9-2013 7-02-19 PM

10-9-2013 7-05-34 PM

Summary

With the help of my Toolkit script and the VMware automation tool, replacing the SSO certificate is not as painful as it was in the early months of vCenter 5.1. Even if you need to replace it manually, the toolkit does a lot of the tedious work to make success more likely. Next up in Part 11 is installing the Web Client, updating the SSL certificates, and configuring IE 10.

vSphere 5.5 Install Pt. 8: Online SSL Minting

10-3-2013 8-04-58 PM

In the last installment of the vSphere 5.5 series we installed the SSO service. Now that the Java JRE is installed (via the SSO installer), we have the tools ready to create our vCenter SSL certificates via an online Microsoft CA. If you don’t have an online Microsoft CA that can issue your VMware SSL certificates, skip this section and go to Part 9 (coming soon), where we go through the manual process.

I’m recommending you create the SSL certificates now, so that you have a variety of methods at your disposal to use these certificates. VMware’s stance is that you fully install vCenter 5.5 with self-signed certificates then use their free certificate automation tool to replace all of the certificates as the last step. That is certainly a good route to go, and I would not dissuade you from that method. Or, you could also replace certificates as you install components so each layer is trusted as you go. Either way, I recommend the VMware certificate tool even if it is a bit primitive. It is flexible enough to let you incrementally replace certificates.

In order to make life easier for installers I’ve written a “toolkit” PowerShell script to help with the SSL process. More details are below, but I would like to give credit to Chrissy LeMaire for some of the (modified) building blocks of this script. She wrote a vCenter 5.1 PowerShell SSL replacement script that was more automated than VMware’s batch script. My script does not replace VMware’s automation tool, but helps you prepare the files it needs.

Download Toolkit Here

Blog Series

SQL 2012 AlwaysOn Failover Cluster for vCenter
vSphere 5.5 Install Pt. 1: Introduction 
vSphere 5.5 Install Pt. 2: SSO 5.5 Reborn 

vSphere 5.5 Install Pt. 3: vCenter Upgrade Best Practices and Tips
vSphere 5.5 Install Pt. 4: ESXi 5.5 Upgrade Best Practices and Tips
vSphere 5.5 Install Pt. 5: SSL Deep Dive
vSphere 5.5 Install Pt. 6: SSL Certificate Template
vSphere 5.5 Install Pt. 7: Install SSO
vSphere 5.5 Install Pt. 8: Online SSL Minting
vSphere 5.5 Install Pt. 9: Offline SSL Minting
vSphere 5.5 Install Pt. 10: Update SSO Certificate
vSphere 5.5 Install Pt. 11: Install Web Client 
vSphere 5.5 Install Pt. 12: Configure SSO
vSphere 5.5 Install Pt. 13: Install Inventory Service
vSphere 5.5 Install Pt. 14: Create Databases
vSphere 5.5 Install Pt. 15: Install vCenter
vSphere 5.5 Install Pt. 16: vCenter SSL
vSphere 5.5 Install Pt. 17: Install VUM
vSphere 5.5 Install Pt. 18: VUM SSL
vSphere 5.5 Install Pt. 19: ESXi SSL Certificate

…possibly more to come…

Derek’s Toolkit Script

The PowerShell script performs several tasks and is menu driven. It’s an all in one script, meaning it handles online/offline CAs, and will also do other install tasks like create your ODBC connectors. That functionality is not yet in there, but will be added in the coming weeks. The full feature list will change and so will the menus. But I’ll try and keep this updated as often as I can.

The CSRs are in strict accordance with VMware KB articles regarding certificate requirements, including an optional IP address in the SAN field. I want to strictly color within the KB article lines, so if you do use this script and then have to call up VMware support they won’t roll their eyes and have you re-do your certs because some blogger got it wrong.

The script has the following features:

  • Downloads and installs the proper version of OpenSSL if it’s not already installed
  • Creates 2048 bit RSA private keys in the proper format
  • Creates a directory for each service bundle of SSL certificates
  • Generates seven OpenSSL configuration files, one for each certificate, in the appropriate directory
  • Downloads both root and subordinate root public certificates
  • Submits the CSRs to the online CA and downloads the certificates
  • Creates the needed service PEM files for the vCenter certificate automation tool
  • Creates the required root/subordinate PEM files
  • Handles the special SSO 5.5 certificate requirements
  • Does NOT require PowerCLI
  • Assumes all vCenter components are on one server
  • Automatically uses the hostname of the server you run the script on for all certificates
  • Creates a pre-filled vCenter Certificate Automation environment script – Just run!
  • Works with offline CAs
  • Creates SSO 5.5 certificate replacement files – Only used if manual replacing certs
  • Creates customized SQL vCenter and VUM database creation script
  • Creates SQL ODBC DSNs for vCenter and VUM
  • Automatically downloads and installs SQL 2008 R2 or SQL 2012 client package
  • Linux vCenter Server Appliance support for online minting and offline CSR creation
  • Creates certificates for Auto Deploy, Dump Collector, Syslog collector, Authentication Proxy
  • Support Microsoft CAs that require manual certificate approval
  • Requires PowerShell 3.0 or higher

Download Toolkit Here

Toolkit Change Log

v1.56, January 19, 2014

  • Fixed bug when no subordinate CA was present
  • Changed Microsoft “renewal” default to 0 for root/sub CA

v1.55, January 12, 2014

  • Added additional CA/subordinate error checking

v1.50, December 22, 2013 Changes:

  • Added support for ESXi hosts

v1.42, December 3, 2013 Changes:

  • Modified how the certificate hash files are created
  • Added Authentication Proxy certificate creation
  • Changed MS CA download parameter Renewal from 0 to 1

v1.41, Nov 14, 2013 Changes:

  • Changed the root/intermediate CA download order and added more error checking

v1.4, Nov 10, 2013 Changes:

  • Added Auto Deploy, Dump Collector and Syslog Collector SSL certificates
  • Added support for manually approved CA certificates (Windows & Linux)
  • Added SHA512 request in CSR
  • Added request for vCenter FQDN in Option 3

Special thanks to Ryan Bolger from Trace3 for the CA manual approval code.

 

Online SSL Minting

1. Download my vCenter 5.5 toolkit script from the links above. Open it in the PowerShell ISE (or favorite editor). The PowerShell script requires a few variable modifications before you run it. In the first block of variables you need to setup the directory where you want all of the certificates to go. If OpenSSL is already installed, change the path so the script knows where the root directory is. If that directory does not exist OpenSSL will be downloaded and installed for you. Next up are the certificate properties. Change those to suite your environment. If you want the server’s IP address in the SAN field, then uncomment the line and change the IP.

10-10-2013 7-04-44 PM

2. The script is semi-intelligent about using only a root, or one subordinate and root. Simply comment out $SubCA with a # if you only have an online Microsoft root CA. If you have two or more subordinates, then you will need to follow VMware SSL KBs or modify my script. Sorry!

10-7-2013 8-17-46 PM

If for some reason the script can’t download your CA certs automatically, try changing the download protocol. The default is HTTPS. If that still fails, you can manually place the Base-64 encoded root (and intermediate) certificates into your $Cert_Dir path. The root should be called Root64.cer and the intermediate called interm64.cer.

10-10-2013 7-08-34 PM

10-10-2013 7-12-20 PM

3. The next section are the details for your issuing CA and the template. The issuing CA is your online CA that will actually mint your certificates. If you only have one CA, then clearly that is what you should use. The $ISSUING_CA field can be a little tricky. The first field is the shortname (or FQDN) of your CA (e.g. d001dc01). Next up is your CA name. This can be anything, so you must open the Certificate Authority MMC on your CA to find out what it’s called. As you can see from my screenshot below my CA name is contoso-D001DC01-CA.

10-3-2013 9-06-22 PM

10-9-2013 4-35-46 PM

Now if you don’t have MMC access on the CA to look up the actual CA name, then you can find it another way. Open a browser and go to https://yourCAserver.domain.com/certsrv and select Download a CA certificate, certificate chain, or CRL. Select Download CA Certificate Chain. Open the files, then open the certificate(s) in the file. Now depending on how your CAs are setup, it may take a little thought to correlate the “Issued to” and “Issued by” to your root/subordinate CA. In my case contoso-D001DC02-CA is my root CA, and contoso-D001DC01-CA is my subordinate. I want my VMware certs issued from the subordinate, so just like the MMC screenshot above, the CA name that corresponds to D001DC01 is contoso-D001DC01-CA.

12-3-2013 6-43-46 PM

4. Next up is the template name. This can also be any value, but if you followed my guide then it will be called VMware-SSL. This is the Template Name not the Template display name.

10-3-2013 9-08-57 PM

10-9-2013 4-37-42 PM

5.  Your account must have the required CA permissions to enroll for the VMware-SSL template. If it does not, then find a CA administrator, have them logon the vCenter server and run the script. If your online Microsoft CA is configured for manual certificate approval, you can still use Option #1.

1-9-2014 9-01-08 PM

6. Since you have properly configured the script variables and have one or more online Microsoft CAs, you should select option 1. New to v1.3 and later it will ask you to confirm your vCenter FQDN. If the FQDN is correct, just press ENTER. If it is wrong, just enter the right FQDN.

10-22-2013 8-30-54 PM

After confirming/entering your FQDN the process is automated. If you do get errors, then you either goofed up the variables, have insufficient permissions, or my script is broke and needs fixing. If its broke, now is an excellent time to learn PowerShell. Script is provided as-is, and bugs/issues may or may not be fixed.

Below is a screenshot with a sample of the script output as it runs using an online CA with automatic approval. A lot more has scrolled off the screen, but you get the idea. There is limited error checking, but subtle issues could fly by on the screen. Review the output for any issues.

10-9-2013 5-00-02 PM

7. If your CA is configured for manual approval you will get a list of the request IDs for all 10 certificates. Have your CA administrator approve the requests, then run Option #3 to complete the process. You will only see the Request IDs when manual request approval is required. When you run Option #3 the output should look like the previous screenshot.

11-10-2013 7-10-23 PM

Output Validation

1. When the script completes you should have eleven directories, and either one or three certificate (.cer) files in the root of your working directory. If you have a subordinate CA then you will have three files. If you have a single CA you will only have a Root64.cer file. The two files with funny names are hash files of the root and intermediate CAs. If you only have a root CA you will see a single hash file.

12-3-2013 6-52-23 PM

2. If you take a peek inside one of the folders you will see a series of files. Each service, except SSO, will have the same set of files (except the .csr and .cfg with are uniquely named). The

  • chain.pem: Used for the VMware vCenter certificate automation tool
  • rui.crt: Public half of your SSL certificate
  • rui.key: Private half of your SSL certificate
  • rui.pfx: Combined private and public SSL keys
  • *.cfg:  Certificate signing request file
  • *.csr: Certificate signing request

10-9-2013 5-09-43 PM

3. In the vCenterSSO you will see a plethora of files. Depending on how you replace your SSL certificates, you may only use some of these files. But to help you out as much as possible, all the SSO files that are tedious to create manually are created for you. If you are missing files, then something went wrong. Please match up all filenames to validate the toolkit script worked. Some files are copies of each other, but they are needed to avoid confusion and more easily follow the KBs.

  • *.properties: Use for manual SSO SSL replacement
  • *_id: Use for manual SSO SSL replacement
  • ca_certificates.crt: Use for manual SSO SSL replacement
  • root-trust.jks: Used for SSO/STS certificate validation
  • server-identity.jks: Same file as above with a different name (per VMware KBs)
  • ssoserver.p12: Same functionality as rui.pfx, but VMware changed the name and format for SSO 5.5
  • ssoserver.crt: Copy of chain.pem
  • ssoserver.key: Copy of rui.key

10-9-2013 10-06-14 PM

Certificate Validation

Now that your certificates are minted, let’s quickly validate all of the properties are present. Some users have reported corrupted/incorrect root and subordinate certificates, so please do NOT skip this section. Also, even if your CSR requests a property (such as client authentication), that doesn’t mean your CA will honor it. The OU in each subject name should be unique and match the directory its in.

10-10-2013 7-17-04 PM

The Subject Alternative Name should contain the short name and FQDN. Optionally it can contain your IP address too.

10-10-2013 7-18-18 PM

Enhanced key usage should show server and client authentication. Client authentication can be missing if the CA template is wrong.

10-10-2013 7-18-59 PM

Key usage should contain digital signature, key encipherment and data encipherment.

10-10-2013 7-19-43 PM

We also need to validate that the root and intermediate certificates are in the right format. Some users have reported corrupted root or intermediate certificates. In Notepad open both your root64.cer and interm64.cer files. They should both start with —–BEGIN CERTIFICATE—–. If they contain HTML or any non-printable characters, then they are corrupted or in the wrong format. STOP, as they will most certainly NOT work.

1-9-2014 8-52-00 PM

If your root or subordinate CA certs contain HTML, and you are using the script to automatically download them from a Microsoft CA, I have a suggestion that may work. Locate the code snippet below and change Renewal=1 to Renewal=0. That tweaks how the script downloads the certificate from the Microsoft CA. In addition, if the script downloads an old subordinate certificate, change renewal=0 to renewal=1 in the DownloadSub function.

1-9-2014 8-55-17 PM

Summary

Assuming you have an online Microsoft CA and you were successful in running the script, you now have all of the files needed to use the VMware certificate automation tool, or go through the manual certificate replacement process. In Part 9 I cover how to use an offline or non-Microsoft CA. At the end of that article you will have exactly the same files as you do from this installment. Starting in Part 10 we will resume our configuration and installation of vCenter components.

vSphere 5.5 Install Pt. 5: SSL Deep Dive

SSL certificateNow that you’ve read up on upgrade best practices and SSO enhancements in the first four installments, we can get to the fun stuff! Understanding vCenter 5.5 SSL certificates. Ok maybe not fun for all, but fun to me. Last year in my 5.1 install series I just plowed right into the minting process, but didn’t really explain the big SSL picture. Since SSL is probably the biggest point of confusion in vCenter 5.1 and later, this post will serve to illuminate one of my favorite subjects. After all I have to keep up my “SSL guy” nickname. Do not fear, this series will eventually get to actually mounting the vCenter ISO and doing a software install. But I hope these foundational pre-install articles enable you to better understand the product.

Blog Series

SQL 2012 AlwaysOn Failover Cluster for vCenter
vSphere 5.5 Install Pt. 1: Introduction
vSphere 5.5 Install Pt. 2: SSO 5.5 Reborn 
vSphere 5.5 Install Pt. 3: vCenter Upgrade Best Practices and Tips
vSphere 5.5 Install Pt. 4: ESXi 5.5 Upgrade Best Practices and Tips
vSphere 5.5 Install Pt. 5: SSL Deep Dive
vSphere 5.5 Install Pt. 6: SSL Certificate Template
vSphere 5.5 Install Pt. 7: Install SSO
vSphere 5.5 Install Pt. 8: Online SSL Minting
vSphere 5.5 Install Pt. 9: Offline SSL Minting
vSphere 5.5 Install Pt. 10: Update SSO Certificate
vSphere 5.5 Install Pt. 11: Install Web Client 
vSphere 5.5 Install Pt. 12: Configure SSO
vSphere 5.5 Install Pt. 13: Install Inventory Service
vSphere 5.5 Install Pt. 14: Create Databases
vSphere 5.5 Install Pt. 15: Install vCenter
vSphere 5.5 Install Pt. 16: vCenter SSL
vSphere 5.5 Install Pt. 17: Install VUM
vSphere 5.5 Install Pt. 18: VUM SSL
vSphere 5.5 Install Pt. 19: ESXi SSL Certificate

Permalink to this series: vexpert.me/Derek55
Permalink to the Toolkit script: vexpert.me/toolkit55

Who cares about SSL?

Why should you go through the headaches of replacing all the VMware self-signed certificates? What’s the risk of using untrusted certificates? What can happen if the SSL connection is compromised?

Hypervisors are likely the underpinnings of your business critical apps and intellectual property. If your hypervisor is compromised then it’s just a few short commands to access your critical business data. Unless you like your infrastructure being p0wn3d, then you don’t want your VMware infrastructure compromised. If you don’t use trusted certificates, and just click through all the VI client SSL warnings (you have clicked Ignore and trust this certificate many times…haven’t you?) then you won’t know that a man-in-the-middle attack has taken place.

A man-in-the-middle attack is where a third party intercepts your “secure” communications and relays data between you, the attacker’s device, and the end host (like an ESXi server). This can be accomplished by ARP spoofing, or other means. If it’s second nature to ignore and click through all VI certificate warnings, you will have no idea your credentials have been intercepted….in clear text. No fancy brute force decryption required. Just sit back, grab a coke, and enjoy cleartext flowing across your screen. An interesting article on attacking VMware is here.

The diagram below shows the man-in-the-middle attack, and the following graphic is the clear text credentials from the VI client. With those credentials the bad guys can now own your infrastructure.

10-2-2013 9-04-19 PM

10-2-2013 9-23-20 PM

If you use trusted SSL certificates and untrain your staff to ignore SSL warnings, then it’s significantly harder to spoof the communications.  Think of this this way: Would you enter your credit card details or social security number on a site that threw up constant SSL warnings? No. So why trust potentially billions of dollars of IP to untrusted certificates that your grandma could spoof. There’s a reason why financial institutions, Government and other high value targets require real certificates. Don’t think that just because you run Nana’s Cookie shop that the bad guys aren’t after you…they are.

To be clear, man-in-the-middle attacks and SSL certificate spoofing are not unique to VMware or because they did something wrong. It’s a problem across the industry with self-signed certificates. I hope you are sufficiently scared to take certificates seriously and replace them. Even if you bake cookies in your garage and sell them on the corner stand, I’d update your SSL certs.

Certificate Requirements

Last year with vCenter 5.1 there was some confusion around certificate requirements. Ok, A LOT of confusion. How many certs do you need? What CA template options? What is a certificate OU?  I was stumbling around in the dark last October since official guidance at that time was a wee bit skimpy. Thankfully that situation has been largely remedied by VMware. The list below is a combination of the vCenter Certificate automation tool requirements and of the certificates themselves:

  • Platform Support: Windows Server 2008 R2 SP1 or later, including Windows Server 2012 (not R2)
  • Private Key Algorithm: RSA with 1024 bits or higher (I use 2048). No fancy elliptical curve support.
  • Recommended Signature Algorithms: SHA256, SHA384, SHA512
  • NOT Recommended algorithms: MD2, MD5, SHA1
  • Path or certificate filename can’t contain: ^ (caret), % (percent), & (ampersand), ; (semicolon), ) (closing parenthesis)
  • Use PEM certificate format, with a header of —–BEGIN CERTIFICATE——
  • Must use a complete certificate chain
  • Use OpenSSL v0.9.8y (Do NOT use OpenSSL 1.x !!). Recommend 32-bit version.
  • Does NOT support wildcard certificates
  • Have the subject alternative name (SAN) field included in them
  • Have unique OrganizationalUnitNames for the components (which in turn creates a unique Distinguished Name (DN))
  • Key Usage: Include digitalSignature, keyEncipherment, dataEncipherment
  • Seven unique certificates: InventoryService, SSO, UpdateManager, vCenter, WebClient, LogBrowser, Orchestrator

As you can see, there are a number of very specific requirements to meet in order to have success with replacing your vCenter SSL certificates. These are essentially unchanged from vSphere 5.1, but are now documented. Thank you VMware! Official sources of the requirements are here and here, if you have a burning desire to read more…before you fall asleep.

One final requirement that can trip people up is the format of the RSA private key file (rui.key). VMware states that only a key file with the —–BEGIN RSA PRIVATE KEY—- string will be accepted. Newer versions of OpenSSL may format it differently, creating problems down the road.

RSA private key

Certificate Subject Properties

One of the areas of confusion is exactly what certificate properties are important for a successful trusted certificate implementation. If we look at a SSL certificate which we will be generating in an upcoming post, we see a number of properties that may be Greek to most people.

In the example below we see properties common to all our SSL certificates. For example, we can see the signature hash algorithm is the recommended SHA256. You can also see validity dates, which are important. The primary point of confusion, and what took a lot of detective work when 5.1 went GA, is what needed to be in the Subject field. The SSO service in 5.1 and carried through to 5.5, requires a unique DN (distinguished name) for each certificate. The most common way to make the DN unique is to modify the OU value for each certificate. You could modify another field, but I would not recommend it. For all practical purposes the OU must be unique.

As you can see in the first certificate the OU value is vCenterSSO. In the second certificate the OU is VMwareUpdateManager.  While you don’t have to use these exact strings, they must be unique within the SSO certificate store. I’m opting go with the ‘official’ OU descriptions. Yes, even if you do the simple install with all components on a single VM, every component on that server must have a different certificate. You can NOT use wildcard certificates, either.

Commercial CAs can also have problems minting these properties. Although I’m not sure why you’d want to spend the money on a public CA certificate for an internal service. But some people try it…and are usually disappointed with the results.

vCenter SSO certificate VMware Update manager certificate

Certificate Key Usage

There also has been some lively discussion around other certificate properties, and what is and is not required. Certificate key usage defines how the OS or application can use the certificate. For example, can you only use it to digitally sign an email? Or only for code signing? Or just to secure RDP traffic? Those ‘features’ are enabled by Key Usage and Enhanced Key Usage fields.

For the purposes of this series I’m using KB 2037432 as a reference. As you can see in the graphic below, which is a VMware provided certificate request template, you see various fields. Highlighted are the KeyUsage and extendedKeyUsage fields. Of some controversy are whether dataEncipherment and clientAuth options are really required. Those are not properties normally allowed if you use the Microsoft “web server” template. For the purposes of this series I’ll assume they are required and will show you how to create a custom CA template to mint them. YMMV, and possibly the “web server” template will work fine for you. VMware must have a good reason to include them, so I’m not going to argue. My cooler is stocked with bigger VMware fish that need to be fried.

VMware certificate request

If you look at the properties of a properly formatted certificate you will see I used the correct template which enabled the minting of a certificate with said key usage requirements.

SSL data encipherment SSL authentication

Subject Names

Another important certificate property is the subject name and the subject alternative name (SAN). The subject name (commonName) is the FQDN of your vCenter server (assuming all components are on a single VM). Even if you are using the simplified deployment model all seven certificates will have the same commonName. The subjectAltName are alternative names that the host is known by. For those of you familiar with Microsoft Lync or Exchange deployments, you will be familiar with SANs.

There’s a little debate on what needs go to into the SAN field. Specifically, do you include the IP address or not. In the official VMware certificate template request they include the IP (shown below). However, at VMworld they said not to include the IP. Authentication uses Kerberos, which requires hostnames (can’t use IPs). Since the certificate replacement process is challenging, having to re-issue certs if the IP changes is not remotely appealing. My stance is to NOT include IPs in the certificate.

An interesting note I found in KB2037432 was that the FQDN in the SAN field was moved to the last value to support SRM. Curious that the order of SAN fields would matter, but apparently it does for SRM. Sounds like a bug to me.

SSL subject alternative name

Certificate Signing Request (CSR)

In order for a CA to mint you a certificate you must submit what is called a CSR, or certificate signing request, to your CA. This request contains all of the properties in the request template that you see above. It also contains data which is tied to the private RSA key. In our case we will use OpenSSL to process configuration files for each of the seven certificates, then submit the CSR to the CA, where we can then download our freshly minted certificate. The CSR will be in Base64 format, meaning it’s a neatly formatted text file you can open with Notepad and peek at. Although you will need to be Neo to decode it on the fly.

Certificate Authority Hierarchy

In many environments you will have more than one certificate authority. Typically an enterprise would have an offline root CA that is tightly controlled and secured. You then have one or more online issuing CAs which are joined to the domain and issue your certificates. Or, you may have a simplistic model where a single online CA does everything. If you have two or more tiers of CA, then you have some extra work cutout for you. Why? vSphere needs information about all tiers so that it can properly validate the chain of trust all the way to the root. Subsequent installments will show you exactly what extra steps are needed.

Certificate Creation Process

The certificate creation process is fairly simple once you understand the process. First, OpenSSL creates a private RSA key, usually 2048 bits long. That is used in conjunction with the certificate configuration file by OpenSSL to generate a CSR. The CSR is submitted to the CA, and out pops an SSL certificate at the other end. Nifty!

SSL certificate process

PEM Certificates

The vCenter Certificate automation tool requires certificate files to be in a very specific format. If they are not in that format (PEM), then you will have show-stopping problems and wonder why everything is a smoking pile of ashes. PEM encoded files are a series of Base64 certificates in a specific order. First up the tool needs the public certificates for your CA(s). If you have a subordinate CA, then this file (chain.cer) needs to have both the root and subordinate CA BASE64 certificates concatenated together. Do not add any spaces or lines, as that will not work. An example is shown below.

PEM certificate

In addition to the root CAs, each of the seven services needs a chain.pem file that is a combination of your root CA(s) and the certificate for your service. Again, don’t add spaces or lines in the file or you will run into problems.

PEM certificate

Don’t worry about creating the files now or where they go. Coming up is a PoweShell script that does all of the work for you. But if you are troubleshooting, then understanding the file format should help. I know I got tripped up on it last year and embarrassingly opened a support ticket. But I probably wasn’t the only one. Now you won’t have to be one!

Summary

Now that you hopefully understand a bit more about the certificate requirements, you will be in a better position to mint all of our SSL certificates. SSL certificates are complicated, and even in vSphere 5.5 they remain largely unchanged. Paths where VMware stores the certs are somewhat different, but the big overhaul that was needed did not happen. There are some cool ‘under the covers’ changes to certificates that I haven’t seen publically talked about yet which I’ll cover in the future. The teaser is that it sets the foundation for better certificate management in the future.

Next up in Part 6 is how to configure your Microsoft CA to issue SSL certificates with all the required properties.

VMware SSL pain? vCert Manager to the Rescue

VSS LabsOne of the most critical aspects to securing your VMware vSphere environment is replacing the self-signed certificates with trusted certificates. VSS Labs is releasing a new product, called vCert Manager, which will vastly improve the SSL certificate management experience for VMware customers.

Over the years I’ve written several articles detailing the process, which got exponentially harder with vSphere 5.1. I often hear users complain about the SSL certificate replacement process. Feedback on my blog has been quite verbose and colorful…some posts had to be censored. If you are struggling with the vSphere 5.1 installation, you can check out my vSphere 5.1 resources here.

While VMware did release the vCenter Certificate Automation tool earlier this year for usage with vCenter 5.1, it is command line/menu driven and does not fully automate the entire process. Given this tool was the only game in town I recommended my readers use the tool. You can find my multi-part usage series for the tool here.

The real game changer is the enterprise-scale VSS Labs vCert Manager. This goes way beyond the basic vCenter Certificate Automation tool, and provides full certificate lifecycle support for securing your vSphere environment with trusted SSL certificates. By full lifecycle I mean from automated vSphere infrastructure discovery, CSR generation, certificate minting, certificate installation, email expiration notices, automatic expired certificate updating, auditing, reporting, plus a lot more.

The entire process is managed via a web interface (securable with SSL), and requires no knowledge of OpenSSL, CSRs, PEM files, or leaf certificates. Plus it supports vSphere 4.x and 5.x environments, including ESXi hosts. The full support matrix is below.

If you are attending VMworld 2013 in San Francisco, be sure to stop by the New Innovator Pavilion and find the VSS Labs booth. Once you see the tool in action, you will be chomping at the bit to get a copy.

vCert Manager Features

Since the product has a very long feature list I’ve put together three tables that compare and contrast the features of the VSS Labs vCert Manager product and the VMware Certificate Automation Tool v1.01. The first table covers the VMware product/feature support of each tool. The second table covers certificate support, and the third table compares the tool core feature set and licensing.

VMware Product Support

VSS Labs VMware Support

As you can see from the table above, vCert Manager supports a broad range of VMware infrastructure. The VMware vCenter Certificate Automation tool is limited to just replacing vCenter 5.1 certificates. Version 1.01 has no support for prior versions of vCenter or ESXi hosts. Good tool for a specific use case, don’t get me wrong.

In contrast the vCert Manager product supports vSphere 4.0 through 5.1, including their respective ESX/ESXi host versions. I expect both companies to release updated certificate management products as new vSphere releases hit the streets. Neither tool has yet expanded to vCOPS, vCloud Director, or Horizon View. VSS Labs is targeting vCloud Director and Horizon View support in the near term, likely around the end of September.

Another great feature is that vCert Manager can manage dozens of vCenter instances and thousands of ESXi hosts from a single server and interface. No need to install the tool on each vCenter server. Credentials are server and host specific, so you can manage them all even if they are in different business units with different credentials.

Certificate Support

VSS Labs Certificate Support

Next to comprehensive VMware product support, broad certificate authority support and a high degree of automation is extremely important. This is where the vCert Manager product really shines. The VMware vCenter Certificate Automation tool requires a bit of manual pre-requisite work and using intermediate CAs can be tricky. Version 1.0.1 added CSR generation capability, so I certainly appreciate the added ease of use.

In contrast, vCert Manager automates the ENTIRE process, and you don’t have to know anything about certificate properties, SANs, how many certificates you need to mint, or how to submit and download certs from your CA. It can be configured to interface with an online Microsoft CA, so everything happens behind the scenes. Offline CAs are supported to, and as much of the process as possible is automated given the human intervention needed to mint the certificates.

In short, if you have an online CA the entire certificate request creation, minting, and deployment is executed through a GUI interface in an automated and orchestrated fashion. This makes certificate replacement almost foolproof.

Tool Features

VSS Labs vCert Manager Features

The most striking difference to me between the two tools is the user interface. vCert Manager sports a fully webserver driven GUI interface for all operations. If you haven’t used the VMware vCenter Certificate Automation tool you may not know that it’s completely command window based with text menus that instruct you in what order to perform the various steps. The VMware tool also needs to be installed on each server that has a vCenter component, unlike the vCert Manager which is installed on a single server and can manage thousands of vSphere components.

When updating ESXi hosts it requires DRS be enabled on the cluster so that it can put the host in maintenance mode, update the certificate, and return the host into production. This is all automated and when certificate is about to expire, it will automatically replace the certificates in the same manner.

Other great enterprise features of vCert Manager include syslog, full audit trail, basic reporting, and role-based access controls. Unlike the VMware tool, vCert Manager DOES require a Microsoft SQL database. So that’s a little extra configuration, but the installer does most of that work too. It supports Windows Authentication to the SQL database. SQL 2008, 2008 R2, and 2012 are supported.

Licensing and Support

There of course has to be a discussion about licensing and support. The VMware tool is free and if you have active SnS with VMware, they will support the tool for production usage. Clearly that’s a great value, and does ease much of the pain with vSphere 5.1 certificates. I certainly appreciate VMware releasing the tool, and the improvements in v1.01. I expect more improvements in the future.

vCert Manager is a licensed product, and support is provided by VSS Labs. The licensing model is per vCenter server, and per actively managed component (such as an ESXi host). For pricing details you will need to contact VSS Labs.

How to get a Free License

VSS Labs has a free NFR license available for home lab usage, with a special bonus for current fellow VMware vExperts. The tool will discover all of your components, even if they exceed your license, but you can only actively manage the number up to your licensed threshold.

8-19-2013 8-50-46 PM

Summary

vCert Manager should go GA by the end of August, if not sooner. When it does go live I’ll post a link to the download. I’ve been using the beta versions, and once the code goes GA I will post a multi-part blog post series on using the tool. So stay tuned!

For a version 1.0 product, I’m very excited to see the depth of support vCert Manager has packed into their first release. Throughout the beta process the company took feedback very seriously. After spending countless hours writing blog articles on VMware SSL certificates over the years, you have no idea how glad I am to see such a comprehensive tool. Maybe less blog articles for me to publish, but better for the community. Less hair pulling, and frustration for all!

The small home lab NFR licenses will give anyone interested in the product a good feel for how it works. I really appreciate the vExpert “bonus”, which will help the community test it against various vCenter versions more easily.

Sample Screenshots

The screenshots below are from a beta version, but will give you a little taste of the user interface and features. My installation series will contain a boatload of screenshots, once I get the GA code.

vCert Manager

vCert Manager Summary

vCert Manager Configuration

vCert Manager certificate authority

vCert Manager managed components

Import IIS SSL Certificate to Citrix NetScaler

5-18-2013 10-03-29 PMFor a recent project I’ve been configuring a Citrix NetScaler (which are wickedly cool) for load balancing of a web service over SSL. The web service is hosted on a Windows server using IIS, so I wanted to re-use the SSL certificate on the NetScaler. The steps to import IIS SSL certificate to NetScaler are actually fairly easy. I found various blog articles and Citrix KB articles on the process, but they were a bit convoluted and I thought there had to an easier process than using OpenSSL and WinSCP/NotePad to manipulate the certificate files.

The first thing you need to do is look in the server’s computer certificate personal store for your IIS certificate. In my case I’m looking for the StoreFront.contoso.net certificate. Since I knew I’d be exporting the whole certificate (including the private key), I made sure when I was requesting the certificate to allow the private key to be exported. You can request certificates from your MS CA a variety of ways, so I’ll assume you can find the option to allow private key export.

5-18-2013 8-52-39 PM

Exporting the Certificate

1. Right click on the certificate select All Tasks then select Export. You should be presented with the option to export the private key. If not, then your certificate’s private key is “stuck” in the computer’s store and you can’t get it out. Issue a new certificate with the private key export option.

5-18-2013 8-58-01 PM

2. Assuming you can export the private key you are now given some options for the PKCS#12 certificate file. You shouldn’t need to select any of the options.

5-18-2013 8-59-29 PM

3. Select a strong password to protect the file with. Remember it.

4. Chose an appropriate filename for the certificate. I strongly suggest using the FQDN of the certificate, because the NetScaler will store the files with the name you choose. So don’t do something like “cert.pfx” since you will have no clue what site it is for. In my case I chose StoreFront.contoso.net.pfx.

5. Run through the same export wizard again, but this time select No, do not export the private key.

5-18-2013 9-04-19 PM

6. Select Base-64 encoding for your certificate.

5-18-2013 9-05-03 PM

7. Again, I suggest using the FQDN of the certificate for the filename (e.g. StoreFront.contoso.net.cer). Make sure the file ends in “.cer”.

8. At this point you should have two certificate files, both with the FQDN, and one ending in .PFX and the other in .cer.

5-18-2013 9-06-50 PM

Importing Certificates into NetScaler

1. Logon to your Citrix NetScaler and open the root SSL page. Under Tools click Import PKCS#12.

import iis ssl netscaler

2. In the import window click on Browse next to the PKCS12 filename (NOT the output file name). Browse to your pfx file. Type in the password you entered during the certificate export process. Enter a new password to protect the private key on the Netscaler (PEM passphrase). In the Output File Name use the FQDN of the certificate and add a .key suffix. Change the encoding format to DES3. The NetScaler will automatically extract the private key from the PFX file and put it into the .key file.

5-18-2013 9-12-17 PM

3. Click on Manage Certificates / Keys / CSRs. Upload your .cer file. You should now see three certificate files with your certificate’s FQDN.

5-18-2013 9-17-12 PM

4. At this point you can delete the .pfx file if you wish, since we no longer need it. I suggest you do remove it, to reduce clutter on your NetScaler.

5. In the left pane under SSL click on Certificates then in the middle pane click on Install.

5-18-2013 9-18-41 PM

6. Enter the FQDN of your certificate in the Certificate-Key Pair Name. For the Certificate FIle Name select the .cer file you uploaded. For the Private Key File Name select your .key file. Enter the password you entered back in step 2.

5-18-2013 9-20-59 PM

7. If all goes well you will now have a new certificate from IIS installed on your NetScaler with no command line effort or manual modification of certificate files.

import iis ssl to netscaler

vCenter Certificate Automation Tool: Part 1 (Pre-reqs and Config)

For those of you that have installed vSphere 5.1 and tried to use your own trusted SSL certificates, you will probably find the experience extremely tedious, cumbersome, error prone, and vastly harder than any product you’ve used before. Now out is the VMware vCenter Certificate Automation tool v1.0, to make the process somewhat easier.

My 15-part vSphere 5.1 series goes into all of the gory details of the manual replacement process, and closely follows the associated VMware KB articles but in a more understandable format that people seem to appreciate. But, now the process can be made easier and less painful.

So what’s new on the vSphere 5.1 SSL front? Late last week VMware released their first stab at easing the SSL certificate replacement torture in vSphere 5.1 with a basic command line tool which helps automate the process. I covered the announcement here. Since I’m pretty familiar with the pain and suffering vSphere 5.1 certificates has caused me, I wanted to see if this tool would make life easier.

Since the process is a bit long and only semi-automated, I’ve broken down the process into a series of posts:

Part 2 (SSO and Inventory)
Part 3 (vCenter and Orchestrator)
Part 4 (Web client and Log Browser)

UPDATE 5/18/2013: I’ve included some certificate and chain.pem validation steps. The certificate tool is very picky about the formatting and contents of the chain.pem files. If you get the certs in the wrong order or have too many certs, the tool produces cryptic error messages. I strongly urge you validate your certificates and chain.pem files or you may be opening a support case with VMware.

Before you begin this process, you MUST read through all of the limitations (“Known Issues”) of the v1.0 tool. The list is pretty extensive, and there was one biggie that jumped out at me. If during the vCenter VUM installer you elect to use the server FQDN instead of the IP address (which I would argue is a best practice), then you can’t use this tool to replace the VUM certificate. Really? Ouch. The tool also doesn’t help you generate any of the certificates, or a new requirement of certificate chain files for each of the seven services. So you still have a lot of pre-work to get to the point of even trying to use the tool. This is not a fully automated end-to-end tool that is wizard driven, which we desperately need.

Going into this with my eyes wide open, and somewhat tempered expectations of what it will do for me, I decided to give it a whirl. VMware’s KB article on the tool guides you through the process, but as usual, I think the process can use a bit more elaboration and screenshots.

In my case I have a Windows Server 2012 CA, and all of the vCenter services are installed on a single Windows Server 2008 R2 VM. My vCenter databases are on an external SQL 2008 R2 VM.

Prerequisites

Since this tool doesn’t help you create the certificate request files, generate the certificates, or the new PEM chain files we must do that prior to using the tool. I’ve updated my vCenter 5.1 U1 Installation: Part 2 (Create vCenter SSL Certificates) to address the new requirements and the addition of the Orchestrator certificate. So open up that post and follow through the entire certificate generation process, except for creating the JKS keystore. The script at the end of that post has been updated to create the newly required chained PEM files for each service. So if you’ve used that script before, grab the updated version and run it.

If you did not use my script which creates the chain.pem files, don’t worry. I’ve included steps later in this article on how to manually create them. Now that you’ve run through that long post to create all the certificate files, you should have a directory structure that looks like the screenshot below. I have these folders residing under D:\certs.

Inside each of the seven folders you should have the same set of files, as shown below with the appropriate configuration file. Again, if your chain.pem file is not present, I’ve included steps below on how to create and validate them.

You will also need the following accounts and passwords handy to complete the process:

  • SSO administrator and password
  • vCenter administrator and password
  • Original vCenter database password

Configuring the Tool

1. Download the SSL Certificate Automation Tool from My VMware.

2. Copy it to your vCenter server and unzip it to a safe place, such as D:.

3. Open the ssl-environment.bat  file and fill in all of the missing paths. In my case I set the following:

set sso_cert_chain=D:\certs\sso\chain.pem
set sso_private_key=D:\certs\sso\rui.key
set sso_node_type=single
set sso_admin_is_behind_lb=
set sso_lb_certificate= set sso_lb_hostname=

set is_cert_chain=D:\certs\inventory\chain.pem
set is_private_key_new=D:\certs\inventory\rui.key

set vc_cert_chain=D:\certs\vCenter\chain.pem
set vc_private_key=D:\certs\vCenter\rui.key

set ngc_cert_chain=D:\certs\WebClient\chain.pem
set ngc_private_key=D:\certs\WebClient\rui.key

set logbrowser_cert_chain=D:\certs\LogBrowser\chain.pem
set logbrowser_private_key=D:\certs\LogBrowser\rui.key

set vco_cert_chain=D:\certs\Orchestrator\chain.pem
set vco_private_key=D:\certs\Orchestrator\rui.key

set vum_cert_chain=D:\certs\UpdateManager\chain.pem
set vum_private_key=D:\certs\UpdateManager\rui.key

set sso_admin_user=admin@system-domain
set vc_username=contoso\svc-vctr02-001

set last_error=
set ROLLBACK_BACKUP_FOLDER=%~dp0backup
set LOGS_FOLDER=%~dp0logs

4. Open an elevated command prompt and run the ssl-environment.bat script.

Creating PEM Files

If you used my script to automate the minting of your certificates then you already have the required chain.pem files. If you decided to go the more manual route, then you probably don’t have the required chain.pem files. They are easy to create, but also easy to screw up as the tool is very picky. Inside each service directory you need to have a chain.pem file. That file is comprised of the service’s certificate, intermediate CA (if you have one) and root CA certificates.

The VMware vCenter Certificate automation tool is very picky about whether your certificates are minted from a root CA or a subordinate CA. To find out which situation applies to you, open one of the certificates you’ve minted. If you only see one CA and the name of your vCenter server then then you don’t have a subordinate CA. If you see two or more CAs, then the one at the top is the root and the second one is the subordinate.

5-18-2013 3-49-48 PM

From inside each service directory you can use the following command to create the chain.pem file (assuming a subordinate CA):

copy /B rui.crt + D:\certs\root64-2.cer + D:\certs\root64-1.cer chain.pem

Repeat this process for every service directory. If you only have a root CA then the command would look like:

copy /B rui.crt + D:\certs\root64.cer chain.pem

Validate Chain.pem Files

You really should validate that the chain.pem files are properly configured. Having the certificates in the wrong order, or including an intermediate CA when your certificates are issued from a root CA will all cause the tool to #fail. Below is a truncated example of my root CA certificate file. I like to look at the last few characters of the cert, since they are very likely unique. Yours will of course be different.

Root64-1.cer (Root)

5-18-2013 4-26-36 PM

Root64-2.cer (Intermediate)

5-18-2013 4-28-46 PM

Opening one of the chain.pem files, if you are using an intermediate CA, shows that the service certificate is at the top, intermediate is in the middle, and the root is at the bottom. This is the correct order, and there must not be any blank lines before or after any certificate.

The certificate headers should also look exactly the same as below, with no extra information present. Even if your environment has an intermediate CA, if your certificates are issued from the root, do NOT include the intermediate CA. In that case you’d only have the service certificate at the top and the root at the bottom. Only include the CAs shown in your minted certificate, no more, no less.

5-18-2013 4-21-16 PM

If you jack up the order of the certs in the file you will get:

ERROR: One or more required parameters are not set or have invalid values: Certificate chain is incomplete: the root authority certificate is not present and cannot be detected automatically. The presence of the root certificate is required so the other service can establish trust to this service. Try adding the authority certificate manually.

Or if you include an intermediate CA but your certs are issued from the root:

ERROR: One or more required parameters are not set or have invalid values: The certificate chain file does not contain a valid certification path. PKIX path validation failed with: Could not validate certificate signature. (at certificate #1)

So please double check that your files are properly formatted, as the tool’s error messages are not that enlightening and don’t deal well with extra CA certificates.

Now that we have generated all of the required certificate files and set the environmental variables, we can walk through the planner and then actually replace the certificates. Replacing the SSO and Inventory SSL certificates are covered in Part 2.

VMware Horizon View 5.2 Install Part 2: SSL Certificate

This is the second part in a blog series of how to install and configure VMware Horizon View 5.2. In Part 1 we did the basic connection server install, and installed Adobe Flash player. Next up is configuring a trusted SSL certificate for VMware Horizon View.

There are a number of ways to request and mint SSL certificates. You could use a commercial CA, Microsoft internal CA or another flavor of CA if you wish. Unlike some vCenter components the View SSL certificate does not need any unusual properties beyond Server Authentication usage. No unique OU properties, no client authentication, no data encryption, etc. I would advise using a SAN certificate, so you can access the server via shortname and the FQDN without certificate errors.

I am using an Enterprise online Windows Server 2012 Certificate Authority in this example. The CA has been pre-configured to issue a variety of certificate template types, one of which I called “Server Authentication-SAN”. You don’t need a template with this name, but the template needs to support the SAN field, which the basic “computer” template will NOT. For general steps on how to configure a custom certificate template for a Microsoft CA, see my article here.

Additional articles in this series:

VMware Horizon View 5.2 Part 1: Basic Installation
VMware Horizon VIew 5.2 Part 3: Initial Config
VMware Horizon View 5.2 Part 4: VM and Pool Creation

VMware Horizon View SSL Certificate Installation

1. On the View server open a blank MMC. Add the Certificates snap-in and chose Computer account.

2. Open the Personal certificates container and expand Certificates. Depending on the auto-enrollment policy (if any) in your domain, you may find two or more certificates listed. One of the certificates will be the self-signed VMware certificate that we no longer want to use. You can see this by looking at the “Issued By” field.

3. Now we want to request a new certificate from our online CA via a the certificate request wizard. Right click on Certificates, select All Tasks, then Request New certificate.

4. A couple of clicks into the wizard you should see an Active Directory Enrollment Policy listed.

5. Click Next and you should now see one or more templates that your CA administrator has published. If you use the standard “Computer” template the CA will strip any SAN values that you enter. So if you want a SAN certificate you will need to use a CA template that allows for such usage. Since SAN certificate are not uncommon, I already had a certificate template ready. Again, for a link how to create a custom CA template see my article here.

6. Check the box next to your SAN template. Click on the line of text next to the yellow warning. On the Subject tab you now need to configure the “Common name” for the subject name and add two “DNS” alternative names. Use the View server FQDN for the Subject Name and add both the FQDN and short name DNS names for the alternative name, as shown below.

7. Click on the General tab and enter a friendly name of vdm.

8. Click on the Private Key tab and under Key Options allow the private key to be exportable.

9. Click OK then click on Enroll. If all goes well you should get a succeeded message.

10. In the MMC double click on the new certificate and validate all properties, including Subject Alternative Name are properly populated.

11. At this point you can either delete the self-signed VMware certificate, OR you must remove the vdm friendly name from the VMware certificate. View looks for a single certificate with the vdm friendly name. To remove the VDM friendly name from the VMware certificate just right click on the VMware certificate and select Properties, then delete the friendly name.
12. Restart all of the View services on your View server. The critical one is the VMware View Security Gateway Component. If it stops running shortly after you start it, there’s a problem with your certificate. The most common cause is having a certificate that does NOT allow exporting of the private key. You may see something like:

A fatal error occurred when attempting to access the SSL server credential private key. The error code returned from the cryptographic module is 0x8009030d. The internal error state is 10001.

13. Now you can launch the View administrator and change the URL to either the server’s short name or FQDN, and you should NOT see any browser SSL errors.

14. Once you login you can click on the Dashboard icon on the left and view the server details for your connection server. It should show a valid SSL certificate.

Congratulations on configuring your View Connection Server SSL certificate. Very easy, and straight forward (vCenter team are you listening?). Next up is Part 3, where configure basic parameters in the View Connection Server.

© 2017 - Sitemap