vSphere 6.0 Toolkit Update

In my new role at Nutanix I’ve had the pleasure of working with end customers, and configuring their vSphere 6.0 environment. During this process, SSL certificates have come up. Surprisingly, thus far my clients have chosen the VMCA method of deploying certificates. This is great, as it automates certificate deployments in a vSphere 6.0 environment. Even with the VMware certificate tools, there are some manual steps for configuring the VMCA. My vSphere 6.0 toolkit automates most of those steps.

However, while going through the process we stumbled upon a slight bug in my Toolkit when using an intermediate certificate authority. I’ve since fixed that bug, and uploaded the latest vSphere 6.0 SSL Toolkit here.

I’ve been exceptionally busy the last few months, which is why blogging and updating the Toolkit script has taken a back seat. But I did want to get this script update pushed out so other customers don’t run into VMCA problems.

If you are unfamiliar with my vSphere 6.0 SSL Toolkit, then read up on my full vSphere 6.0 installation series here.

vSphere 6.0 Pt. 12: PSC Machine Certificate

Back in Part 11 of this series we configured the VMCA to be a subordinate CA to our enterprise CA. This ensures that all certificates which get used by vCenter components are automatically trusted. But as previously mentioned, not all organizations can use the VMCA. The US Federal Government would be a prime example, where there’s no way you can stand up your own subordinate CA.

So if you are one of the organizations that can NOT use the VMCA and need to use custom SSL certificate throughout, this post is for you. In this post we will replace the PSC’s machine SSL Certificate with¬†a certificate issued by your enterprise CA, not the VMCA. If you followed Part 11 and are using the VMCA, skip this post.

Just like Part 11, I’ll go through the same process of using a Microsoft online CA, offline CA, and updating the certificates for both Windows and the VCSA. This should cover most scenarios that people have to deal with. If that’s not exactly what your scenario is, you can probably figure out what to do between VMware documentation and my Toolkit posts.

As always, download the latest version of my Toolkit script, as it is rapidly changing as I add more blog posts about SSL and work through issues. The download permalink is below. For this post you will need at least version 0.75 (April 2, 2015) or later to follow along.

Ironically, the VMware supplied certificate tool in it’s GA form has a bug when you replace the machine certificate with multiple intermediate CAs. You can find the KB here. So I’d recommend using my Toolkit script for a Windows PSC, as it does not have the bug and is easier anyway. ūüôā I am told VMware is working on an updated script, but I have no ETA on a release date. If you are using the VCSA you will need to use the workaround, which I cover in my post below.

Blog Series

vSphere 6.0 Install Pt. 1: Introduction
vSphere 6.0 Install Pt. 2: Platform Services Controller
vSphere 6.0 Install Pt. 3: Certificate Management
vSphere 6.0 Install Pt. 4: vCenter Upgrade Best Practices
vSphere 6.0 Install Pt. 5: ESXi Upgrade Best Practices
vSphere 6.0 Install Pt. 6: Install Windows PSC
vSphere 6.0 Install Pt. 7: Config SQL DBs
vSphere 6.0 Install Pt. 8: Toolkit Configuration
vSphere 6.0 Install Pt. 9: SSL Templates
vSphere 6.0 Install Pt. 10: Install VCSA PSC
vSphere 6.0 Install Pt. 11: VMCA as Subordinate
vSphere 6.0 Install Pt. 12: PSC Machine Certificate
vSphere 6.0 Install Pt. 13: Directory Services Certificate
vSphere 6.0 Install Pt. 14: Windows vCenter Install

Permalink to this series: vexpert.me/Derek60
Permalink to my Toolkit script: vexpert.me/toolkit60

Mint Machine Certificate (Online)

You should run the Toolkit script on your Windows external PSC, so you have all the files needed locally and it will also automate the installation. If you are using the VCSA PSC, then run this script from a Windows server that has PowerShell 3.0. Use this online procedure if your Microsoft CA will issue the subordinate certificate either with or without approval.

1. Run the Toolkit PowerShell script on your external PSC or a Windows server VM if using the VCSA PSC. Select the Machine SSL Certificate menu (option 4). Select the option to create a Machine SSL certificate with an online MS CA (option 1).

2. Enter the FQDN of the PSC, or press ENTER, if running from the PSC to accept the name. If no certificate approval is needed, the new machine certificate will be minted and downloaded.2015-04-02_8-02-13If your MS CA is configured to require CA manager approval before issuing a certificate, you will see the following:

2015-04-02_8-06-50

Have the CA manager approve the request ID, then re-run my Toolkit script and select the ‚ÄúResume a pending online request for¬†Machine SSL certificate‚ÄĚ (option 4). The script will show you the paths to the chained PEM file and the private key file.

2015-04-02_8-18-27

After the request is complete, you will see the following files in the C:\Certs\Machine directory.

2015-04-02_8-20-02

You have now minted your Machine SSL certificate, but it is not yet installed. Read on further in this post on how to install it.

Mint Machine Certificate (Offline)

Use this procedure if your issuing certificate authority is NOT a Microsoft online CA. It could be an offline Microsoft CA, or a non-MS CA as well.  This assumes you have the proper templates configured in your CA, per my Part 9 post.

1. Run the Toolkit script and in the Machine SSL menu (option 4) select the option to create a Machine SSL certificate with an offline or non-MS CA (option 2). The script will verify that you have downloaded the root chain certificates.

2. Because I was running this on the external PSC, I just pressed enter for the PSC name.

2015-04-02_8-53-19

3. Navigate to C:\Certs\Machine and upload the machine_ssl.csr file to your favorite CA and issue a certificate. Download the issued certificate in the base-64 format and save as new_machine.crt in the same folder. You MUST use this file name and it MUST be base-64 encoded. It should only contain the certificate, not a full chain.

4. Re-run the toolkit and from the menu select the option ‚ÄúCreate¬†Machine SSL PEM file from offline or non-Microsoft CA files‚ÄĚ (option 5). No input is needed. This will properly create a PEM file with the full certificate chain.

2015-04-02_9-10-18

Install Machine SSL Certificate (Windows PSC)

Note: For this procedure I am showing you how to use my Tookit script to install your Machine SSL certificate. VMware provides a Certificate Management tool that can perform the same steps. I show you how to use the VMware tool in the next section, when using the VCSA. The tool is the same on Windows and the VCSA. So if you feel more comfortable using the VMware tool to install the cert, skip down to that section. On Windows you can find the tool at C:\Program Files\VMware\vCenter Server\vmcad\Certificate-manager. My tool uses the manual method as documented in the vSphere 6.0 security guide, so the results are the same.

1. Re-run¬†my Toolkit script and in the Machine SSL¬†menu (option 4)¬†select the option ‚ÄúInstall custom machine SSL certificate on this computer‚ÄĚ (option 6).2015-04-02_9-13-45

2. Sit back and wait while the script stops services, installs the new certificate, and restarts the services. Keep an eye on the process, as mid way through you will need to confirm the deletion of the existing machine certificate. Simply press Y.

2015-04-02_9-29-56

Install Machine Certificate (VCSA PSC)

Note: For this procedure I am showing you how to use the VMware Certificate Manager tool to install the VMCA signing certificate. This assumes you used my Toolkit to generate the certificate files. There’s a bug documented in this VMware KB about the tool failing with multiple intermediate CAs. I’ll include the workaround here, so you have a one stop shop for replacing your certificates.

1. If you haven‚Äôt already enabled BASH on your VCSA let‚Äôs do that now. Open a console into the VCSA. Press F2 to customize the system. Login.¬†Arrow down to ‚ÄúTroubleshooting Mode Options‚ÄĚ then enable BASH shell. Exit the VCSA console.

2. Open a SSH session to the VCSA and type the following:

shell

chsh -s ‚Äú/bin/bash‚ÄĚ root

Make sure you run the ‚Äėchsh‚Äô command from the ‚Äėshell‚Äô prompt and not the VMware restricted shell‚Ķit won‚Äôt recognize the chsh command. Thanks to William Lam‚Äôs blog post here¬†for this step!

2. Download and install your favorite SCP client. I like WinSCP. Connect via SCP using the VCSA credentials.2015-04-01_14-40-003. Create a folder to put your SSL certificates. I like¬†the ‚Äė/root/ssl‚Äô directory.

4. In WinSCP navigate to the C:\Certs\machine folder. Upload the new_machine.cer and ssl_key.priv files to the SSL directory on the VCSA. The other files in the machine folder are not needed, so don’t upload them. From the C:\certs folder upload the chain.cer AND the root64.cer files to the /root/ssl directory on the VCSA. Note that all the options begin with a double dash, not a single dash. Cut/paste may mangle the dashes and cause the command to fail. Best to manually type the whole command instead of cut/paste.

5. SSH into the VCSA and ensure you get a ‚Äėshell‚Äô prompt. This will be in red, and have the short name of the VCSA. Type the following command.

Windows :

“C:\Program Files\VMware\vCenter¬†Server\vmafdd\dir-cli.exe” trustedcert publish –chain –cert¬†c:\certs\chain.cer

VCSA:

/usr/lib/vmware-vmafd/bin/dir-cli¬† trustedcert publish –chain –cert /root/ssl/chain.cer

6. In the VCSA shell run the following command:

/usr/lib/vmware-vmca/bin/certificate-manager

6. Choose Option 1 from the main menu. Enter the SSO password as requested.

7. From the new menu select Option 2, Import custom certificates. Input the root certificate file names when prompted. Use /root/ssl/new_machine.cer for the first prompt and /root/ssl/ssl_key.priv for the second. For the third and final prompt enter /root/ssl/root64.cer.

2015-04-02_14-42-00a

 

8. After you enter all the certificate paths you will be prompted to continue. The whole replacement process takes less than two minutes.

Inspecting the Machine Certificate

Now that we have installed a new machine SSL certificate, we want to make sure it was issued by our enterprise CA and is trusted. This can easily be done via any browser of your choosing.

1. Launch your favorite browser and go to https://PSC-FQDN/websso/. Open the certificate properties for the SSL site.

2015-04-01_19-07-392. Click on the Certification Path, and verify that all of your enterprise CAs are listed. If you only see a single entry in this list, and not the full chain, that likely means your Windows computer does NOT trust the full chain. See your CA administrator for getting all of your enterprise CAs published through Active Directory.

2015-04-02_15-25-05

Solution Warning

A reader pointed out the SRM and other solutions may fail when replacing the machine certificate on vCenter or the PSC. If you find yourself in this situation, check out this VMware KB article for remediation.

Summary

When you aren’t using the VMware VMCA, you must mint and install a machine SSL certificate for the PSC from your enterprise CA. This certificate is used for all reverse proxy services, such as those accessed by HTTP. You can elect to either use my Toolkit script to install the machine cert, or the VMware tool. Either way, you end up with a trusted machine SSL certificate on your PSC.

vSphere 6.0 Install Pt. 8: Toolkit Configuration

Now that we have the PSC installed, it’s time to configure the variables for the Toolkit script, and also make sure we can download our root certificates. Depending on your configuration, you may need to manually download your root public certificates. VMware needs certificates in a specific format, and they need the full certificate chain. So in this installment I show you all the variables in the Toolkit script that you will need to change to make it successful. In subsequent installments we will then use the Toolkit to setup the VMCA and other certificate options.

April 2, 2015 Update: Per VMware, VUM 6.0 can NOT use the vSphere¬†6.0 SSL template. So I’ve added a new variable called $VUMTemplate for the old 5.5 SSL template name. You can find instructions for creating the vSphere 5.5 template here.

Blog Series

vSphere 6.0 Install Pt. 1: Introduction
vSphere 6.0 Install Pt. 2: Platform Services Controller
vSphere 6.0 Install Pt. 3: Certificate Management
vSphere 6.0 Install Pt. 4: vCenter Upgrade Best Practices
vSphere 6.0 Install Pt. 5: ESXi Upgrade Best Practices
vSphere 6.0 Install Pt. 6: Install Windows PSC
vSphere 6.0 Install Pt. 7: Config SQL DBs
vSphere 6.0 Install Pt. 8: Toolkit Configuration
vSphere 6.0 Install Pt. 9: SSL Templates
vSphere 6.0 Install Pt. 10: Install VCSA PSC
vSphere 6.0 Install Pt. 11: VMCA as Subordinate
vSphere 6.0 Install Pt. 12: PSC Machine Certificate
vSphere 6.0 Install Pt. 13: Directory Services Certificate
vSphere 6.0 Install Pt. 14: Windows vCenter Install

Permalink to this series: vexpert.me/Derek60
Permalink to my Toolkit script: vexpert.me/toolkit60

Derek’s Toolkit Script

My Toolkit PowerShell script performs several tasks and is menu driven. It’s an all in one script, meaning it handles online/offline CAs, Windows CA and non-Windows CAs, and will also do other install tasks like create your ODBC and SQL database files connectors. New for vSphere 6.0 are automation steps for the VMCA and added support for three tier CA hierarchy (root and two subordinates).

My Toolkit script does NOT replace the VMware certificate replacement tools, it only augments them. So you would normally use the combination of my Toolkit script plus the VMware certificate management tools for full SSL certificate replacement. I did this specifically so that customers would be fully supported by VMware, even if they use my tool. I just make the process easier, I don’t do any behind the scenes hacking or unsupported commands.

I am still in the process of developing the script, so some of the vCenter SSL features are disabled in the initial versions until I work through the full process. But much of the script is functional in this initial version.

The script has the following features:

  • Downloads and installs the proper version of OpenSSL¬†if it‚Äôs not already installed
  • Creates 2048 bit RSA private keys in the proper format
  • Downloads both the root and up to two subordinate public certificates
  • Submits the CSRs to the online CA and downloads the certificates
  • Creates the needed service PEM files for the vCenter certificate tool
  • Creates the required root/subordinate PEM files
  • Does NOT require PowerCLI
  • Automatically uses the hostname of the server you run the script on for all certificates
  • Works with offline CAs
  • Creates customized SQL vCenter and VUM database creation script
  • Creates SQL ODBC DSNs for vCenter and VUM (SQL 2008 R2, 2012, 2014)
  • Automatically downloads and installs SQL 2008 R2¬†client package
  • Provides download URL for SQL 2012/2014 client
  • Support Microsoft CAs that require manual certificate approval
  • Requires PowerShell 3.0 or higher

Configure Toolkit Variables

1. Login to your external PSC and download my Toolkit script from here. You can run it from anywhere, but I think this is the optimal place for the first run.

2. My script will automatically download OpenSSL for you. Since OpenSSL versions change frequently, I put the download name up front for this version of the script. If you run the script and it errors out, it will display a friendly failure message. Just go to the URL shown, update the download filename and Voila! Unlike my vSphere 5.5 script, I won’t be releasing new versions every time OpenSSL is updated.

2015-03-31_7-11-21

 

 

3. Open the script in your favorite PowerShell editor and find the certificate details section. Modify the company name, organization, etc. for your environment.

2015-03-29_17-05-19

 

 

 

4. Modify the CA names as needed for your environment. My script now supports a root CA plus two subordinates. If you don’t have one or more subordinates, just add a # in front of the appropriate line.

2015-03-29_9-08-07

 

 

5.¬†If you are¬† using a Microsoft CA with the certificate web enrollment service enabled, then select whether you will be accessing the CA web site via HTTP or HTTPS. HTTPS is recommended, but sometimes there are certificate errors that don’t allow that to work.

2015-03-29_9-10-57

6. Next up you need to configure your Issuing CA information. This can be a little confusing, due to the way Microsoft labels the CA. The best way to find the proper name is login to your issuing CA, launch the Certificate Authority snap-in. This could be called anything, depending on how your CA was setup. Look for the name next to the green check mark. In the script prepend that name with the hostname of your CA.

2015-03-29_9-22-13

2015-03-29_9-18-57

7. For VUM 6.0 we need to use the vSphere 5.5 SSL template. So enter the name of your vSphere 5.5 SSL template here. If you followed my 5.5 guide, then it will be called VMware-SSL. Do NOT use your vSphere 6.0 template name here, as it will NOT work.

2015-04-02_10-40-57

8. Now you need to configure your VMware SSL template name. These certificates will be used for vCenter services and ESXi host certificates. The steps for vSphere 6.0 are NOT the same, so refer to my blog article here in Part 9 for the template instructions. This template names assumes you will follow that article. You can NOT use your vSphere 5.5 template.

2015-04-04_18-52-05

9. Next up, you need to define the Subordinate template name. VMware¬†requires using a custom template and not the Microsoft default. If you follow my blog post here, then your template name will be called vSphere6.0VMCA. If you won’t be using the VMCA subordinate CA feature, just ignore this section.

2015-03-30_11-38-47

If you have a custom template and need to know the “Template Name”, just open your CA MMC, go to “Certificate Templates”, right click and select “Manage”. Open the properties of the template in¬†question¬†and look for the “Template name” NOT the “Template display name”.

2015-03-30_11-36-39

 

 

 

 

 

10.¬†To download the proper certificate chain, my script must download the public certificates from each of the CAs that are in the chain. Depending on the age of your CA, you may need to increment up the “renewal” numbers to get the latest certificate. If you increment too high it will download garbage and my script will alert you to that fact. “0” is the default, but you may find you need 1 or more here.

2015-03-29_9-13-52

Configure Windows CA

11.¬†Next up we need to make sure your Windows CA can issue subordinate certificates if you will be using the VMCA as a subordinate CA. Ignore this section if you won’t be making the VMCA subordinate to your Windows CA. Go into your issuing CA, launch the Certificate Authority tool and look in the “Certificate Templates” folder. You should see a “vSphere 6.0 VMCA” template listed after you complete Part 9 of my guide. 2015-03-31_14-26-08

12.¬†If you do not see this listed then you haven’t read Part 9 (sorry I didn’t blog about this before, but it was a last minute lesson learned) and created the template. Go to that part now, create the new vSphere 6.0 templates, then come back here.

Download Root Certs

If all of your¬†CAs are serviced by an online¬†Microsoft CA and you have correctly configured the Toolkit script variables, and you have web services enabled on the CA, then the script will automatically download the public¬†certificates for you. However, if you have an offline CA or they aren’t web enrollment enabled, you will need to download them manually. Or if you are using a non-MS CA, then you need to get them manually as well. Sometimes the MS CA web services won’t cooperate so manual downloads are needed as well.

13.  Open a blank MMC, then add the Certificates snap-in for the Computer account.

14.¬†Navigate to the “Intermediate Certification Authorities” folder and open the Certificates folder. If you don’t see your CAs there, poke around in the other folders until you find them.

15. Find the certificate authorities for your environment. Right click on each one, and export as a base-64 encoded x.509 certificate. Save the root certificate as C:\Certs\root64.cer. Save the first subordinate certificate (if applicable) as C:\Certs\interm64.cer. If you have a second subordinate, save that certificate as C:\certs\interm264.cer.

2015-03-29_11-19-12

In case you are unsure of the base-64 certificate format, it will look like the following graphic if opened in a text editor.

2015-03-29_11-44-50

 Summary

If you are familiar with my Toolkit¬†script for vSphere 5.5, then you will be right at home in the 6.0 version. I’ve cleaned up the¬†configurable variables, added a few new ones, and added full VMCA support. We will use the Toolkit to¬†configure the remaining SSL certificates, which include vCenter and ESXi.¬†Next up is configuring the SSL template in Part 9.

vSphere 5.5 Toolkit Updated

This weekend I did a minor update to my VMware vSphere 5.5 SSL Toolkit script. It’s now at v1.59. I updated the OpenSSL download to use 0.9.8.zb, and also added a primitive PowerShell 3.0 check. PowerShell 3.0 and higher has always been required, but now I try and check for it. If you are running PS 3.0 and still get an error, then please leave a comment in this post. The logic isn’t all that intelligent, so may need tweaking.

If you aren’t familiar with my vSphere 5.5 toolkit script, then you can check out Part¬†8 of my 19 part vSphere 5.5 installation series. As always, you can download the latest version from vExpert.me/toolkit55.

Join the over 10,000 downloads of my Toolkit script and make your SSL life a lot easier.

VMware vSphere 5.5 Toolkit v1.58 Live

As many of you know, one of my passions throughout my IT career has been security. Having worked in the Federal Government space for most of my career, making sure solutions are secure is always a top priority. Securing your VMware infrastructure is very important, and one of the primary tasks is using trusted SSL certificates. So last year I wrote the vSphere 5.5 Toolkit PowerShell script, which has had over 9,000 downloads! I¬†had no idea¬†it would be so popular. Here’s a screenshot of the main menu:

vsphere 5.5 toolkit

Features of the SSL toolkit script include:

  • Downloads and installs the proper version of OpenSSL (0.9.8.za) if it‚Äôs not already installed
  • Creates 2048 bit RSA private keys in the proper format
  • Creates a directory for each service bundle of SSL certificates
  • Generates seven OpenSSL configuration files, one for each certificate, in the appropriate directory
  • Downloads both root and subordinate root public certificates
  • Submits the CSRs to the online CA and downloads the certificates
  • Creates the needed service PEM files for the vCenter certificate automation tool
  • Creates the required root/subordinate PEM files
  • Handles the¬†special¬†SSO 5.5 certificate requirements
  • Does NOT require PowerCLI
  • Assumes all vCenter components are on one server
  • Automatically uses the hostname of the server you run the script on for all certificates
  • Creates a pre-filled vCenter Certificate Automation environment script ‚Äď Just run!
  • Works with offline CAs
  • Creates SSO 5.5 certificate replacement files ‚Äď Only used if manual replacing certs
  • Creates customized SQL vCenter and VUM database creation script
  • Creates SQL ODBC DSNs for vCenter and VUM
  • Automatically downloads and installs SQL 2008 R2 or SQL 2012 client package
  • Linux vCenter Server Appliance support for online minting¬†and offline CSR creation
  • Creates certificates for Auto Deploy, Dump Collector, Syslog collector, Authentication Proxy
  • Support Microsoft CAs that require manual certificate approval

I’ve now updated the script with some minor modifications for v1.58, dated July 12, 2014:

  • Updated OpenSSL download to 0.9.8za
  • Removed SQL 2012 SP1 client download (link broken)
  • Fixed Database creation script bug
  • Added additional error handling and Powershell-ized more commands
  • Changed the sts.properties file to use sts in the URI per KB2058519

These are incremental updates, and the base functionality has remained the same. I am hoping for vSphere v.Next that VMware will streamline the whole process and give SSL replacement a makeover. I have no idea if this is in the works or not.

As always, you can download the latest version of the toolkit script from: vExpert.me/toolkit55 If you are using an older version I suggest you grab the latest copy. If you want full SSL lifecycle management and a paid solution, I recommend you check out the VSS Labs vCert Manager, which you can find out about here.

Also remember to check out my 20 part vSphere 5.5 series, which covers the usage of the toolkit script and a whole lot more. You can find that series at: vExpert.me/Derek55

vSphere 5.5 Toolkit v1.57 Released

Now that vSphere 5.5 has been out for a few months, if you haven’t already started working with it in your test environment, you should! A lot of great new features, and major improvements to the SSO experience. However, you still may find it a bit challenging to secure vCenter and ESXi hosts with trusted SSL certificates. So hot off the press is a minor bug fix version of my vSphere 5.5 Toolkit script. You can download the new version here. In case you aren’t familiar with it, here are some of the features:

  • Downloads and installs the proper version of OpenSSL (0.9.8.Y) if it‚Äôs not already installed
  • Creates 2048 bit RSA private keys in the proper format
  • Creates a directory for each service bundle of SSL certificates
  • Generates seven OpenSSL configuration files, one for each certificate, in the appropriate directory
  • Downloads both root and subordinate root public certificates
  • Submits the CSRs to the online CA and downloads the certificates
  • Creates the needed service PEM files for the vCenter certificate automation tool
  • Creates the required root/subordinate PEM files
  • Handles the¬†special¬†SSO 5.5 certificate requirements
  • Automatically uses the hostname of the server you run the script on for all certificates
  • Creates a pre-filled vCenter Certificate Automation environment script ‚Äď Just run!
  • Works with offline CAs
  • Creates SSO 5.5 certificate replacement files ‚Äď Only used if manual replacing certs
  • Creates customized SQL vCenter and VUM database creation script
  • Creates SQL ODBC DSNs for vCenter and VUM
  • Automatically downloads and installs SQL 2008 R2 or SQL 2012 client package
  • Linux vCenter Server Appliance support for online minting¬†and offline CSR creation
  • Creates certificates for Auto Deploy, Dump Collector, Syslog collector, Authentication Proxy
  • Support Microsoft CAs that require manual certificate approval

Version 1.57, which you can download here, has a couple of minor fixes:

  • More robust handling of non-internet connected servers
  • Removed line continuation characters that caused issues for some people
  • Fixed bug when no subordinate CA was present (v1.56)
  • Changed Microsoft “renewal” default to 0 for root/subordinate (v1.56)

Although the script was developed for all-in-one vCenter servers, it will work for instances where services are distributed across several servers. You will just have to be intelligent about using the correct hostnames and merging together directories with the proper certificates. Not rocket science, but does take a little manual work.

In case you missed it, I also have a 19 part vCenter 5.5 installation and configuration series that covers how to use the Toolkit script in gory detail. You can check out that series here. The ESXi SSL portion of the tool also works with vSphere 5.0 and 5.1, so you aren’t just limited to 5.5 hosts.

1-11-2014 2-27-28 PM

vSphere 5.5 Toolkit v1.55 Released

Yes, time to update my vSphere 5.5 Toolkit with a few more features and bug fixes. For those of you that need to replace your vSphere 5.5 SSL certificates, the process can be somewhat cumbersome and time consuming. While VMware has a tool to help you replace the certificates once you create them (SSL certificate automation tool), it has limited functionality in helping you create all the files needed as pre-reqs to running the tool.

Since my vSphere 5.1 installation series was so popular, for vSphere 5.5 I wanted kick it up a few notches. So I wrote the vSphere 5.5 Toolkit script that has a number of features to ease your SSL pain. For a complete list of features, click here. To date it has had over 3,200 downloads. Now live is a minor update, for your deployment pleasure. v1.55 of my Toolkit script is now available for download here.

Derek Seaman vCenter 5.5 Toolkit

What’s new since v1.50?

Root Certificate Validation (New)

This version¬†addresses an issue where sometimes the automatic download of a root or subordinate CA¬†certificate would result in HTML code and not a Base64 certificate. The root cause of this issue is how¬†Microsoft¬†implemented the certificate download feature.¬†Because the¬†root certificates can be renewed, there’s a counter called “renewal” in the download URL to specify which certificate to download.

My script does not have logic to download all certificates and pick out the newest one (maybe in future versions).¬†But what it will do is validate the file contents to ensure a certain string is present which indicates the file contains a Base64 encoded certificate. If the file is invalid an error will appear and the script¬†halts. If that happens,¬†search for “renewal” in the script (two locations) and decrement the number to 0. If it downloads an old certificate that expired, increment the number up by one until it gets the most recent version.

The script also checks manually downloaded base64.cer and interm64.cer certificate files for the same string, to validate they are Base64 encoded. It’s easy to use the wrong file type, which will greatly confuse the VMware certificate replacement tool. All of your certificate files should look like the example below, with —–BEGIN CERTIFICATE—–.

1-11-2014 2-25-42 PM

If your certificates are invalid, then you will get a red warning as shown below.

1-11-2014 2-46-54 PM

Certificate Request Changes (New)

VMware notified me that an upcoming change to a KB article was in the works. According to VMware the Web Client certificate needs the IP address in the SAN field with both DNS and IP extensions (e.g. DNS:10.10.10.10, IP:10.10.10.10). Apparently this is for maximum cross-browser compatibility across IE, Chrome and Firefox. For simplicity all certificate requests have both extensions in this version. If you don’t have any web client issues due to using an IP address vice the FQDN, then you don’t need to re-issue the web client certificate. If you do have issues, then this is probably the reason. You only need to update the web client certificate, not the 250 other vCenter certificates.

ESXi Host Support (If you missed it)

While not new to v1.55, version v1.50 released on December 22, 2013¬†added fairly robust ESXi host support. I didn’t blog about that version, so some of you may not be aware of it. I did Tweet, so make sure you follow me on Twitter for more timely news. You can manually enter several ESXi hosts to replace the certificates on, or give it an input file of hostnames. SSH is NOT required (uses HTTPS), and should be backwards compatible with vSphere 4.x and later although I have not personally tested it. This supports an Online Microsoft CA, offline CA, or third-party CA.

Summary

Given the positive feedback on the tool, it appears to be doing what I intended: Simplify the vCenter 5.5 installation process and make security easier. If you experience any problems or bugs, please leave a comment. I can’t promise to fix everything, but I’ll try to fit it into my schedule. Again, you can download the latest version from here.

vCenter 5.5 SSL Certificate and SQL Toolkit Updated

11-17-2013 7-03-32 PMFresh off the press is an updated version of my vCenter 5.5 SSL certificate Toolkit script. Last year when I did my popular vCenter 5.1 install series the posts contain a series of scripts and CLI commands to replace the SSL certificates. While that process worked for many people, it still was not as easy as it should be.

So for vCenter 5.5 I wrote a PowerShell script that did all the SSL certificate creation ‘magic’ in one place. In the intervening weeks since the first version went up, I’ve made a number of changes based on user feedback (and code submission) and my own development effort.¬†I¬†want to develop it further, but that will have to wait for a number of weeks while I complete a big project I’m working on. But¬†for those that did download the¬†first version and haven’t seen my Tweets about updates, I¬†wanted¬†a dedicated post to highlight¬†the full feature set of v1.41 (November 10th).

The script is designed to be used in conjunction with the VMware vCenter certificate automation tool, NOT replace it. While that tool will create CSRs, I find it a bit cumbersome and does not help you in minting the certs. Regardless of what kind of CA you have, the script will help. The degree of automation varies, as the script is targeted for an online Microsoft CA. Once you use my tool to mint all of your certificates, then it’s a straight forward matter of using the VMware certificate tool to replace the self-signed certificates with your freshly minted ones.

As you will see in the feature list, the script goes beyond just SSL assistance and can also aid in your SQL database and DSN creation.

The script has the following features:

  • Downloads and installs the proper version of OpenSSL (0.9.8.Y) if it‚Äôs not already installed
  • Creates 2048 bit RSA private keys in the proper format
  • Creates a directory for each service bundle of SSL certificates
  • Generates¬†ten OpenSSL configuration files, one for each certificate, in the appropriate directory
  • Creates certificates for AutoDeploy, Dump Collector and Syslog collector
  • Downloads both root and subordinate root public certificates
  • Submits the CSRs to the online CA and downloads the certificates
  • Creates the needed service PEM files for the vCenter certificate automation tool
  • Creates the required root/subordinate PEM files
  • Handles the¬†special¬†SSO 5.5 certificate requirements
  • Assumes all vCenter components are on one server
  • Automatically uses the hostname of the server you run the script on for all certificates
  • Creates a pre-filled vCenter Certificate Automation environment script ‚Äď Just run!
  • Works with offline CAs
  • Creates SSO 5.5 certificate replacement files ‚Äď Only used if manual replacing certs
  • Creates customized SQL vCenter and VUM database creation script
  • Creates SQL ODBC DSNs for vCenter and VUM
  • Automatically downloads and installs SQL 2008 R2 or SQL 2012 client package
  • Linux vCenter Server Appliance support for online minting¬†and offline CSR creation
  • Creates certificates for Auto Deploy, Dump Collector and Syslog collector
  • Support Microsoft CAs that require manual certificate approval

On the potential roadmap is replacing the ESXi 5.x host certificates, and a bit more robust Linux VCSA support. A screenshot of the main menu is shown below.

As always you can download the latest version from: vExpert.me/toolkit55¬†It’s gotten over 1,500 downloads in the few weeks that its been available, which is great. Hopefully it is helping people install vCenter 5.5 and more easily configure trusted certificates. For instructions on how to use the tool and a change log, start in Part 8 of my vCenter 5.5 install series.

11-10-2013 5-29-56 PM

vSphere 5.5 Install Pt. 9: Offline SSL Minting

10-4-2013 6-19-17 PMNot everyone has an online Microsoft Certificate Authority, or maybe my toolkit script has issues in your environment. So in this installment we will go over manual SSL minting. By that I mean we will use my Toolkit script to create the CSRs, you will download the certificates yourself, then run my Toolkit script again to create all of the required files. So in reality the only manual process is getting the certificate.

Even if you don’t have an online Microsoft CA, I suggest reading through Part 8. It will familiarize you with my vCenter 5.5 Toolkit script¬†and has the change log. If have an online Microsoft CA and ran the script in the previous post¬†then you can skip this installment and go to Part 10 (coming soon).

Blog Series

SQL 2012 AlwaysOn Failover Cluster for vCenter
vSphere 5.5 Install Pt. 1: Introduction 
vSphere 5.5 Install Pt. 2: SSO 5.5 Reborn 

vSphere 5.5 Install Pt. 3: vCenter Upgrade Best Practices and Tips
vSphere 5.5 Install Pt. 4: ESXi 5.5 Upgrade Best Practices and Tips 
vSphere 5.5 Install Pt. 5: SSL Deep Dive
vSphere 5.5 Install Pt. 6: SSL Certificate Template
vSphere 5.5 Install Pt. 7: Install SSO
vSphere 5.5 Install Pt. 8: Online SSL Minting
vSphere 5.5 Install Pt. 9: Offline SSL Minting
vSphere 5.5 Install Pt. 10: Replace SSO Certificates
vSphere 5.5 Install Pt. 11: Install Web Client 
vSphere 5.5 Install Pt. 12: Configure SSO
vSphere 5.5 Install Pt. 13: Install Inventory Service
vSphere 5.5 Install Pt. 14: Create Databases
vSphere 5.5 Install Pt. 15: Install vCenter
vSphere 5.5 Install Pt. 16: vCenter SSL
vSphere 5.5 Install Pt. 17: Install VUM
vSphere 5.5 Install Pt. 18: VUM SSL
vSphere 5.5 Install Pt. 19: ESXi SSL Certificate

Permalink to this series: vexpert.me/Derek55
Permalink to the Toolkit script: vexpert.me/toolkit55

Offline SSL Method

1. Download my vCenter 5.5 toolkit script from the link above. Open it in the PowerShell ISE (or favorite editor). The PowerShell script requires a few variable modifications before you run it. In the first block of variables you need to setup the directory where you want all of the certificates to go. If OpenSSL is already installed, change the path so the script knows where the root directory is. If that directory does not exist OpenSSL will be downloaded and installed for you. Next up are the certificate properties. Change those to suite your environment. If you want the server’s IP address in the SAN field, then uncomment the line and change the IP.

10-10-2013 7-04-44 PM

2. Execute the PowerShell Toolkit script. Unlike part 8 where we selected option 1 and everything was automated, here we need to select the option behind door number 2. This will create all of the required directories, private RSA keys and CSRs for you.

10-9-2013 4-52-21 PM

2. The first screenshot are the seven service directories which get automatically created. Inside each directory are three files. In the second screenshot the rui.key file is your private 2048 bit RSA key. The .cfg file is the OpenSSL configuration file that was used to generate the CSR. The .csr file is what you will submit to your CA.

10-4-2013 6-51-01 PM

10-4-2013 6-44-06 PM

3. Now you need to take¬†each of the seven¬†CSR files and submit it to your CA. In case you have an offline Microsoft CA or there are strong security measures in place so the vCenter can’t access your CA directly, I’ll cover the manual issuing and downloading process with a Microsoft Windows Server¬†2012 CA. If you have a non-Microsoft CA, then just skim over the Microsoft CA section, save your¬†certificates as rui.crt in each directory, and pick back up at step 8.

4. Go to the URL of your Microsoft CA. The default address is https://hostname/certsrv. Make sure you are accessing the CA page with credentials that can request VMware-SSL certificates. Click on Request a certificate.

10-4-2013 7-00-57 PM

5. Select the second option, Submit a certificate request by using a base-64-encoded….

10-4-2013 7-03-33 PM

6. Copy and paste the CSR information from the first service into the top pane. Make sure the VMware-SSL template is selected. If that template is NOT listed then you probably goofed up one of three things 1) You¬†accessing the CA web site with your non-admin¬†account¬†2) You didn’t properly publish the VMware-SSL certificate template 3) You don’t have enroll permissions on the VMware-SSL template. Do not enter any additional attributes.

10-4-2013 7-05-27 PM

7. After you submit the certificate request you need to download the Base-64 encoded version WITHOUT the certificate chain. Name the file rui.crt and save it back into the same service directory that you submitted the CSR from. These certificates are NOT interchangeable, so don’t get the rui.crt files mixed up. The system will barf later on and you will lose some hair. Each certificate must match the service it was intended for.

10-4-2013 7-09-29 PM

8. After you’ve done this for all seven certificates, each service directory should now look like the following, with a rui.crt file now present.

10-4-2013 7-15-23 PM

9. Next up we need to create one or two root CA files, depending on your CA architecture. Double click on one of your .crt files and go to the Certification Path tab. In my example below we have two CAs: A root and a subordinate. The CA at the top is the root and the next one down is the subordinate. vCenter needs the public certificate from both, so that it can properly chain.

10-4-2013 7-17-20 PM

10. If you are using a Microsoft CA then go back to the Home page of the CA. But this time select the last option, Download a CA certificate…

10-4-2013 7-22-02 PM

11. Click on Download CA certificate chain if you have a Root/subordinate CA architecture. If you have just a root CA click on Download CA Certificate. If you are downloading the chain, just save it to your desktop with any ole name and skip to step 12. If you have just a root CA, then save the file as Root64.cer in the root of your certificate directory (screenshot below).

10-4-2013 7-23-33 PM

Root only CA:

10-4-2013 7-40-48 PM

12. For those that downloaded their chain (and ball), double click the certificate and locate the two certificates. Right click on your ROOT (see step 9), select All Tasks, and Export. Save the certificate as a Base-64 encoded file and name it Root64.cer. Put it in the root of your certificate directory as show in step 11.

10-4-2013 7-37-28 PM

13. Repeat the process on the subordinate CA, but save the file as interm64.cer. You should now have a directory that looks like:

10-4-2013 7-47-04 PM

13.  Re-run the Toolkit script but now we select Option 3. This will process all of the files and create the exact same output as the online option in Part 8. Review the screen events for any errors.

10-10-2013 7-31-55 PM

A sample of the screen output is below.

10-10-2013 8-12-40 PM

Output Validation

1. Assuming no errors occur, you should now see additional files in the root of your certificate directory. A chain.cer file should now appear if you have an intermediate CA. A hash file (which ends in 0) for each root certificate will also be listed.  If you only have a root CA then you will have one hash file.

10-9-2013 5-05-54 PM

2. If you take a peek inside one of the folders you will see a series of files. Each service, except SSO, will have the same set of files (except the .csr and .cfg with are uniquely named). The

  • chain.pem:¬†Used for the VMware vCenter certificate automation tool
  • rui.crt: Public half¬†of your SSL certificate
  • rui.key: Private half of your SSL certificate
  • rui.pfx: Combined private and public SSL keys
  • *.cfg: ¬†Certificate signing request file
  • *.csr: Certificate signing request

10-9-2013 5-09-43 PM

3. In the vCenterSSO you will see a plethora of files. Depending on how you replace your SSL certificates, you may only use some of these files. But to help you out as much as possible, all the SSO files that are tedious to create manually are created for you. If you are missing files, then something went wrong. Please match up all filenames to validate the toolkit script worked. Some files are copies of each other, but they are needed to avoid confusion and more easily follow the KBs.

  • *.properties: Use for manual SSO SSL replacement
  • *_id: Use for manual SSO SSL replacement
  • ca_certificates.crt: Use for manual SSO SSL replacement
  • root-trust.jks: Used for SSO/STS certificate validation
  • server-identity.jks: Same file as above with a different name (per VMware KBs)
  • ssoserver.p12: Same functionality as rui.pfx, but VMware changed the name and format for SSO 5.5
  • ssoserver.crt:¬†Copy of chain.pem
  • ssoserver.key:¬†Copy of rui.key

10-9-2013 10-06-14 PM

Certificate Validation

Now that your certificates are minted, let‚Äôs quickly validate all of the properties are present. Even if your CSR requests a property, that doesn‚Äôt mean your CA will honor it. The OU in each subject name should be unique and match the directory it’s in.

10-10-2013 7-17-04 PM

The Subject Alternative Name should contain the short name and FQDN. Optionally it can contain your IP address too.

10-10-2013 7-18-18 PM

Enhanced key usage should show server and client authentication. Client authentication can be missing if the CA template is wrong.

10-10-2013 7-18-59 PM

Key usage should contain digital signature, key encipherment and data encipherment.

10-10-2013 7-19-43 PM

Summary

After a bit more work than the automated method, you now have all of the required certificate files to either use the vCenter certificate automation tool, or try the complex manual replacement method. Next up in Part 10 we update the SSO service SSL certificates.