vSphere 5.5 Install Pt. 10: Replace SSO Certs

10-11-2013 6-44-21 PMJust like replacing a hip joint, replacing vCenter SSO SSL certificates can induce some pain, is a bit complex, and the outcome can be questionable. The replacement process in SSO 5.5 is pretty much like that in 5.1, but now we have the VMware certificate automation tool and my vCenter toolkit to make this a safer operation. Outcome is not guaranteed, and there may be side effects.

My approach to replacing SSL certificates in this series is replacement right after the service is installed. Basically you build up a trusted infrastructure from the get-go. VMware would likely say to build it up all untrusted, then go back and replace the certs. Certainly a valid point. If the VMware automation tool was more automated, then I might agree. But by building up trusted layers there are less replacement steps and chance of messing up, IMHO.

I do strongly recommend using the VMware certificate automation tool instead of manual replacement steps. But for the SSO service I’ll show you both ways, each using my vCenter Toolkit script to prepare the needed files.

Blog Series

SQL 2012 AlwaysOn Failover Cluster for vCenter
vSphere 5.5 Install Pt. 1: Introduction 
vSphere 5.5 Install Pt. 2: SSO 5.5 Reborn 

vSphere 5.5 Install Pt. 3: vCenter Upgrade Best Practices and Tips
vSphere 5.5 Install Pt. 4: ESXi 5.5 Upgrade Best Practices and Tips 
vSphere 5.5 Install Pt. 5: SSL Deep Dive
vSphere 5.5 Install Pt. 6: SSL Certificate Template
vSphere 5.5 Install Pt. 7: Install SSO
vSphere 5.5 Install Pt. 8: Online SSL Minting
vSphere 5.5 Install Pt. 9: Offline SSL Minting
vSphere 5.5 Install Pt. 10: Replace SSO Certs
vSphere 5.5 Install Pt. 11: Install web client
vSphere 5.5 Install Pt. 12: Configure SSO
vSphere 5.5 Install Pt. 13: Install Inventory Service
vSphere 5.5 Install Pt. 14: Create Databases
vSphere 5.5 Install Pt. 15: Install vCenter
vSphere 5.5 Install Pt. 16: vCenter SSL
vSphere 5.5 Install Pt. 17: Install VUM
vSphere 5.5 Install Pt. 18: VUM SSL
vSphere 5.5 Install Pt. 19: ESXi SSL Certificate

Permalink to this series: vexpert.me/Derek55
Permalink to the Toolkit script: vexpert.me/toolkit55

Automated Certificate Replacement

1. Download the vCenter Certificate Automation Tool v5.5. You can find it under the Drivers and Tools section of the vSphere 5.5 downloads. Direct link is here. Unzip the contents on your vCenter server.

10-9-2013 6-10-08 PM

2. At the beginning of my toolkit script are two variables that set the SSO and vCenter administrator account names. Now during the SSO install you can’t change these names, so you shouldn’t need to change them here. But just FYI, I wanted to point out they are defined in the toolkit script.

10-9-2013 6-23-42 PM

3. Re-run my vCenter 5.5 Toolkit script, but this time we want to create the automation batch file. The menu number may change, but in today’s version of the script it is option 4.

10-9-2013 6-17-27 PM

4. The script will run in a split second and provide you the path to the batch file. The batch file sets the same variables as the stock VMware ssl-environment.bat file. I stripped out all of the comments and set the paths to where your certificate files are stored, assuming you used my script for part 8 or 9, minting your SSL certs.

5. Copy the ssl-environment.bat file that the toolkit created and overwrite the one in the VMware tool directory. You don’t need to run the batch file as the main updater script will execute it in the background.

10-9-2013 7-14-01 PM

6. Run the VMware ssl-updater.bat file. On the main menu select Option 3.

10-9-2013 6-27-30 PM

7. On the next menu select option 1.

10-9-2013 6-29-58 PM

8. When you select option 1 it will ask you a series of questions (in yellow below). Most of the information should be pre-populated for you. But you do need to input the administrator@vsphere.local password and answer if you are using a load balancer or not (don’t use one, VMware does not recommend it). Cross your fingers and toes, and watch for success messages.

10-9-2013 6-53-45 PM

9. Skip down to the verification section to validate that your certificate was replaced and that the service is in a healthy state. Let’s hope for no side-effects of this delicate operation.

Manual Replacement

If you for some reason you can’t use the VMware certificate tool (I recommend you DO use it, since it provides some certificate checking and is less error prone. But it could have issues that prevent you from using it. If so, then you can follow the steps below. They are directly from KB 2058519, with a couple of corrections (I’ve submitted fixes, so hopefully they will correct them).

Thanks to my vCenter 5.5 toolkit script 90% of the tedious work in that KB article is already done for you. Whoohoo. All we need to do is issue three commands to update the lookup service, then copy over our new certs. That’s it!

1. Open an elevated command prompt (not PowerShell) and enter the following command, replacing the vCenter FQDN, paths, and password as needed.

"C:\Program Files\VMware\Infrastructure\VMware\cis\vmware-sso\ssolscli" updateService -d https://d001vctr01.contoso.net:7444/lookupservice/sdk -u administrator@vsphere.local -p YourPassword -si D:\certs\vCenterSSO\gc_id -ip d:\certs\vCenterSSO\gc.properties

2. Enter the following command:

"C:\Program Files\VMware\Infrastructure\VMware\cis\vmware-sso\ssolscli" updateService -d https://d001vctr01.contoso.net:7444/lookupservice/sdk -u administrator@vsphere.local -p YourPassword -si D:\certs\vCenterSSO\admin_id -ip d:\certs\vCenterSSO\admin.properties

3. Enter the following command:

"C:\Program Files\VMware\Infrastructure\VMware\cis\vmware-sso\ssolscli" updateService -d https://d001vctr01.contoso.net:7444/lookupservice/sdk -u administrator@vsphere.local -p YourPassword -si D:\certs\vCenterSSO\sts_id -ip d:\certs\vCenterSSO\sts.properties

4. If all goes well then your screen should look similar to the one below, with three success messages. If not, you did something wrong. Depending on what’s goofed up, SSO may be in a hosed state and require a re-install.

10-9-2013 8-16-46 PMA

5. Now that the services have been updated, we need to overwrite some certificates. Navigate to the C:\ProgramData\VMware\CIS\runtime\VMwareSTS\conf directory. Backup the ssoserver.crt, ssoserver.key and ssoserver.p12 files. In the vCenterSSO directory that the toolkit script created, copy the ssoserver.crt, ssoserver.key and ssoserver.p12 files and overwrite the old versions.

6. In an elevated command prompt type:

net stop vmwarests
net start vmwarests

Verification

To verify that the SSO service is using the new certificate and didn’t suffer fatal stab wounds, open your favorite browser and go to the lookup service URL. That should be https://vCenterName:7444/lookupservice/sdk It should open without any SSL errors and if you look at the certificate by clicking on the lock icon, it will be issued by your CA. The XML response below is normal (yes even the Unexpected EOF), since we didn’t give the SDK service any input data.

10-9-2013 7-02-19 PM

10-9-2013 7-05-34 PM

Summary

With the help of my Toolkit script and the VMware automation tool, replacing the SSO certificate is not as painful as it was in the early months of vCenter 5.1. Even if you need to replace it manually, the toolkit does a lot of the tedious work to make success more likely. Next up in Part 11 is installing the Web Client, updating the SSL certificates, and configuring IE 10.

Comments

  1. Great series as always Derek.

    I only had two issues, none of them a result of your coding. The first was that my root CA is offline, so I had to comment out the lines in your script and drop the certificate (root64.cer) in myself.

    The second is that I had to manually enter in the FQDN of my VC box, as the script took the domain name of the logged in the user – but for my setup the CA user is in the parent domain, and the SSO box is in the child domain.

    All good now and up and running!

    • Thanks for the feedback. The newer versions of the script let you pre-download the root/sub CA files and place them in the directory. If they are found then downloading will be skipped. So that should resolve your offline root issue.

      The VC FQDN issue is interesting and a scenario I had not thought of. I'll see if I can accomodate that scenario in a future version.

  2. Excuse my ignorance of certificates but I see you mention the SSO and vCenter administrator account names $sso_admin and $vc_admin set to administrator@vsphere.local and that I should not have to modify these. Later you mention that I have to enter the password for this. Are both user id's usint the password that is set when you install sso? The SSO is obvious but is the vc_admin something that was around before in vsphere 5.0. I ask b/c I may have to upgrade my current vc environment and I can not find any reference to a password for a local administrator account.

  3. Derek great Blog as always.
    I ran into an issue installing the web client after replacing the SSO certificates. On the screen where it asks for your administrator password it errors sayin it cant contact the lookup service. I rolled back the certificates to the one installed with SSO and then was able to install the web client.
    I think given this issue I will install all components before upgrading the ceretificates.

  4. Johan de Stigter says:

    Hi Derek,

    Thanks for all of this, much appreciated. However something goes wrong on my side I guess. I get the error message 'The certificate chain file does not contain a valid certification path'. I checked the chain.pem file and it looks OK. When I go for the manual update is goes OK and even the verification is OK then. However when I then start with the install of the webclient, I get a error that it can't connect. I checked the log files and in there ther is an error regarding the certificates as well. I followed all the steps (tried 6 times and checked everything 3 times now) but is starting to drive me insane :)

    Mant thanks

  5. Johan de Stigter says:

    Hi Derek,

    I resolved the issue finally :)
    The issue turned out to be in the CA. In our CA for some reason we have 2 CA certificates (on old and one new, but both active for some reason) Your script downloads the old in my case (I gues due to the name) but due to some reason the vCenter server does not accept that certificate, even if it is in the trusted root.
    In your script I found the line that if a certificate was already in the Cert directory, then it would skip the download, so I downloaded and exported the correct CA certificate and let the script do the rest. Everything works perfectly now :)
    So if others are also having trouble with the script (with an error about the chain.pem file) and after that having trouble installing the Web Client, then check your CA Certificates on the CA.

  6. Hi Derek,

    great Job, thx!

    I had the same issue than Johan: 'The certificate chain file does not contain a valid certification path'. The Problem was that we have renewed the root certificate, because the old one expired. Than you have to use this URL in Line 190:

    $url = "$CADownload"+"://$rootCA/certsrv/certnew.cer?ReqID=CACert&Renewal=1&Enc=b64"

    Just increment the number after 'Renewal'. Hope it helps!

    Greets
    Michael

    • I've updated the v1.42 script to include Renewal=1. It worked just fine for me, even on a CA that has not renewed. If that trips up people, I'll revert back to =0 and put notes in the readme about changing it to 1 if needed.

Speak Your Mind

*

© 2014 - Sitemap