Just like replacing a hip joint, replacing vCenter SSO SSL certificates can induce some pain, is a bit complex, and the outcome can be questionable. The replacement process in SSO 5.5 is pretty much like that in 5.1, but now we have the VMware certificate automation tool and my vCenter toolkit to make this a safer operation. Outcome is not guaranteed, and there may be side effects.
My approach to replacing SSL certificates in this series is replacement right after the service is installed. Basically you build up a trusted infrastructure from the get-go. VMware would likely say to build it up all untrusted, then go back and replace the certs. Certainly a valid point. If the VMware automation tool was more automated, then I might agree. But by building up trusted layers there are less replacement steps and chance of messing up, IMHO.
I do strongly recommend using the VMware certificate automation tool instead of manual replacement steps. But for the SSO service I’ll show you both ways, each using my vCenter Toolkit script to prepare the needed files.
SQL 2012 AlwaysOn Failover Cluster for vCenter
vSphere 5.5 Install Pt. 1: Introduction
vSphere 5.5 Install Pt. 2: SSO 5.5 Reborn
vSphere 5.5 Install Pt. 3: vCenter Upgrade Best Practices and Tips
vSphere 5.5 Install Pt. 4: ESXi 5.5 Upgrade Best Practices and Tips
vSphere 5.5 Install Pt. 5: SSL Deep Dive
vSphere 5.5 Install Pt. 6: SSL Certificate Template
vSphere 5.5 Install Pt. 7: Install SSO
vSphere 5.5 Install Pt. 8: Online SSL Minting
vSphere 5.5 Install Pt. 9: Offline SSL Minting
vSphere 5.5 Install Pt. 10: Replace SSO Certs
vSphere 5.5 Install Pt. 11: Install web client
vSphere 5.5 Install Pt. 12: Configure SSO
vSphere 5.5 Install Pt. 13: Install Inventory Service
vSphere 5.5 Install Pt. 14: Create Databases
vSphere 5.5 Install Pt. 15: Install vCenter
vSphere 5.5 Install Pt. 16: vCenter SSL
vSphere 5.5 Install Pt. 17: Install VUM
vSphere 5.5 Install Pt. 18: VUM SSL
vSphere 5.5 Install Pt. 19: ESXi SSL Certificate
Permalink to this series: vexpert.me/Derek55
Permalink to the Toolkit script: vexpert.me/toolkit55
Automated Certificate Replacement
1. Download the vCenter Certificate Automation Tool v5.5. You can find it under the Drivers and Tools section of the vSphere 5.5 downloads. Direct link is here. Unzip the contents on your vCenter server.
2. At the beginning of my toolkit script are two variables that set the SSO and vCenter administrator account names. Now during the SSO install you can’t change these names, so you shouldn’t need to change them here. But just FYI, I wanted to point out they are defined in the toolkit script.
3. Re-run my vCenter 5.5 Toolkit script, but this time we want to create the automation batch file. The menu number may change, but in today’s version of the script it is option 4.
4. The script will run in a split second and provide you the path to the batch file. The batch file sets the same variables as the stock VMware ssl-environment.bat file. I stripped out all of the comments and set the paths to where your certificate files are stored, assuming you used my script for part 8 or 9, minting your SSL certs.
5. Copy the ssl-environment.bat file that the toolkit created and overwrite the one in the VMware tool directory. You don’t need to run the batch file as the main updater script will execute it in the background.
6. Run the VMware ssl-updater.bat file. On the main menu select Option 3.
7. On the next menu select option 1.
8. When you select option 1 it will ask you a series of questions (in yellow below). Most of the information should be pre-populated for you. But you do need to input the ad***********@vs*****.local password and answer if you are using a load balancer or not (don’t use one, VMware does not recommend it). Cross your fingers and toes, and watch for success messages.
9. Skip down to the verification section to validate that your certificate was replaced and that the service is in a healthy state. Let’s hope for no side-effects of this delicate operation.
If you for some reason you can’t use the VMware certificate tool (I recommend you DO use it, since it provides some certificate checking and is less error prone. But it could have issues that prevent you from using it. If so, then you can follow the steps below. They are directly from KB 2058519, with a couple of corrections (I’ve submitted fixes, so hopefully they will correct them).
Thanks to my vCenter 5.5 toolkit script 90% of the tedious work in that KB article is already done for you. Whoohoo. All we need to do is issue three commands to update the lookup service, then copy over our new certs. That’s it!
1. Open an elevated command prompt (not PowerShell) and enter the following command, replacing the vCenter FQDN, paths, and password as needed.
"C:\Program Files\VMware\Infrastructure\VMware\cis\vmware-sso\ssolscli" updateService -d https://d001vctr01.contoso.net:7444/lookupservice/sdk -u ad***********@vs*****.local -p YourPassword -si D:\certs\vCenterSSO\gc_id -ip d:\certs\vCenterSSO\gc.properties
2. Enter the following command:
"C:\Program Files\VMware\Infrastructure\VMware\cis\vmware-sso\ssolscli" updateService -d https://d001vctr01.contoso.net:7444/lookupservice/sdk -u ad***********@vs*****.local -p YourPassword -si D:\certs\vCenterSSO\admin_id -ip d:\certs\vCenterSSO\admin.properties
3. Enter the following command:
"C:\Program Files\VMware\Infrastructure\VMware\cis\vmware-sso\ssolscli" updateService -d https://d001vctr01.contoso.net:7444/lookupservice/sdk -u ad***********@vs*****.local -p YourPassword -si D:\certs\vCenterSSO\sts_id -ip d:\certs\vCenterSSO\sts.properties
4. If all goes well then your screen should look similar to the one below, with three success messages. If not, you did something wrong. Depending on what’s goofed up, SSO may be in a hosed state and require a re-install.
5. Now that the services have been updated, we need to overwrite some certificates. Navigate to the C:\ProgramData\VMware\CIS\runtime\VMwareSTS\conf directory. Backup the ssoserver.crt, ssoserver.key and ssoserver.p12 files. In the vCenterSSO directory that the toolkit script created, copy the ssoserver.crt, ssoserver.key and ssoserver.p12 files and overwrite the old versions.
6. In an elevated command prompt type:
net stop vmwarests
net start vmwarests
To verify that the SSO service is using the new certificate and didn’t suffer fatal stab wounds, open your favorite browser and go to the lookup service URL. That should be https://vCenterName:7444/lookupservice/sdk It should open without any SSL errors and if you look at the certificate by clicking on the lock icon, it will be issued by your CA. The XML response below is normal (yes even the Unexpected EOF), since we didn’t give the SDK service any input data.
With the help of my Toolkit script and the VMware automation tool, replacing the SSO certificate is not as painful as it was in the early months of vCenter 5.1. Even if you need to replace it manually, the toolkit does a lot of the tedious work to make success more likely. Next up in Part 11 is installing the Web Client, updating the SSL certificates, and configuring IE 10.
Derek great Blog as always.
I ran into an issue installing the web client after replacing the SSO certificates. On the screen where it asks for your administrator password it errors sayin it cant contact the lookup service. I rolled back the certificates to the one installed with SSO and then was able to install the web client.
I think given this issue I will install all components before upgrading the ceretificates.
I have this same issue. Any updates?
Given that your question was over half a year ago, I'm guessing not. Maybe something changed with U2? I have to roll back as well, my guess is (since it didn't last time) it's not going to make it through the cert process this time either.
Hi Derek, Thanks for all of this, much appreciated. However something goes wrong on my side I guess. I get the error message 'The certificate chain file does not contain a valid certification path'. I checked the chain.pem file and it looks OK. When I go for the manual update is goes OK and even the verification is OK then. However when I then start with the install of the webclient, I get a error that it can't connect. I checked the log files and in there ther is an error regarding the certificates as well. I followed all the… Read more »
Hi Derek, I resolved the issue finally 🙂 The issue turned out to be in the CA. In our CA for some reason we have 2 CA certificates (on old and one new, but both active for some reason) Your script downloads the old in my case (I gues due to the name) but due to some reason the vCenter server does not accept that certificate, even if it is in the trusted root. In your script I found the line that if a certificate was already in the Cert directory, then it would skip the download, so I downloaded… Read more »