vSphere 6.0 Install Pt. 2: Platform Services Controller


In this second installment of the vSphere 6.0 installation series we cover the new service called the Platform Services Controller. This new service has consumed the SSO service, and added a number of new features. vCenter functionality is now divided into two primary roles, the management node and the PSC. The remainder of this article will cover this split, what the PSC is, and what it means to your vCenter topology.

Blog Series

vSphere 6.0 Install Pt. 1: Introduction
vSphere 6.0 Install Pt. 2: Platform Services Controller
vSphere 6.0 Install Pt. 3: Certificate Management
vSphere 6.0 Install Pt. 4: vCenter Upgrade Best Practices
vSphere 6.0 Install Pt. 5: ESXi Upgrade Best Practices
vSphere 6.0 Install Pt. 6: Install Windows PSC
vSphere 6.0 Install Pt. 7: Config SQL DBs
vSphere 6.0 Install Pt. 8: Toolkit Configuration
vSphere 6.0 Install Pt. 9: SSL Templates
vSphere 6.0 Install Pt. 10: Install VCSA PSC
vSphere 6.0 Install Pt. 11: VMCA as Subordinate
vSphere 6.0 Install Pt. 12: PSC Machine Certificate
vSphere 6.0 Install Pt. 13: Directory Services Certificate
vSphere 6.0 Install Pt. 14: Windows vCenter Install

Permalink to this series: vexpert.me/Derek60
Permalink to my Toolkit script: vexpert.me/toolkit60

Prior to vSphere 6.0 most people thought of vCenter as a single component. Some larger complex deployments may have had multiple SSO servers, but you still think of SSO as fundamental to vCenter. Starting in vSphere 6.0 vCenter is being decoupled into two distinct parts. The first is known as the Platform Services Controller (PSC), and the remainder are all the vCenter services. You can see this in the VMware graphic below, showing the management node (vCenter services) and the PSC. Other VMware products like the vRealize suite can utilize the PSC services, such as authentication.


When you install vCenter now, it no longer prompts you for various components. It installs the full suite every time…whether you want everything or not. This full suite includes the following components:

  • vCenter Server
  • vSphere Web Client
  • Inventory Service
  • Profile-driven storage
  • Auto Deploy
  • Syslog Collector
  • ESXi Dump Collector
  • and more…

Separately installable IS vSphere Update Manager. You can combine the VCSA (vCenter appliance) and a Windows VUM server, so it makes sense they give you the option to separately install VUM. The only other separately installable component is the vSphere authentication proxy. Keep in mind you still need the Windows thick C# client to manage all the VUM features. VMware is working on a VUM replacement, but it’s TBD when that will be released. Personally I wouldn’t look for that anytime soon.

Platform Services Controller (PSC)

As we all remember with vSphere 5.1, VMware introduced a major new component called SSO, or single sign on. In the 5.1 days this caused a lot of headaches and pain, particularly around SSL certificates. The situation was improved in vSphere 5.5 with SSO 2.0, and VMware provided more guidance to customers for SSO topologies. Fast forward to vSphere 6.0, and now there’s a whole new component called Platform Services Controller (PSC). PSC is a shared service that supports vCenter server and vCenter server components. Various VMware products can use the PSC such as vCenter and vRealize Operations. Think of PSC as a foundational service which provides the following services:

  • License service (formerly held by vCenter)
  • Single Sign-on Service (Secure Token Service, identity management service, directory service)
  • Lookup Service
  • VMware Certificate Authority (VMCA)
  • VMware Endpoint Certificate Store (VECS)
  • Authentication framework daemon
  • Component Manager Service (CM)
  • HTTP reverse proxy

The big news here is the brand new VMware certificate authority (VMCA) and VMware Endpoint Certificate store (VECS). This will have a major impact on how you deploy and manage certificates in a vCenter environment. More to come on that in my next post.

Also keep in mind that you can mix and match PSC embedded installs along with external instances. All will share the same SSO domain, and will replicate other information like licensing. This lets you start out small, say with an embedded install, and expand in the future with a replicated external instance. Or perhaps you add a second datacenter down the road, with a local vCenter instance. There’s a limit of 8 PSCs per site.

Do keep in mind that if you start out with an embedded install (vCenter + PSC), there’s no supported method to “move” the PSC to a dedicated server or add more PSCs. So my advice is to start out by dedicating a VM to the PSC, and anther VM to all the other vCenter services. This provides maximum scalability, and future proofs your infrastructure for the foreseeable future.

High availability (HA) for the PSC service is via an external load balancer, such as a Citrix Netscaler or F5. Unfortunately built-in HA didn’t make it into this release, so we must rely on a third party solution. In the VMware graphic below you can see three vCenter instances, all pointing to a redundant and load balanced PSC deployment.



vCenter single sign-on allows vSphere components to communicate with each other using secure tokens rather than authenticating separately to each component. The SSO service is comprised of the security token service, administration server, VMware directory service (vmdir) and identity management service. The STS service uses SAML tokens for authentication. The administration server allows administrators with the proper privileges to manage the SSO service. VMdir is a multi-master, multi-tenant directory service that uses LDAP, much like Active Directory. However it does not use AD nor ADAM/ADLS. It uses port 389 and 11711. If there is more than one instance of PSC in your environment then one update in vmdir will be replicated to all other instances. Vmdir also stores some certificate information. The identity management service processes identity sources and STS authentication requests.

Starting with vSphere 6.0 SSO is either deployed via the embedded PSC (co-resident with your vCenter server) or as part of an external PSC deployment. Of great importance is the order of installation. If you are using an external PSC, then it must be installed prior to vCenter server. This make sense, as vCenter depends on a number of the services in the PSC. If you choose the embedded install then everything is installed in the right order.

A single external PSC deployment can support up to eight vCenter instances. Each new vCenter instance connects to the same PSC server. If you are a monster enterprise environment and have more than 8 vCenters then you can deploy multiple PSC instances.

When you deploy the SSO component you will need to configure a password for the administrator@vsphere.local account. This password needs to be at least 8 characters, one lower case character, one numeric character and one special character. And the length must NOT exceed 20 characters. Only use visible ASCII characters, meaning you can’t use a space. You are also not allowed to use the single quote (‘) either. Other special characters may cause problems (remembering the SSO 5.1 and 5.5 days) so be sure to test out your “complex” passwords.

By default users are locked out of the SSO service after five consecutive failed attempts in three minutes. Accounts are unlocked after five minutes. These policies can be changed, if desired. See the vSphere Security Guide for additional details.

Below is a graphic from VMware showing the progression of SSO to the PSC, from vSphere 5.1 to 6.0.



As you can see, the Platform Services controller is new to vSphere 6.0 and incorporates a number of new services. Medium to large deployments can get complex, with a mix of multiple vCenter servers and highly available PSCs. Carefully evaluate your business requirements for scalability and high availability, then chose a the simplest deployment model that best meets those requirements.

VMworld 2014: vCenter Server Architecture Deep Dive

Session INF2311, Justin King

vCenter Server Configuration Options: 5.0 –

Configuration Option for v5.5 #1:

  • Use simple installer
  • Multiple vCenters for different geographic locations
  • Single SSO authentication domain

Configuration for 5.5 #2:

  • Centralized SSO on dedicated VM
  • A datacenter with 3 or more solutions (e.g. vCAC, etc.)
  • Availability uses vSphere HA and network load balancer

Utilize A management cluster

  • Run multiple vCenter components together on same virtual machine minus the database
  • Recommendations: 3 vSphere Hosts, enable vSphere HA

vCenter SSO recommendations

  • Embedded vCenter SSO reduces complexity
  • Up to 8 instances
  • 12ms latency
  • Same vSphere.local domain
  • Centralized SSO-only VMs (3 or more solution like vCenter, vCAC, etc.)
  • All configurations: Backup each instance

vSphere client in vSphere 5.5 Update 2 supports HW v10. Yippee!

vCenter Server Tech Preview

  • VMware Platform services controller is now known as a “platform services controller” in addition to SSO. Certificates, licensing, etc. will register with the platform controller.
  • You can embed the PSC (platform service controller) with vCenter, just like you did with SSO
  • Think of the PSC as the new SSO, but with a lot more services
  • The existing SSO topologies (embedded or external) are valid for the SSO
  • You can mix and match PSC embedded and external instances, all sharing the same SSO domain

vSphere 6.0 Tech Preview Install and Upgrade

  • One installer that allows you to choose the deployment type (embedded or external PSC)
  • Asks for all input up front, validate, then will deploy the software
  • Scripted install for advanced users
  • Linux appliance install is completely new with a guided install with pre-check installer
  • Full upgrade path from 5.0, 5.1 and 5.5


  • Appliance model is now on par with Windows in terms of number off VMs and ESXi hosts
  • Linked mode drops ADAM and will be supported on the Linux appliance


vSphere 5.5 Install Pt. 12: Configure SSO

10-12-2013 8-02-44 AMNow that the SSO service and web client are installed, it’s time to do a little SSO configuration. In this installment we will configure the SSO STS certificate chain, add an Active Directory identity and source, and delegate SSO administrative rights to a AD group.

If you recall the vCenter 5.1 installation order, you will realize they’ve now moved up the web client install. This was done consciously so you could troubleshoot/configure the SSO service prior to vCenter being installed. Great idea VMware!

Blog Series

SQL 2012 AlwaysOn Failover Cluster for vCenter
vSphere 5.5 Install Pt. 1: Introduction
vSphere 5.5 Install Pt. 2: SSO 5.5 Reborn 
vSphere 5.5 Install Pt. 3: vCenter Upgrade Best Practices and Tips
vSphere 5.5 Install Pt. 4: ESXi 5.5 Upgrade Best Practices and Tips 
vSphere 5.5 Install Pt. 5: SSL Deep Dive
vSphere 5.5 Install Pt. 6: SSL Certificate Template
vSphere 5.5 Install Pt. 7: Install SSO
vSphere 5.5 Install Pt. 8: Online SSL Minting
vSphere 5.5 Install Pt. 9: Offline SSL Minting
vSphere 5.5 Install Pt. 10: Update SSO Certificate
vSphere 5.5 Install Pt. 11: Install Web Client
vSphere 5.5 Install Pt. 12: Configure SSO
vSphere 5.5 Install Pt. 13: Install Inventory Service
vSphere 5.5 Install Pt. 14: Create Databases
vSphere 5.5 Install Pt. 15: Install vCenter
vSphere 5.5 Install Pt. 16: vCenter SSL
vSphere 5.5 Install Pt. 17: Install VUM
vSphere 5.5 Install Pt. 18: VUM SSL
vSphere 5.5 Install Pt. 19: ESXi SSL Certificate

Permalink to this series: vexpert.me/Derek55
Permalink to the Toolkit script: vexpert.me/toolkit55

Configure SSO STS Chain

For some reason the VMware certificate tool does not automatically import the trusted CA chain into the SSO STS store. So we need to manually do that. My Toolkit script creates the complex Java keystore file, which is quite tedious. See Part 8 for the low down on my vCenter 5.5 Toolkit script. So all we need to do here is import the Java keystore file. I’m opting to leave the default self-signed chain in place, just in case there is a dependency.

1. Login to the vSphere web client with the administrator@vsphere.local account. In the left pane click Administration.

10-12-2013 8-04-42 AM

2. Under Single Sign-On click Configuration. Then click on the Certificates tab and then STS Signing.

10-12-2013 8-08-04 AM

3. Click on the green Plus sign and navigate to the vCenterSSO certificate directory the Toolkit script created. Select the server-identity.jks file. When prompted for a password enter testpassword.

10-12-2013 8-10-08 AM

4. Depending on your CA configuration you should see two or three certificates listed. In my case I have three, since I have a root and intermediate CA. Click on the ssoserver line and then click OK. Enter testpassword again.

10-12-2013 8-12-34 AM

If the import is successful you should see two certificate chains.

10-12-2013 8-14-37 AM

5. Reboot your vCenter server so that all the services are refreshed and pickup the new certificate chain.

Add Identity Source

In vSphere 5.5 your Active Directory identity source is not automatically added. So we will need to add AD as a source so you can authenticate with domain-based accounts.

1. Login to the vSphere web client, in the left pane click on Administration. Under Single Sign-On click Configuration. Click on Identity Sources in the middle pane.

10-12-2013 8-40-28 AM

2. Click on the green plus sign. If you want rich Active Directory support then choose Active Directory (integrated Windows Authentication). Chosing Active Directory as LDAP Server is for 5.1 backwards compatibility and should NOT be used. You will have issues with domain trusts, etc. Should be avoided!

10-12-2013 8-39-34 AM

3. After the source is added you should see three Identity Sources.

10-12-2013 8-43-30 AM

Delegate SSO Admin Rights

1. Create a group in Active Directory that you want to delegate SSO administrator rights too. In my case the group is called APP_VCTR_SSO_Admin. You can use whatever name you wish. Put your account into that group.

1. On the Groups tab click on Administrators, then in the lower Group Members pane click on the Blue Man Group person.

10-12-2013 8-59-54 AM

2. Change the domain to your AD domain, then find your group. Highlight the group then click on Add. Then you can click on OK to add the group.

10-12-2013 9-12-39 AM

3. If you log out of Windows then log back in (to refresh your group membership), you should now be able to use the Windows credential option to access the vSphere web client. The first time you try it a warning message will likely appear. I would uncheck the Always Ask box unless you like exercising your fingers.

10-12-2013 11-34-48 AM

10-12-2013 11-25-55 AM


Configuring some basic SSO settings is not rocket science, but common to many environments. At a minimum you need to import the SSO STS certificate chain. Nearly everyone has AD, so adding the more intelligent SSO 5.5 AD identity source will be on everyone’s agenda. Shared accounts are never a good idea, so setting up a group for SSO admin delegation is a great idea.

Next up in lucky Part 13 we install the Inventory Service and secure it with trusted SSL certificates.

vSphere 5.5 Install Pt. 10: Replace SSO Certs

10-11-2013 6-44-21 PMJust like replacing a hip joint, replacing vCenter SSO SSL certificates can induce some pain, is a bit complex, and the outcome can be questionable. The replacement process in SSO 5.5 is pretty much like that in 5.1, but now we have the VMware certificate automation tool and my vCenter toolkit to make this a safer operation. Outcome is not guaranteed, and there may be side effects.

My approach to replacing SSL certificates in this series is replacement right after the service is installed. Basically you build up a trusted infrastructure from the get-go. VMware would likely say to build it up all untrusted, then go back and replace the certs. Certainly a valid point. If the VMware automation tool was more automated, then I might agree. But by building up trusted layers there are less replacement steps and chance of messing up, IMHO.

I do strongly recommend using the VMware certificate automation tool instead of manual replacement steps. But for the SSO service I’ll show you both ways, each using my vCenter Toolkit script to prepare the needed files.

Blog Series

SQL 2012 AlwaysOn Failover Cluster for vCenter
vSphere 5.5 Install Pt. 1: Introduction 
vSphere 5.5 Install Pt. 2: SSO 5.5 Reborn 

vSphere 5.5 Install Pt. 3: vCenter Upgrade Best Practices and Tips
vSphere 5.5 Install Pt. 4: ESXi 5.5 Upgrade Best Practices and Tips 
vSphere 5.5 Install Pt. 5: SSL Deep Dive
vSphere 5.5 Install Pt. 6: SSL Certificate Template
vSphere 5.5 Install Pt. 7: Install SSO
vSphere 5.5 Install Pt. 8: Online SSL Minting
vSphere 5.5 Install Pt. 9: Offline SSL Minting
vSphere 5.5 Install Pt. 10: Replace SSO Certs
vSphere 5.5 Install Pt. 11: Install web client
vSphere 5.5 Install Pt. 12: Configure SSO
vSphere 5.5 Install Pt. 13: Install Inventory Service
vSphere 5.5 Install Pt. 14: Create Databases
vSphere 5.5 Install Pt. 15: Install vCenter
vSphere 5.5 Install Pt. 16: vCenter SSL
vSphere 5.5 Install Pt. 17: Install VUM
vSphere 5.5 Install Pt. 18: VUM SSL
vSphere 5.5 Install Pt. 19: ESXi SSL Certificate

Permalink to this series: vexpert.me/Derek55
Permalink to the Toolkit script: vexpert.me/toolkit55

Automated Certificate Replacement

1. Download the vCenter Certificate Automation Tool v5.5. You can find it under the Drivers and Tools section of the vSphere 5.5 downloads. Direct link is here. Unzip the contents on your vCenter server.

10-9-2013 6-10-08 PM

2. At the beginning of my toolkit script are two variables that set the SSO and vCenter administrator account names. Now during the SSO install you can’t change these names, so you shouldn’t need to change them here. But just FYI, I wanted to point out they are defined in the toolkit script.

10-9-2013 6-23-42 PM

3. Re-run my vCenter 5.5 Toolkit script, but this time we want to create the automation batch file. The menu number may change, but in today’s version of the script it is option 4.

10-9-2013 6-17-27 PM

4. The script will run in a split second and provide you the path to the batch file. The batch file sets the same variables as the stock VMware ssl-environment.bat file. I stripped out all of the comments and set the paths to where your certificate files are stored, assuming you used my script for part 8 or 9, minting your SSL certs.

5. Copy the ssl-environment.bat file that the toolkit created and overwrite the one in the VMware tool directory. You don’t need to run the batch file as the main updater script will execute it in the background.

10-9-2013 7-14-01 PM

6. Run the VMware ssl-updater.bat file. On the main menu select Option 3.

10-9-2013 6-27-30 PM

7. On the next menu select option 1.

10-9-2013 6-29-58 PM

8. When you select option 1 it will ask you a series of questions (in yellow below). Most of the information should be pre-populated for you. But you do need to input the administrator@vsphere.local password and answer if you are using a load balancer or not (don’t use one, VMware does not recommend it). Cross your fingers and toes, and watch for success messages.

10-9-2013 6-53-45 PM

9. Skip down to the verification section to validate that your certificate was replaced and that the service is in a healthy state. Let’s hope for no side-effects of this delicate operation.

Manual Replacement

If you for some reason you can’t use the VMware certificate tool (I recommend you DO use it, since it provides some certificate checking and is less error prone. But it could have issues that prevent you from using it. If so, then you can follow the steps below. They are directly from KB 2058519, with a couple of corrections (I’ve submitted fixes, so hopefully they will correct them).

Thanks to my vCenter 5.5 toolkit script 90% of the tedious work in that KB article is already done for you. Whoohoo. All we need to do is issue three commands to update the lookup service, then copy over our new certs. That’s it!

1. Open an elevated command prompt (not PowerShell) and enter the following command, replacing the vCenter FQDN, paths, and password as needed.

"C:\Program Files\VMware\Infrastructure\VMware\cis\vmware-sso\ssolscli" updateService -d https://d001vctr01.contoso.net:7444/lookupservice/sdk -u administrator@vsphere.local -p YourPassword -si D:\certs\vCenterSSO\gc_id -ip d:\certs\vCenterSSO\gc.properties

2. Enter the following command:

"C:\Program Files\VMware\Infrastructure\VMware\cis\vmware-sso\ssolscli" updateService -d https://d001vctr01.contoso.net:7444/lookupservice/sdk -u administrator@vsphere.local -p YourPassword -si D:\certs\vCenterSSO\admin_id -ip d:\certs\vCenterSSO\admin.properties

3. Enter the following command:

"C:\Program Files\VMware\Infrastructure\VMware\cis\vmware-sso\ssolscli" updateService -d https://d001vctr01.contoso.net:7444/lookupservice/sdk -u administrator@vsphere.local -p YourPassword -si D:\certs\vCenterSSO\sts_id -ip d:\certs\vCenterSSO\sts.properties

4. If all goes well then your screen should look similar to the one below, with three success messages. If not, you did something wrong. Depending on what’s goofed up, SSO may be in a hosed state and require a re-install.

10-9-2013 8-16-46 PMA

5. Now that the services have been updated, we need to overwrite some certificates. Navigate to the C:\ProgramData\VMware\CIS\runtime\VMwareSTS\conf directory. Backup the ssoserver.crt, ssoserver.key and ssoserver.p12 files. In the vCenterSSO directory that the toolkit script created, copy the ssoserver.crt, ssoserver.key and ssoserver.p12 files and overwrite the old versions.

6. In an elevated command prompt type:

net stop vmwarests
net start vmwarests


To verify that the SSO service is using the new certificate and didn’t suffer fatal stab wounds, open your favorite browser and go to the lookup service URL. That should be https://vCenterName:7444/lookupservice/sdk It should open without any SSL errors and if you look at the certificate by clicking on the lock icon, it will be issued by your CA. The XML response below is normal (yes even the Unexpected EOF), since we didn’t give the SDK service any input data.

10-9-2013 7-02-19 PM

10-9-2013 7-05-34 PM


With the help of my Toolkit script and the VMware automation tool, replacing the SSO certificate is not as painful as it was in the early months of vCenter 5.1. Even if you need to replace it manually, the toolkit does a lot of the tedious work to make success more likely. Next up in Part 11 is installing the Web Client, updating the SSL certificates, and configuring IE 10.

vSphere 5.5 Install Pt. 8: Online SSL Minting

10-3-2013 8-04-58 PM

In the last installment of the vSphere 5.5 series we installed the SSO service. Now that the Java JRE is installed (via the SSO installer), we have the tools ready to create our vCenter SSL certificates via an online Microsoft CA. If you don’t have an online Microsoft CA that can issue your VMware SSL certificates, skip this section and go to Part 9 (coming soon), where we go through the manual process.

I’m recommending you create the SSL certificates now, so that you have a variety of methods at your disposal to use these certificates. VMware’s stance is that you fully install vCenter 5.5 with self-signed certificates then use their free certificate automation tool to replace all of the certificates as the last step. That is certainly a good route to go, and I would not dissuade you from that method. Or, you could also replace certificates as you install components so each layer is trusted as you go. Either way, I recommend the VMware certificate tool even if it is a bit primitive. It is flexible enough to let you incrementally replace certificates.

In order to make life easier for installers I’ve written a “toolkit” PowerShell script to help with the SSL process. More details are below, but I would like to give credit to Chrissy LeMaire for some of the (modified) building blocks of this script. She wrote a vCenter 5.1 PowerShell SSL replacement script that was more automated than VMware’s batch script. My script does not replace VMware’s automation tool, but helps you prepare the files it needs.

Download Toolkit Here

Blog Series

SQL 2012 AlwaysOn Failover Cluster for vCenter
vSphere 5.5 Install Pt. 1: Introduction 
vSphere 5.5 Install Pt. 2: SSO 5.5 Reborn 

vSphere 5.5 Install Pt. 3: vCenter Upgrade Best Practices and Tips
vSphere 5.5 Install Pt. 4: ESXi 5.5 Upgrade Best Practices and Tips
vSphere 5.5 Install Pt. 5: SSL Deep Dive
vSphere 5.5 Install Pt. 6: SSL Certificate Template
vSphere 5.5 Install Pt. 7: Install SSO
vSphere 5.5 Install Pt. 8: Online SSL Minting
vSphere 5.5 Install Pt. 9: Offline SSL Minting
vSphere 5.5 Install Pt. 10: Update SSO Certificate
vSphere 5.5 Install Pt. 11: Install Web Client 
vSphere 5.5 Install Pt. 12: Configure SSO
vSphere 5.5 Install Pt. 13: Install Inventory Service
vSphere 5.5 Install Pt. 14: Create Databases
vSphere 5.5 Install Pt. 15: Install vCenter
vSphere 5.5 Install Pt. 16: vCenter SSL
vSphere 5.5 Install Pt. 17: Install VUM
vSphere 5.5 Install Pt. 18: VUM SSL
vSphere 5.5 Install Pt. 19: ESXi SSL Certificate

…possibly more to come…

Derek’s Toolkit Script

The PowerShell script performs several tasks and is menu driven. It’s an all in one script, meaning it handles online/offline CAs, and will also do other install tasks like create your ODBC connectors. That functionality is not yet in there, but will be added in the coming weeks. The full feature list will change and so will the menus. But I’ll try and keep this updated as often as I can.

The CSRs are in strict accordance with VMware KB articles regarding certificate requirements, including an optional IP address in the SAN field. I want to strictly color within the KB article lines, so if you do use this script and then have to call up VMware support they won’t roll their eyes and have you re-do your certs because some blogger got it wrong.

The script has the following features:

  • Downloads and installs the proper version of OpenSSL if it’s not already installed
  • Creates 2048 bit RSA private keys in the proper format
  • Creates a directory for each service bundle of SSL certificates
  • Generates seven OpenSSL configuration files, one for each certificate, in the appropriate directory
  • Downloads both root and subordinate root public certificates
  • Submits the CSRs to the online CA and downloads the certificates
  • Creates the needed service PEM files for the vCenter certificate automation tool
  • Creates the required root/subordinate PEM files
  • Handles the special SSO 5.5 certificate requirements
  • Does NOT require PowerCLI
  • Assumes all vCenter components are on one server
  • Automatically uses the hostname of the server you run the script on for all certificates
  • Creates a pre-filled vCenter Certificate Automation environment script – Just run!
  • Works with offline CAs
  • Creates SSO 5.5 certificate replacement files – Only used if manual replacing certs
  • Creates customized SQL vCenter and VUM database creation script
  • Creates SQL ODBC DSNs for vCenter and VUM
  • Automatically downloads and installs SQL 2008 R2 or SQL 2012 client package
  • Linux vCenter Server Appliance support for online minting and offline CSR creation
  • Creates certificates for Auto Deploy, Dump Collector, Syslog collector, Authentication Proxy
  • Support Microsoft CAs that require manual certificate approval
  • Requires PowerShell 3.0 or higher

Download Toolkit Here

Toolkit Change Log

v1.56, January 19, 2014

  • Fixed bug when no subordinate CA was present
  • Changed Microsoft “renewal” default to 0 for root/sub CA

v1.55, January 12, 2014

  • Added additional CA/subordinate error checking

v1.50, December 22, 2013 Changes:

  • Added support for ESXi hosts

v1.42, December 3, 2013 Changes:

  • Modified how the certificate hash files are created
  • Added Authentication Proxy certificate creation
  • Changed MS CA download parameter Renewal from 0 to 1

v1.41, Nov 14, 2013 Changes:

  • Changed the root/intermediate CA download order and added more error checking

v1.4, Nov 10, 2013 Changes:

  • Added Auto Deploy, Dump Collector and Syslog Collector SSL certificates
  • Added support for manually approved CA certificates (Windows & Linux)
  • Added SHA512 request in CSR
  • Added request for vCenter FQDN in Option 3

Special thanks to Ryan Bolger from Trace3 for the CA manual approval code.


Online SSL Minting

1. Download my vCenter 5.5 toolkit script from the links above. Open it in the PowerShell ISE (or favorite editor). The PowerShell script requires a few variable modifications before you run it. In the first block of variables you need to setup the directory where you want all of the certificates to go. If OpenSSL is already installed, change the path so the script knows where the root directory is. If that directory does not exist OpenSSL will be downloaded and installed for you. Next up are the certificate properties. Change those to suite your environment. If you want the server’s IP address in the SAN field, then uncomment the line and change the IP.

10-10-2013 7-04-44 PM

2. The script is semi-intelligent about using only a root, or one subordinate and root. Simply comment out $SubCA with a # if you only have an online Microsoft root CA. If you have two or more subordinates, then you will need to follow VMware SSL KBs or modify my script. Sorry!

10-7-2013 8-17-46 PM

If for some reason the script can’t download your CA certs automatically, try changing the download protocol. The default is HTTPS. If that still fails, you can manually place the Base-64 encoded root (and intermediate) certificates into your $Cert_Dir path. The root should be called Root64.cer and the intermediate called interm64.cer.

10-10-2013 7-08-34 PM

10-10-2013 7-12-20 PM

3. The next section are the details for your issuing CA and the template. The issuing CA is your online CA that will actually mint your certificates. If you only have one CA, then clearly that is what you should use. The $ISSUING_CA field can be a little tricky. The first field is the shortname (or FQDN) of your CA (e.g. d001dc01). Next up is your CA name. This can be anything, so you must open the Certificate Authority MMC on your CA to find out what it’s called. As you can see from my screenshot below my CA name is contoso-D001DC01-CA.

10-3-2013 9-06-22 PM

10-9-2013 4-35-46 PM

Now if you don’t have MMC access on the CA to look up the actual CA name, then you can find it another way. Open a browser and go to https://yourCAserver.domain.com/certsrv and select Download a CA certificate, certificate chain, or CRL. Select Download CA Certificate Chain. Open the files, then open the certificate(s) in the file. Now depending on how your CAs are setup, it may take a little thought to correlate the “Issued to” and “Issued by” to your root/subordinate CA. In my case contoso-D001DC02-CA is my root CA, and contoso-D001DC01-CA is my subordinate. I want my VMware certs issued from the subordinate, so just like the MMC screenshot above, the CA name that corresponds to D001DC01 is contoso-D001DC01-CA.

12-3-2013 6-43-46 PM

4. Next up is the template name. This can also be any value, but if you followed my guide then it will be called VMware-SSL. This is the Template Name not the Template display name.

10-3-2013 9-08-57 PM

10-9-2013 4-37-42 PM

5.  Your account must have the required CA permissions to enroll for the VMware-SSL template. If it does not, then find a CA administrator, have them logon the vCenter server and run the script. If your online Microsoft CA is configured for manual certificate approval, you can still use Option #1.

1-9-2014 9-01-08 PM

6. Since you have properly configured the script variables and have one or more online Microsoft CAs, you should select option 1. New to v1.3 and later it will ask you to confirm your vCenter FQDN. If the FQDN is correct, just press ENTER. If it is wrong, just enter the right FQDN.

10-22-2013 8-30-54 PM

After confirming/entering your FQDN the process is automated. If you do get errors, then you either goofed up the variables, have insufficient permissions, or my script is broke and needs fixing. If its broke, now is an excellent time to learn PowerShell. Script is provided as-is, and bugs/issues may or may not be fixed.

Below is a screenshot with a sample of the script output as it runs using an online CA with automatic approval. A lot more has scrolled off the screen, but you get the idea. There is limited error checking, but subtle issues could fly by on the screen. Review the output for any issues.

10-9-2013 5-00-02 PM

7. If your CA is configured for manual approval you will get a list of the request IDs for all 10 certificates. Have your CA administrator approve the requests, then run Option #3 to complete the process. You will only see the Request IDs when manual request approval is required. When you run Option #3 the output should look like the previous screenshot.

11-10-2013 7-10-23 PM

Output Validation

1. When the script completes you should have eleven directories, and either one or three certificate (.cer) files in the root of your working directory. If you have a subordinate CA then you will have three files. If you have a single CA you will only have a Root64.cer file. The two files with funny names are hash files of the root and intermediate CAs. If you only have a root CA you will see a single hash file.

12-3-2013 6-52-23 PM

2. If you take a peek inside one of the folders you will see a series of files. Each service, except SSO, will have the same set of files (except the .csr and .cfg with are uniquely named). The

  • chain.pem: Used for the VMware vCenter certificate automation tool
  • rui.crt: Public half of your SSL certificate
  • rui.key: Private half of your SSL certificate
  • rui.pfx: Combined private and public SSL keys
  • *.cfg:  Certificate signing request file
  • *.csr: Certificate signing request

10-9-2013 5-09-43 PM

3. In the vCenterSSO you will see a plethora of files. Depending on how you replace your SSL certificates, you may only use some of these files. But to help you out as much as possible, all the SSO files that are tedious to create manually are created for you. If you are missing files, then something went wrong. Please match up all filenames to validate the toolkit script worked. Some files are copies of each other, but they are needed to avoid confusion and more easily follow the KBs.

  • *.properties: Use for manual SSO SSL replacement
  • *_id: Use for manual SSO SSL replacement
  • ca_certificates.crt: Use for manual SSO SSL replacement
  • root-trust.jks: Used for SSO/STS certificate validation
  • server-identity.jks: Same file as above with a different name (per VMware KBs)
  • ssoserver.p12: Same functionality as rui.pfx, but VMware changed the name and format for SSO 5.5
  • ssoserver.crt: Copy of chain.pem
  • ssoserver.key: Copy of rui.key

10-9-2013 10-06-14 PM

Certificate Validation

Now that your certificates are minted, let’s quickly validate all of the properties are present. Some users have reported corrupted/incorrect root and subordinate certificates, so please do NOT skip this section. Also, even if your CSR requests a property (such as client authentication), that doesn’t mean your CA will honor it. The OU in each subject name should be unique and match the directory its in.

10-10-2013 7-17-04 PM

The Subject Alternative Name should contain the short name and FQDN. Optionally it can contain your IP address too.

10-10-2013 7-18-18 PM

Enhanced key usage should show server and client authentication. Client authentication can be missing if the CA template is wrong.

10-10-2013 7-18-59 PM

Key usage should contain digital signature, key encipherment and data encipherment.

10-10-2013 7-19-43 PM

We also need to validate that the root and intermediate certificates are in the right format. Some users have reported corrupted root or intermediate certificates. In Notepad open both your root64.cer and interm64.cer files. They should both start with —–BEGIN CERTIFICATE—–. If they contain HTML or any non-printable characters, then they are corrupted or in the wrong format. STOP, as they will most certainly NOT work.

1-9-2014 8-52-00 PM

If your root or subordinate CA certs contain HTML, and you are using the script to automatically download them from a Microsoft CA, I have a suggestion that may work. Locate the code snippet below and change Renewal=1 to Renewal=0. That tweaks how the script downloads the certificate from the Microsoft CA. In addition, if the script downloads an old subordinate certificate, change renewal=0 to renewal=1 in the DownloadSub function.

1-9-2014 8-55-17 PM


Assuming you have an online Microsoft CA and you were successful in running the script, you now have all of the files needed to use the VMware certificate automation tool, or go through the manual certificate replacement process. In Part 9 I cover how to use an offline or non-Microsoft CA. At the end of that article you will have exactly the same files as you do from this installment. Starting in Part 10 we will resume our configuration and installation of vCenter components.

vSphere 5.5 Install Pt. 7: Install SSO

10-5-2013 8-45-11 PMYes, seven parts into this series we can finally mount our handy dandy vCenter 5.5 ISO and start installing software. Hopefully I haven’t lost anyone along the way with all of the background and SSL information. But with the complexities in vCenter 5.5 and all the moving parts, I think it’s important to know what’s going on in case you run into issues. I want this series to be more than just screenshots and scripts blindly leading you through an install.

Blog Series

SQL 2012 AlwaysOn Failover Cluster for vCenter
vSphere 5.5 Install Pt. 1: Introduction
vSphere 5.5 Install Pt. 2: SSO 5.5 Reborn 
vSphere 5.5 Install Pt. 3: vCenter Upgrade Best Practices and Tips
vSphere 5.5 Install Pt. 4: ESXi 5.5 Upgrade Best Practices and Tips
vSphere 5.5 Install Pt. 5: SSL Deep Dive
vSphere 5.5 Install Pt. 6: SSL Certificate Template
vSphere 5.5 Install Pt. 7: Install SSO
vSphere 5.5 Install Pt. 8: Online SSL Minting
vSphere 5.5 Install Pt. 9: Offline SSL Minting
vSphere 5.5 Install Pt. 10: Update SSO Certificate
vSphere 5.5 Install Pt. 11: Install Web Client 
vSphere 5.5 Install Pt. 12: Configure SSO
vSphere 5.5 Install Pt. 13: Install Inventory Service
vSphere 5.5 Install Pt. 14: Create Databases
vSphere 5.5 Install Pt. 15: Install vCenter
vSphere 5.5 Install Pt. 16: vCenter SSL
vSphere 5.5 Install Pt. 17: Install VUM
vSphere 5.5 Install Pt. 18: VUM SSL
vSphere 5.5 Install Pt. 19: ESXi SSL Certificate

Permalink to this series: vexpert.me/Derek55
Permalink to the Toolkit script: vexpert.me/toolkit55

Provision vCenter VM

Before we install SSO, we need to provision the vCenter VM. Per VMware recommendations, KB2052334, the VM needs at least 12GB of RAM for a “simple” all in one installation. Don’t skip on memory as performance will likely take a beating, depending on the number of hosts and VMs you are managing.

  • At least 2 vCPUs
  • At least 12GB of RAM
  • At least 70GB D drive (more with VUM)
  • Use hardware version 9 or earlier
  • Recommend Windows Server 2012
  • Enable hot add of memory/CPU
  • Fully patched

If you want to use the web client on the vCenter server with IE, then you must install the Desktop Experience feature. Why? That’s the only way to get Flash player in IE with Windows Server 2012. VMware really needs to dump the Flash interface and go HTML5. If you use a third party browser, make sure you get the very latest Flash player.

After you install the Desktop Experience make sure you patch it. Why? The stock Flash player version is not compatible with the web client and needs to be updated via Windows Update/WSUS/SCCM to the latest version.

10-8-2013 6-11-01 AM

If you will be using IE on the vCenter server you also need to turn off the IE enhanced security mode.

10-8-2013 5-40-17 PM

Basic SSO Install

The installation process in SSO 5.5 is vastly different from vCenter 5.1. As previously mentioned gone is the SQL database requirement, which caused untold grief. Instead of spending days trying to get the SQL JDBC connector working with SSL (which ultimately never did work), you can now click through the install wizard in about 60 seconds. No fuss, no pain, no hair loss. Pure bliss.

1. Login to your vCenter VM and mount the vSphere 5.5a (note the ‘a’ or use the latest available) ISO. Your user account must NOT have an exclamation point in it. If it does, the installer may fail. Use a different account.  Even though we are doing a “Simple Install” in concept, I want to go through the Custom Install. Why? That way we can modify the installation paths (which you can’t do with the simple install), and also more clearly walk through each component. Click on vCenter Single Sign-On then Install.

10-7-2013 7-17-29 PM

2. On the Welcome screen click Next.

10-7-2013 7-20-34 PM

3. Thoroughly read all the entire EULA. (Pausing for 3 hours..)

10-7-2013 7-22-07 PM

4. Review the Prerequisites screen and click Next. Enterprise grade DNS is key, and you must have both forward and reverse records working for your vCenter server. Time is also important, so ensure your vCenter VM is correctly synchronizing with your DCs.

10-7-2013 7-22-54 PM

5. Now you need to choose your SSO deployment mode. In our case we will leave the default option, your very first vCenter server.

10-7-2013 7-25-28 PM

6. Next up we have to enter a password. Now this is tricky, because a number of special characters are illegal and will cause you grief. I do not know the maximum length. Specifically, do NOT use:

Non-ASCII characters
Ampersand (&)
Semicolon  ( ; )
Double quotation mark  ( ” )
Single quotation mark ( ‘ )
Circumflex ( ^ )
Backslash ( \ )
Percent ( % )
Less than ( < )
Exclamation ( ! )
Space (   )

 10-7-2013 7-31-43 PM

7. Now you need to enter a site name. I would change the default value, and make it meaningful. Also, do NOT enter the FQDN or short hostname of your server here. That could cause problems. Site names will become more important in the future, so again, give this a minute or two of thought.

10-7-2013 7-32-58 PM

8. I would not customize the port number unless you REALLY know what you are doing and want to cause yourself some possible future headaches. Just keep the default, guys.

10-7-2013 7-35-41 PM

9. I’m a firm believer of installing most software on a drive other than C. Why? Application logs can fill up a drive, and there could be some security implications as well. My standard is “D” for all major enterprise apps like vCenter. However, per KB 2044953, the web client (not SSO) will not work if installed on any drive but C. So if you want to keep all your vCenter binaries together, you are stuck with the C drive.

10-7-2013 7-37-12 PM

10. On the final screen review all of the settings and verify they are 100% correct. Click Install and wait a few minutes.

10-7-2013 7-39-04 PM

11. You should get a Completed message, and now you can smile.

10-7-2013 7-45-19 PM

SSO Patch Time

With the 5.5 GA version there is a known problem using Windows Server 2012 and Windows Server 2012 domain controllers. VMware has released a patched DLL to resolve the issue. But better than that you should use the vCenter 5.5a (note the ‘a’) ISO which has the fix built in.

If you are using a non-update (i.e. Sept 2013 GA) version of vSphere 5.5, then go to KB2060901 and follow the instructions to replace the indicated DLL. It’s cake to do, so I won’t show you how. Again, please install all components from the 5.5a media or later so you can skip this manual step.


The SSO installation in vSphere 5.5 is vastly easier than it was in 5.1. Just a few clicks and your SSO server is running. No more SQL, JDBC connections, or databases to create. Major improvement! Next up is minting your SSL certificates from an online Microsoft CA in Part 8.

vSphere 5.5 Install Pt. 3: Upgrading vCenter

9-29-2013 7-39-13 AMUpgrades can be scary times with any enterprise product. The more your critical infrastructure relies on a particular solution, or set of solutions, the more imperative it is you fully understand and test the new product. vSphere 5.1 taught us that thorough testing cannot be skipped and you should not rush a new product into production.

Normally for my vSphere installation series I do NOT cover upgrades, or go through an upgrade process in the series. Why? Customer environments wildly vary and a simple lab upgrade will likely not look like or behave like YOUR environment. That’s why its so critical for you to test in your environment. My upgrade would not look like your upgrade.

But, what I am doing in this post and the next installment is covering upgrade best practices to help you understand your road ahead and things to keep in mind. It contains information from VMworld 2013 vSphere 5.5 upgrade sessions, plus links to resources that have been published post-GA. This post covers vCenter only, and the next installment covers VMs, VMFS, ESXi hosts, and other products.

Blog Series

SQL 2012 AlwaysOn Failover Cluster for vCenter
vSphere 5.5 Install Pt. 1: Introduction
vSphere 5.5 Install Pt. 2: SSO 5.5 Reborn
vSphere 5.5 Install Pt. 3: vCenter Upgrade Best Practices and Tips
vSphere 5.5 Install Pt. 4: ESXi Upgrade Best Practices and Tips 
vSphere 5.5 Install Pt. 5: SSL Deep Dive
vSphere 5.5 Install Pt. 6: SSL Certificate Template
vSphere 5.5 Install Pt. 7: Install SSO
vSphere 5.5 Install Pt. 8: Online SSL Minting
vSphere 5.5 Install Pt. 9: Offline SSL Minting
vSphere 5.5 Install Pt. 10: Update SSO Certificate
vSphere 5.5 Install Pt. 11: Install Web Client 
vSphere 5.5 Install Pt. 12: Configure SSO
vSphere 5.5 Install Pt. 13: Install Inventory Service
vSphere 5.5 Install Pt. 14: Create Databases
vSphere 5.5 Install Pt. 15: Install vCenter
vSphere 5.5 Install Pt. 16: vCenter SSL
vSphere 5.5 Install Pt. 17: Install VUM
vSphere 5.5 Install Pt. 18: VUM SSL
vSphere 5.5 Install Pt. 19: ESXi SSL Certificate

Permalink to this series: vexpert.me/Derek55
Permalink to the Toolkit script: vexpert.me/toolkit55

vSphere 5.5 Upgrade Overview

  • Plan your upgrade – Extremely important. KB on update sequence is here.
  • Five major steps: vCenter, VUM, ESXi, VMs, VMFS
  • Key VMware Sites to bookmark: Documentation Center, Compatibility Guide, Interop matrix
  • KB article here for vCenter 5.5 Upgrade using the Simple Installer
  • KB article here for vCenter 5.5 Upgrade using the Custom Installer
  • If you upgrade Windows with a service pack or other system changes and get locked out of SSO, read this KB to regain access
  • Upgrade vCenter to 5.5 before vSphere replication is upgraded (blog post here)

Prior to 5.1 life was simple. You had vCenter Server, vCenter Database server, and vSphere web client (introduced in 5.0, but rarely used). The vCenter server is NOT stateless, meaning the database is not all inclusive. The local vCenter server has SSL certificates and the ADAM database. ADAM is not just for linked mode but holds data such as licenses, roles, and permissions. So don’t stand up a fresh VM, install the “old” version on that VM then do an upgrade to 5.5 and expect everything to be there. It won’t be and further complicates your upgrade process. If you are using vSphere 5.1, then ‘tags’ are also stored locally on the vCenter server and thus not in the database.

Upgrade Matrix

  • In-place upgrade supports vCenter 4.x, 5.0.x, 5.1.x (must be 64-bit host)
  • VMware does NOT support directly migrating an existing 5.x or earlier vCenter Server to a new machine during the upgrade process
  • vCenter Server 5.5 can manage ESX/ESXi 4.x, 5.0.x  and 5.1.x hosts. It will NOT manage ESX 2.x or 3.x hosts.

System Requirements

  • Strongly recommend installing ALL vCenter components on a single VM – Simplified model
  • Simple install – 2 vCPUs, 12GB RAM, 100GB disk
  • Recommended for 400 hosts or 4000 VMs: 4 vCPU, 24GB RAM, 200GB disk
  • vCenter OS Support: Removes WS2003, only supports Windows Server 2008 SP2 and later (including WS2012 but NOT WS2012 R2)

New Install Vs. In Place Upgrade

VMware recommends a fresh install, but sometimes its not just possible. However, do check out the “Inventory Snapshot” Fling, which is a great (unsupported) tool to migrate hosts, VM, and permissions from one vCenter instance to another. It does NOT appear to support tags and currently has some vDS issues. Tags are not stored in the SQL database, so if you use tags then be sure to find a way to migrate them. If you are in a regulated industry and have strict audit requirements you may be legally required to maintain the historical data in your vCenter database and unable to start fresh.

If you have a sprawling 5.1 architecture, with different vCenter components on different VMs, strongly consider a fresh install and do not upgrade. As previously mentioned VMware now urges the “simple install” method where all components are on a single beefy VM. This is a great time to re-visit your architecture and make it easier to manage and follow 5.5 best practices. That’s not to say you can’t upgrade and consolidate at the same time, you can, and VMware has promised some blog posts on how to do just that.

I’ve read reports that upgrading a vCenter 5.1 instance with trusted SSL certificates to 5.5 had problems. I have not personally tried that yet, so I can’t report my own experience. So make sure you have full backups and a tested plan to revert back to 5.1 incase you experience problems.

VMware has stated that the vCenter Server appliance will be the ONLY deployment option sometime in the future. So if you are starting with a fresh install, do take a close look at the VCSA. It still has a few minor gotchas including no support for IPv6, Linked Mode or vCenter Heartbeat. Those features are probably not widely used, so if you aren’t using those features take a serious look at VCSA.

At this time an external SQL database is NOT supported for the VCSA, but in the future when Microsoft releases the ODBC driver for SUSE Linux (currently in tech preview), VMware will support it. VCSA is certified up to 100 hosts and 3000 VMs. If you need to scale beyond that, use Windows.

Installation – Then and Now

vSphere 5.5 features a new Install splash screen, and the component order is different from 5.1. Simple Install should only be used for the first vCenter. All subsequent vCenter/SSO installs should use the custom method. This is due to changes in SSO, and the new automatic replication among SSO servers. Even if you are doing a single vCenter install and want to customize it in ANY way, including directory paths, you must do the custom install.

Upgrade Paths

For “typical” single server upgrades the path is fairly simple. You can do an in place upgrade and all of the required components and configuration settings will be retained. If you are going from pre-5.1, then the only database in play is the vCenter database.

vCenter 5.5 upgrade

If you are already running 5.1, then the upgrade path is ever so slightly different. Since the SSO database in 5.1 is no more, that data is migrated into the new SSO internal database. So post upgrade you are left with only the vCenter upgrade. Yes, no more SQL authentication required or impossible to configure JDBC SSL.

vCenter 5.5 upgrade

If you are one of those adventurous customers that implemented a load balancer with SSO, VMware is really discouraging you to continue with that model. Its complex, SSL creates additional headaches, and just not needed in most environments. Big changes could be coming in the future, but it’s not recommended for 5.5. As mentioned in my previous installment, SSO Reborn, VMware recommends local SSO instances for each site/vCenter. SSO uses multi-master replication to sync data such as identity sources, users, group, and policies. A geographically distributed example is shown below. Notice the local SSO and vCenter instances at each site. VMware SSO 5.5

Linked Mode

Linked mode adds additional complications to the upgrade process. As you may recall you can’t link vCenters of different versions. So you first need to unjoin all vCenters from the linked mode group. Once you upgrade two vCenters to 5.5, you can then re-establish Linked Mode and add other 5.5 vCenters as they come online. The biggest problems with Linked Mode include DNS and NTP failures. It’s critical name resolution works (forward AND reverse) and that the server clocks are all synchronized. All vCenter servers that are linked must also be a part of the same SSO authentication domain.

Host Agent Pre-Upgrade Checker

A tool included on the vSphere 5.5 ISO is the Host Agent Pre-Upgrade checker. Personally I’ve never used it (slipped my mind that it existed). If you choose to use it some simple checks are done against your ESXi hosts to validate that an upgrade will be successful. It’s not exhaustive, so even if your hosts pass the check you could still run into issues. But it’s a little bit of insurance that major gotchas can be discovered ahead of time. It does check items such as sufficient disk space, functional network, file system consistency, required patches are applied.

vCenter Appliance

The VCSA has undergone major scalability increases in 5.5. In 5.1 it was only rated for 5 hosts and 50 VMs when using the embedded database. With 5.5 that is increased to 100 hosts and/or 3000 VMs. So that makes it a much more viable solution for enterprise customers. You can NOT migrate from the Windows vCenter to the VCSA. As mentioned before, there’s also no Linked Mode, vCenter Heartbeat or IPv6. Again, the road map is an appliance only model for vCenter, so now is an excellent time to try it out. VMware said upgrades to future versions will be pretty easy, simplifying life.

Update Manager

You can upgrade VUM from 4.x, 5.0 and 5.1 versions. VUM is still Windows only, so if you do deploy the VCSA you will still need a Windows server to host VUM. The web client in 5.5 also has limited VUM functionality, so the C# is still needed to do things like pushing patches and configuring baselines. During the upgrade you can’t change the installation or download paths. Scheduled tasks remain, but patch baselines are removed.

VMware has hinted/stated that VUM is going the way of the dodo bird. I would expect its replacement to be very different, and probably incorporated into the VCSA. I’m hoping in vSphere 6.0 there’s a good story on the VUM successor.


You need to carefully plan your upgrades, and understand all of the moving components. Generally you would start by upgrading vCenter, then your ESXi hosts. But you may have other products that depend on vCenter which need upgrading first. Thoroughly map out all of your dependencies, read the VMware documentation, then plan in an organized fashion how you are going to upgrade. If you are already on 5.1, custom SSL certificates may trip you up. So really make sure you have a full backup and roll-back plan in case things go pear shaped.

Next up in Part 4 are practices and tip for upgrading ESXi hosts, VMs, and VMFS datastores.

vSphere 5.5 Install Pt. 2: SSO Reborn

9-28-2013 10-05-38 PMImagine having this dream:

You are in the VMware company grocery store where each isle displays a VMware product, shelves fully and neatly stocked with product. You wonder over to the isle labeled ‘SSO 5.1’. To your horror you see a huge mess: SSL certs lying all over the floor, incorrectly configured OUs values, dazed and confused vSphere architects, JDBC connectors missing their SSL wrappers, and an angry mob of customers with a wide assortment of frightening medieval weapons.

You quickly find the store manager, Pat, and tell him “Cleanup on isle 5.1 needed, ASAP.” Magically a person named Justin appears wearing protective gear, a vSphere 5.5 shirt, and asks in British accent, “How may I help you?” You walk over to the mayhem on isle 5.1. Justin pulls out a VMware branded Harry Potter magic wand, and says a proper English incantation. Instantly you are transported one year into the future, the isle is restored to order, customers are cheering and the architects are building vast vSphere 5.5 empires. You suddenly wake up in night sweats and wonder…was that a dream or reality?

The New Reality

Since the SSO service was the center of attention this past year this blog post will highlight some of the issues with 5.1, and how VMware has addressed them in 5.5. The dream sequence above is not too far off from reality….SSO is all new, and dramatically improved. I think this is important to understand, which is why I’m including it the installation series. Think of it as required reading prior to starting the install process.

Blog Series

SQL 2012 AlwaysOn Failover Cluster for vCenter
vSphere 5.5 Install Pt. 1: Introduction
vSphere 5.5 Install Pt. 2: SSO 5.5 Reborn
vSphere 5.5 Install Pt. 3: vCenter Upgrade Best Practices and Tips
vSphere 5.5 Install Pt. 4: ESXi Upgrade Best Practices and Tips
vSphere 5.5 Install Pt. 5: SSL Deep Dive 
vSphere 5.5 Install Pt. 6: SSL Certificate Template
vSphere 5.5 Install Pt. 7: Install SSO
vSphere 5.5 Install Pt. 8: Online SSL Minting
vSphere 5.5 Install Pt. 9: Offline SSL Minting
vSphere 5.5 Install Pt. 10: Update SSO Certificate
vSphere 5.5 Install Pt. 11: Install Web Client 
vSphere 5.5 Install Pt. 12: Configure SSO
vSphere 5.5 Install Pt. 13: Install Inventory Service
vSphere 5.5 Install Pt. 14: Create Databases
vSphere 5.5 Install Pt. 15: Install vCenter
vSphere 5.5 Install Pt. 16: vCenter SSL
vSphere 5.5 Install Pt. 17: Install VUM
vSphere 5.5 Install Pt. 18: VUM SSL
vSphere 5.5 Install Pt. 19: ESXi SSL Certificate

Permalink to this series: vexpert.me/Derek55
Permalink to the Toolkit script: vexpert.me/toolkit55

What is the SSO service?

vSphere 5.1 introduced the SSO service, and it wasn’t just because VMware wanted to write more code and complicate our lives. In fact, it was designed to simplify life and provide common authentication services to the vSphere platform. It NOT a replacement for existing single-sign-on products, and is only for VMware products. The vCenter SSO service creates an authentication domain where users are trusted to access available resources. You no longer directly log into the vCenter service, rather the SSO service first authenticates you. Some products like SRM and vCOPS don’t utilize SSO today, but will in 2014.

Challenges with vSphere 5.1 SSO include:

  • Did not work effectively in multi-forest/trusted domain environments
  • Did not scale well in environments with more than 15,000 users
  • Limited administration
  • Database complexity and insecurity (lack of SQL SSL support, used SQL authentication)
  • Extraordinary complex SSL certificate replacement process (no tool at GA)
  • Difficult to change and update
  • No clear VMware deployment architecture guidance
  • Non-existent diagnostics

vSphere 5.5 SSO Rays of Sunshine

This is not a What’s new in vSphere 5.5 post, but I will focus on one key improvement that is fundamental to the installation experience in vSphere 5.5. The infamous (OEM’d RSA) SSO service was given the boot and VMware wrote a brand new SSO service from scratch. Yes, SSO is still required, but its been fundamentally re-architected. Changes include:

  • No longer requires an external database, such as SQL server
  • Built-in multi-master replication for simplified deployments (no scripting required)
  • Greatly enhanced Active Directory support (no longer treats AD as a simple LDAP server)
  • Fully supports multiple forests and two-way trusts
  • Site awareness – Only supports grouping of objects (e.g. production and DR sites) in this release…exciting roadmap though
  • Multi-tenant
  • One deployment model (very different from vSphere 5.1)
  • Full suite of MMC snap-in management and diagnostics tools
  • Backwards compatible with vSphere 5.1 (important for upgrades)
  • Simple install vCenter scales up to 1,000 hosts and 10,000 VMs
  • SSO scale is not limited by hosts/VMs, since AD lookups are offloaded to AD

So yes, VMware listened to customer feedback and really went back to the (much needed) drawing board. The real proof in the pudding will be testing it in the real world and see how many “features” (aka bugs) crop up. I was recently quoted in a TechTarget article regarding some early issues found with vSphere 5.5. KB article on the issue is here.

SSO Design Considerations


  • Your DNS infrastructure must be rock solid and fully functional. vCenter relies on Kerberos and the easiest way to break it is by bad or non-existent DNS. This is an enterprise solution, so make sure DNS is enterprise grade.
  • Must use Windows Server 2008 x64 SP2 or later (WS 2012 IS supported WS2012 R2 is NOT)
  • If natively authenticating Windows users the vCenter/SSO server(s) must be a member of the same AD domain
  • Make sure you set your vCenter administrator as administrator@vsphere.local, NOT a local OS account


  • Locate all vCenter services, including SSO, in a single VM
  • Do not install multiple SSO servers in a single site, or attempt to load balance multiple servers
  • If you have multiple vCenter servers around the world, install a local SSO server in the same authentication domain as all other vCenters
  • See “Monster Deployments” below if you have a dozen or more of vCenters

Identity Sources

There’s a nuance to “native” Active Directory support and treating AD as a simple LDAP server. SSO 5.5 only supports a single “native” Active directory domain, the one the SSO server is in. Native support is new to SSO 5.5, and addresses the complex AD topology limits in 5.1, and other related issues. Treating AD as an LDAP server brings with it all the issues of SSO 5.1, so that is not recommended. So yes a SSO server could technically use multiple AD domains/forests for authentication, but only one of them will enjoy full native capability.

  • Native Active Directory (STRONGLY recommended) – Only one native source allowed
  • Active Directory as an LDAP server (for 5.1 backwards compatibility and NOT recommended) – Multiple sources allowed
  • OpenLDAP
  • Local operating system accounts (NOT recommended)
  • Single sign-on users (replicated)

Here’s a screenshot from the Web Client Identity Source configuration screen:

10-5-2013 12-03-50 PM


  • Automatic replication between each SSO server in the same vSphere authentication domain
  • MMC snap-in allows you to review/add/remove/edit replication partners
  • Supports geographically separated SSO sites, and the ability to setup bridgehead servers
  • Each site is independent (no authentication failover)
  • Does not provide a single pane of glass view
  • Replicates SSO users and groups, SSO policies, identity sources
  • Site awareness but limited functionality in 5.5 (big futures on the roadmap)

vSphere 5.5 SSO

Monster Deployments

For service providers or monster corporations that have dozens or even 100 vCenter instaces (yes they DO exist), having 100 SSO servers all replicating is probably not ideal. For the limited use case of 6 or more vCenters connected via high speed LAN/MAN, VMware would like you to consider a dedicated SSO VM with a local web client install. All of the vCenter instances then leverage this centralized SSO service, reducing complexity and replication traffic. They hinted that on the road map are big changes to this architecture in the future, so it will be fun to see what 6.0 holds for us next year.

This architecture is NOT for a globally distributed company where vCenters are scattered all over the world. You should have local SSO servers, but they should all be a part of the same authentication domain.

9-29-2013 11-33-54 AMA


If you run vCenter as a VM (recommended) you can use the usual data protection tools including snapshots, and backup to disk and tape. VMware vDP now supports restoring directly to an ESXi host, so you could recover a vCenter VM. Like Active Directory, which is also features multi-master replication, you have to be very careful about restores.

Prior to vSphere 5.1 in combination with Windows Server 2012, USN rollback (which is extremely bad), can occur in AD if a snapshot is reverted or improper VM restore methodology was used. SSO 5.5 does not support the hypervisor GenerationID feature that Windows Server 2012 uses to protect against AD USN rollback problems. So in an environment with multiple replicating SSO 5.5 servers, you must be careful and ensure database integrity. Should you have an ‘oopsie’ moment and cause a SSO “USN rollback” like issue, the SSO database can be zeroed out and re-replicated.


The SSO service is here to stay, and it’s unavoidable if you are using vSphere 5.1 and later. VMware saw the customer pitchforks coming their way and addressed head on the major issues people had with 5.1. SSL certificates, covered in depth in an upcoming post. They are still complex and didn’t undergo major implementation changes (still need seven certs, etc.). But VMware has dramatically refined the tools (such as adding the SSO MMC snap-in), released the vCenter Certificate automation tool at GA time, and now their documentation actually matches the GA’d code. Even though much of the complexity is still there under the covers, it’s a very different world in terms of tools and documentation, which is huge.

Now that you’ve gotten a little background on the all new SSO service, I hope you are excited to see the new and improved version in action. Next up? In Part 3 learn about vCenter 5.5 upgrade best practices and tips.

VMworld 2013: vSphere 5.5 Upgrade Part 1

NOTE: I’ve started my vSphere 5.5 Install series. This is a complete How-To guide for installing, configuring, and securing your vCenter 5.5 installation. Check it out here. The information in this post has been incorporated into that series.  

Twitter: #VSVC5690. This session covered the recommended vCenter 4.x and 5.x to 5.5 upgrade paths. They quickly touched on a few new vCenter 5.5 features, such as a completely re-written SSO service, and simplified install architecture. There are supported and direct upgrade paths from vCenter 4.x through 5.1. You will be very glad to know that vCenter 5.5 does not add any new components, it’s just new and improved.

vSphere 5.5 Upgrade Overview

  • Five steps: vCenter, VUM, ESXi, VMs, VMFS
  • Review: Documentation Center, Compatibility Guide, Intero Matrix
  • Prior to 5.1: vCenter Server, vCenter Database server, and vSphere web client – Simple, life before vSphere 5.1
  • vCenter Server has SSL certificates, ADAM database – The vCenter server is not stateless
  • Web client – Third party products
  • vCenter 5.1 adds Single Sign on and Inventory Service
  • No new components added to vCenter 5.5

Upgrade Matrix

  • In-place supports vCenter 4.x, 5.0.x, 5.1.x
  • VMware does NOT support directly migrating an existing 5.x earlier vCenter Server to a new machine during the upgrade process
  • vCenter Server 5.5 can manage 4.x and 5.x hosts

System Requirements

  • Simple install – 2 vCPUs, 12GB RAM, 100GB disk
  • Strongly recommend installing ALL vCenter components on a single VM – Simplified model
  • Recommended for 400 hosts and 4000 VMs: 4 vCPU, 24GB RAM, 200GB disk
  • vCenter OS Support: Removes WS2003, only supports Windows Server 2008 SP2 and later (including WS2012)

New Install Vs. In Place Upgrade

  • New Install Pros: Clean start, reconfigure architecture, new hardware
  • New Install Cons: Loss of historical data, rebuild environment, settings manually created, time involved
  • VMware recommends a fresh install, but sometimes its not just possible
  • In Place Upgrade Pros: Most common, all settings maintained, slipstreamed process, historical data maintained
  • In Place Upgrade Cons: Carry over of old/unused data, possible performance
  • vCenter 5.5 appliance does NOT support IPv6, Linked Mode, vCenter Heartbeat, but can be a good option
  • vCenter appliance may support SQL server in the future, when Microsoft released from tech-preview the SUSE Linux ODBC connector

Installation – Then and Now

  • New Install splash screen – Simple Install is now strongly, strongly preferred
  • Simple Install should only be used for the first vCenter only
  • 5.5 Install order changed: Single Sign-On, vSphere WebClient, Inventory Service, vCenter
  • vSphere Web client is installed right after SSO to aid in troubleshooting
  • Custom install: Needed if you want to customize install location, distribute the components, or advanced config like SSO

Design Recommendations

  • Use simpler installer
  • Installs/Upgrades core components with single VMs
  • Multi-site vCenters: Single SSO domain (vsphere.local); Multi-master SSO replication
  • Each site is independent, does NOT provide a single pane of glass view
  • SSO automated replication – SSO users and groups, SSO policies, identity sources
  • Site awareness
  • Linked Mode – Maintains single pane of glass, replicates licenses, permissions and roles
  • Availability: Use vSphere HA and vCenter heartbeat

Multi-vCenter Design Recommendations (6 or more vCenters)

  • Centralized SSO authentication, same physical location
  • Single centralized vSphere web client
  • Availability (required) – vSphere HA, vCenter heartbeat, network load balancer
  • VM #1: SSO Server, Web Client; VM 2 – 6: vCenter 5.5, Inventory service

vCenter Linked Mode

  • vCenter Server linked mode groups – Does not support different versions of vCenter
  • DNS and network time are absolutely critical
  • All vCenter servers need to be a part of the same SSO domain

Host Agent Pre-Upgrade Checker

  • Perform checks include: Host is reachable, disk space is sufficient, network is functioning, file system intact, required patches installed
  • Aim is to prevent hosts going disconnected after host

vCenter Appliance

  • No migration from Windows vCenter to VCSA
  • Same license as Windows vCenter
  • Linux, no Windows
  • All in one, self contained

Update Manager

  • Can upgrade to 5.5 from 4.x and 5.x
  • Upgrades cannot change installation or download path
  • Scheduled tasks for scan and remediation are maintained
  • Patch baselines are removed
  • Use update manager utility to replace 512-bit key with 2048 key

VMware ESXi 5.1 Patches Released

VMwareHot off the presses are some ESXi 5.1 patches. This build of ESXi 5.1 (1157734) fixes several bugs and more importantly addresses some security issues. As always in any environment, please test out the patches thoroughly before putting them into production. Each environment is unique, and issues may surface that could cause you some headaches. These bug fixes aren’t earth shattering, so I would not suggest rushing them out to production systems.

ESXi 5.1 Build 1157734

Highlights of the patch bundle included in this release are:

  • Black frames might appear around text boxes in an application running on Virtual Machine Hardware Version 8 or later. This issue occurs on virtual machines with Windows 7 guest operating system and View 5.0 PCoIP.
  • For two ESXi hosts with different host names, identical machine names are generated in the domain controller under certain conditions. As a result, the ctive Directory functionality is lost for one of the two ESXi hosts.
  • After you upgrade to ESXi 5.1 from an earlier version, attempts to power on a virtual machine with static MAC address outside the allowed range (00:50:56:[00-3f] or 00:50:56:[80-BF]) fail with the following error message: The MAC address entered is not in the valid range.
  • If a physical NIC is named using non-standard naming conventions (other than vmnic#) and is added to a vSwitch, host profile creation fails with the following error message: Invalid value chosen for active NICs.
  • ESXi 5.1 hosts might get disconnected randomly from the vCenter Server system. This issue might occur if the heartbeat thread in the vpxa agent does not receive a response from the futex_wait system call. As a result, the heartbeat thread stops responding, and the vCenter Server does not receive heartbeat messages from the ESXi hosts for several hours.
  • Upon reboot, ESXi 5.1 hosts configured to obtain DNS configuration and host name from a DHCP server displays its host name as localhost in syslog rather than displaying the host name obtained from the DHCP server. As a result, for a remote syslog collector, all ESXi hosts appear to be the same, with the same host name.
  • To prevent buffer overflow, the HPSA proc node truncates LUN details on an ESXi host.
  • This patch updates the esx-base VIB to resolve a stability issue.

As always, you can down the ESXi patches from here. The full KB article for the patch bundle is here.