Welcome to the 15-part VMware vCenter 5.1 U1 installation series! Given all of the new components and architecture of VMware vCenter 5.1, I wanted to walk through the entire installation of vCenter 5.1, assuming a greenfield environment. These instructions are assuming you are using vSphere 5.1 Update 1, which was released in April 2013. You can find more out about that release here.
vSphere 5.1 Update 1 installation is very different from the previous 5.0 and 4.x versions, so don’t think you can just click next and get a working and secure install. The GA 5.1 release of vCenter had a lot of, shall we say, bugs and issues related to the SSO service and SSL problems. 5.1.0b has addressed some of them, and now that we have vSphere 5.1 Update 1, even more issues have been resolved.
This 15-part series covers:
- Non-upgrade vCenter 5.1 Update 1 Installation on Windows Server 2012
- SSO service, Inventory Service, vCenter Server, vSphere client, vSphere Web client, and VUM
- Configure SSL certificates for all vSphere 5.1 services
- SQL SSL encryption for vCenter and VUM databases
SQL SSL encryption for the SSO database(Not working in Update 1)
Recently VMware has released their vCenter Certificate Automation Tool v1.0, which makes replacing the SSL certificates a bit easier. I’ve kept all of the manual instructions in this series, but I’ve updated some text to include references to the tool and will point out which sections you can skip should you want to use the tool.
I now recommend you install vCenter using self-signed certificates then use the vCenter Certificate Automation Tool to install your trusted certificates. You should always use trusted SSL certificates in your production environments. While the tool is not fully automated, it is easier than the manual steps or pre-staging. I wrote a complete how-to series on the tool you can find here, which you should follow after you get through the basic vCenter 5.1 installation process.
UPDATE 4/27/2013: Updating content to reflect vSphere 5.1 Update 1, and now officially supporting vCenter server on Windows server 2012 and SQL 2012 databases. I’ve also pulled the SQL SSL configuration, since I’m unable to get that working with Update 1 if the SQL server enforces encryption. Added references to the vCenter Certificate Automation tool which makes SSL certificate replacement a bit easier.
UPDATE 3/6/2013: There has been some discussion on what version of OpenSSL to use for vSphere 5.1. VMware states one should use 0.9.8. In OpenSSL 1.0 they changed the hashing algorithm, so you need to use a switch to produce the “legacy” hash of 0.9.8. -subject_hash_old is the switch you must use in OpenSSL 1.x. I get the same hashes with 0.9.8y, 1.0.1c, and 1.0.1e using that switch for the 1.x tree. In Part 3 under Update Trusted Certificate Store I show both command lines so you can produce the correct hashes. I recommend using 0.9.8, for full compatibility with the vCenter Certificate Automation tool.
UPDATE 2/17/2013: Added Part 15, replacing ESXi host SSL certificate. I also updated the Microsoft CA template creation instructions found here.
UPDATE 12/20/2012: VMware has released vCenter 5.1.0b, which addresses yet more installation bugs. I would advise skipping 5.1.0a, and go straight to 5.1.0b. You can find the 5.1.0b release notes here. I currently do not have time to update all the posts with 5.1.0b information.
This is the first post in a series of blog articles about configuring vSphere 5.1.0 Update 1. Links to the other parts are below:
Part 2 (Create vCenter SSL Certificate)
Part 3 (Install SSO Service SSL Certificate)
Part 4 (Install Inventory Service)
Part 5 (Install Inventory Service SSL Certificate)
Part 6 (Create vCenter and VUM Databases)
Part 7 (Install vCenter Server)
Part 8 (Install Web Client)
Part 9 (Optional SSO Configuration)
Part 10 (Create VUM DSN)
Part 11 (Install VUM)
Part 12 (VUM SSL Configuration)
Part 13 (VUM Configuration)
Part 14 (Web Client and Log Browser SSL)
Part 15 (ESXi Host SSL Certificate)
Note: You can skip parts 3, 5, 14 if you want to use the vCenter Certificate automation tool as mentioned above.
SQL Server Pre-Reqs
With prior versions of vCenter one could easily configure their SQL server and ODBC connection to use SSL. This encrypted all communications between vCenter and the SQL server, which is a great best practice. However, in vCenter 5.1 the SSO service uses a JDBC connector, which I have not been able to reliably configure with SSL.
If your SQL server is forcing SQL SSL encryption, then you won’t get past the SSO installer as it will fail. You can validate your SQL server configuration by looking in the SQL Server Configuration Manager on your SQL server and reviewing the properties of the Protocols for MSSQLSERVER. As shown below, if Force Encryption is set to Yes you will need to change it to NO and restart the SQL services.
On another security note the SQL server MUST be configured to allow both Windows integrated authentication AND SQL authentication. SQL authentication is very weak, which makes the use of SSL for the database connection that much more imperative. Should the SQL server only allow Windows integrated authentication you will likely get the following error:
Error 29115.Cannot authenticate to DB.
Use SQL studio to login to your SQL server, open the server properties then use the less secure option of SQL Server and Windows Authentication mode. Restart the SQL services.
vCenter 5.1 Installation – VM Provisioning
1. Provision one or more VMs for the vCenter 5.1 install. In this blog series I’m assuming an all-in-one server to make things easier. You can certainly split up the services, which would be recommended in medium to large environments.
I provisioned a Windows Server 2012 VM (2008 R2 will work fine as well), with 2 HDs, and all of the latest Windows updates. 6GB of RAM and even 8GB of RAM for an all-in-one server is recommended, otherwise vCenter and SSO will run verrrrrrrry slowly. The 5.1 release has high memory utilization.
2. Create a domain-based service account (e.g. SVC-VCTR02-001) which the vCenter services will use. Add that account to the local Administrator’s group on what will become the vCenter 5.1 server.
You need to ensure the service account also has the “Act as part of the operating system” user right on the vCenter server. If the Administrators group has the right then you are covered. If not, explicitly add the service account to the user right as shown below.
3. Open the Server Manager and add the .NET Framework 3.5 feature and wait for the install to complete.
Windows Server 2008 R2:
Windows Server 2012:
Configure SQL Database
SQL dynamic ports were NOT supported in pre-Update 1 builds (GA, 5.1.0a, 5.1.0b). The release notes for Update 1 state that problem has been fixed. The Update 1 SSO install the wizard has been modified to support dynamic ports for SQL server instances. Remember, VMware still does not officially support clustered SQL servers. They will provide best effort services if you run into issues, but it’s not a validated configuration.
The SSO service requires a database, as do other vCenter services. In this example we are using SQL Server 2012, but 2008 R2 SP2 is perfectly fine as well. Prior to Update 1 SQL Server 2012 was NOT supported, so don’t try it unless you are on vSphere 5.1 Update 1. There are some hard coded restrictions in the SSO service which limit your ability to use customized names for all of the fields. In particular the DB name must only include letters, numbers, underscore (_), the at symbol (@) and the hash (#). No periods and no spaces. As of the 5.1.0b release, hyphens are now allowed though.
As a reader has pointed out, you should be using SQL Server 2008 R2 SP1 and CU6 or later (Build 10.50.2811), which addresses a JDBC issue. You can read the MS KB here. I used SQL Server 2012 in my test environment, since that’s now supported as of vSphere 5.1 Update 1.
Unfortunately, I’m not able to get a SQL SSL configuration that actually works. So I’ve pulled the SSL steps, which were in the last version of this article. Be sure to set passwords on the SQL accounts that meet Windows GPO password complexity and minimum length requirements.
SQL DB Configuration Steps
1. I created a custom database creation script below, based on the canned VMware script included in the installation ISO. In my case I called the database “D001_VMware_SSO”. Run this script in SQL Server Management Studio, modified to your liking. Note that you CAN NOT change “RSA_DATA” or “RSA_INDEX” as the SSO service is hard coded to use them and the install WILL fail if they are not present. The VMware script has auto_shrink enabled, which DBAs tell me is a bad idea. So that is not present in my script below.
USE MASTER GO CREATE DATABASE D001_VMware_SSO ON PRIMARY( NAME='RSA_DATA', FILENAME='K:\Microsoft SQL Server\MSSQL\Data\D001_VMware_SSO_Data.mdf', SIZE=10MB, MAXSIZE=UNLIMITED, FILEGROWTH=10%), FILEGROUP RSA_INDEX( NAME='RSA_INDEX', FILENAME='K:\Microsoft SQL Server\MSSQL\Data\D001_VMware_SSO_Index.mdf', SIZE=10MB, MAXSIZE=UNLIMITED, FILEGROWTH=10%) LOG ON( NAME='translog', FILENAME='L:\Microsoft SQL Server\MSSQL\Data\Logs\D001_VMware_SSO_Log.ldf', SIZE=10MB, MAXSIZE=UNLIMITED, FILEGROWTH=10% ) GO ALTER DATABASE [D001_VMware_SSO] SET RECOVERY SIMPLE GO CHECKPOINT GO
USE MASTER GO CREATE LOGIN RSA_DBA WITH PASSWORD = 'Your Password', DEFAULT_DATABASE = D001_VMware_SSO GO CREATE LOGIN RSA_USER WITH PASSWORD = 'Your Password', DEFAULT_DATABASE = D001_VMware_SSO GO USE D001_VMware_SSO GO ALTER AUTHORIZATION ON DATABASE::D001_VMware_SSO TO RSA_DBA GO CREATE USER RSA_USER FOR LOGIN RSA_USER GO CHECKPOINT GO
1. Login as the newly created vCenter service account and launch the vSphere installer from the ISO image and you are presented with the following screen.
At this point VMware gives you the option of a “Simple Install” or install each component separately. Since we want to replace SSL certificates in an orderly fashion and in the easiest possible manner, so do NOT select Simple Install. We want to deliberately install each service and perform configuration steps along the way.
2. Click on vCenter Single Sign On, then click on Install. Select the appropriate language and wait for the wizard to open. After clicking through the licensing agreements and carefully reading all of the patents, you are presented with a screen with several options.
VMware gives you the option to install multiple instances of the SSO service for high availability. So on the screen below you have the option of creating a new primary node instance, or join an existing SSO instance. Since this is a new deployment, we want to create a primary node.
Even if you don’t want multiple SSO instances now, you may want them in the future. You don’t need to configure additional ones from the outset, so there’s no harm in leaving the door open for future expansion. Thus I selected the second option, as shown below. .
3. Next the installer will prompt you for the password to the default SSO Administrator account. Yes, this is a local account not tied to AD or the Windows host. After SSO is installed, you can configure it for one or more LDAP/AD server and other identity sources, so don’t fret too much about this application password but DO remember it.
The password must have at least eight characters, at least one lowercase character, one uppercase character, one number, and one special character. Maximum password length is 32 characters. Passwords longer than 32 characters will be truncated and cause authentication problems. The password also MUST meet local OS and AD domain length and complexity requirements. Password failures can cause the following SSO installation error:
Error 32010. Failed to create database users. There can be several reasons
for this failure. For more information, see the vmMSSQLCmd.log file in the
system temporary folder.
Note: Do NOT use the following characters, or trailing spaces:
” (double quote)
‘ (single quote)
) (right parenthesis)
< (less than)
> (greater than)
These may cause a “Error 29133.Administrator login error.” further on in the installation process. VMware has a KB article regarding these special characters here.
4. At this point you are presented with a dialog asking what kind of database you want to use. I would never use SQL Express in a lab or production environment, so select the second option.
Click on Next, and if everything is validated, no errors will appear.
8. For the installation path I left the default, as the installer has had problems in the path with custom paths or “unusual” characters in the path.
9. On the next screen I left the HTTPS port the default, then sent the installer off on its merry way.
At this point the vCenter Single Sign On service should have successfully installed. Next up is creating all of the SSL certificates that the vCenter services require. You can check out Part 2 here.