vSphere 5.1 U3 Now Out

Right on the heels of VMware Workstation 11 being released, VMware has released vSphere 5.1 U3. No major new features, but according to the release notes there is support for new guest operating systems (without being specific) and it also resolves a number of issues. Also updated in this release are VMware tools and the SCSI MegaRAID SAS VIB. Some security patches are also included, so be sure to start testing this release and planning your change control windows.

One interesting change in 5.1 U3, which was included in 5.5 U2, is the resetting of the CBT counter when doing a storage vMotion. Now if you do a svMotion CBT state will be maintained. New to vCenter 5.1 U3 is the support for Oracle 12c, and Microsoft SQL 2014. It’s great to see VMware keeping up with database support. vCenter 5.1 U3 also includes an updated Java engine, which addresses a plethora of security issues. So once again, view 5.1 U3 as a security update which you need to plan on rolling out in your environment.

ESXi 5.1 U3 Release notes are here

vCenter 5.1 U3 Release notes are here

As always, you can download the newest updates from My VMware portal. Be sure to conduct thorough testing in a lab environment before deploying this into product. Nutanix supports “U” releases day zero. But remembering back to NFS issues introduced in “U” releases, a good amount of testing is advised before putting this into production.



VMware ESXi 5.1 Patches Released

VMwareHot off the presses are some ESXi 5.1 patches. This build of ESXi 5.1 (1157734) fixes several bugs and more importantly addresses some security issues. As always in any environment, please test out the patches thoroughly before putting them into production. Each environment is unique, and issues may surface that could cause you some headaches. These bug fixes aren’t earth shattering, so I would not suggest rushing them out to production systems.

ESXi 5.1 Build 1157734

Highlights of the patch bundle included in this release are:

  • Black frames might appear around text boxes in an application running on Virtual Machine Hardware Version 8 or later. This issue occurs on virtual machines with Windows 7 guest operating system and View 5.0 PCoIP.
  • For two ESXi hosts with different host names, identical machine names are generated in the domain controller under certain conditions. As a result, the ctive Directory functionality is lost for one of the two ESXi hosts.
  • After you upgrade to ESXi 5.1 from an earlier version, attempts to power on a virtual machine with static MAC address outside the allowed range (00:50:56:[00-3f] or 00:50:56:[80-BF]) fail with the following error message: The MAC address entered is not in the valid range.
  • If a physical NIC is named using non-standard naming conventions (other than vmnic#) and is added to a vSwitch, host profile creation fails with the following error message: Invalid value chosen for active NICs.
  • ESXi 5.1 hosts might get disconnected randomly from the vCenter Server system. This issue might occur if the heartbeat thread in the vpxa agent does not receive a response from the futex_wait system call. As a result, the heartbeat thread stops responding, and the vCenter Server does not receive heartbeat messages from the ESXi hosts for several hours.
  • Upon reboot, ESXi 5.1 hosts configured to obtain DNS configuration and host name from a DHCP server displays its host name as localhost in syslog rather than displaying the host name obtained from the DHCP server. As a result, for a remote syslog collector, all ESXi hosts appear to be the same, with the same host name.
  • To prevent buffer overflow, the HPSA proc node truncates LUN details on an ESXi host.
  • This patch updates the esx-base VIB to resolve a stability issue.

As always, you can down the ESXi patches from here. The full KB article for the patch bundle is here.

HP ESXi 5.1 Update 1 Custom ISO Released

Just like clockwork, the day VMware releases a major update to ESXi, HP releases customized installation HP Logomedia the same day. Hard to beat such great and timely support for your VMware environment! Last week VMware released vSphere 5.1 Update 1, and also posted was the HP ESXi 5.1 update 1 custom installation ISO.

If you use HP ProLiant servers, then using the HP customized ISO is strongly recommended, or even required (Gen8), for a properly functioning host. Baked in a tested drivers and HP management tools that enable full ProLiant functionality with VMware ESXi. The HP ESXi 5.1 update 1 custom ISO is your one stop shop for an optimized ProLiant server.

Remember, as always, to keep your HP ProLiant server firmware up to date with supported versions. That is very important for ESXi hosts, as the firmware and drivers are tested together in bundles. For maximum stability, performance, and best practices you need to ensure your servers are under a supported recipe. You can download the latest HP Service Pack for ProLiant here.

HP Provider Features

  • Report installed licenses for HP Dynamic Smart Array Controller.
  • Report New memory properties.
  • Support for IP Address encoding in SNMP traps.
  • Support SMX MemoryModuleOnBoard association.
  • HP Dynamic Smart Array Controller split cache support.
  • Report New RAID levels for storage volume fault tolerance.
  • HP Smart Cache support.
  • Update reporting of Smart Array Cache Status to align with firmware and iLO.

HP AMS features

  • Report running SW processes to HP Insight Remote Support.
  • Report vSphere 5.1 U1 SNMP agent management IP and enable VMware vSphere 5.1
  • U1 SNMP agent to report iLO4 management IP.
  • IML logging for NIC, and SAS traps.
  • Limit AMS log file size and support log redirection as defined by the ESXi host parameter:
  • ScratchConfig.ConfiguredScratchLocation

SR-IOV Support

  • Updated Intel 10Gb network driver to enable SR-IOV for the HP 560FLB, 560M, 560 SFP+, and 560FLR-SFP+. (Note: ESXi Enterprise Plus is required for SR-IOV)

Other Enhancements

  • FCoE on Emulex CNAs
    To support FCoE on Emulex CNAs on vSphere 5.1 U1, HP recommends using the versions of the vSphere 5.1 U1 Emulex drivers as defined in the HP ProLiant server and option firmware and driver support recipe document available here.
  • iSCSI on Emulex
    iSCSI on Emulex is now supported on vSphere 5.1 U1 using the versions of the Emulex drivers defined in the October HP ProLiant server and option firmware and driver support recipe document available here.


HP VMware vSphere 5.1 U1 Customized Image, April 2013, Release Notes
HP VMware vSphere 5.1 U1, April 2013, Driver VIBs
April 2013 VMware Firmware and Software Recipe
HP Custom Image for ESXi 5.1 Update 1 Install CD
HP ProLiant Server VMware Support Matrix

VMware vSphere 5.1 Update 1..is it for you?

VMware vSphere 5.1 update 1 is probably one of the most recently anticipated updates of the VMware stack and it has finally hit the streets. For those of you following the release of vSphere 5.1, you have seen the GA release last fall, followed by 5.1.0a then a couple of months later 5.1.0b, all addressing bugs and ironing out critical installation issues.

VMware vSphere 5.1 Update 1 has a laundry list of improvements, support for new Microsoft products, and a lot of bug fixes. If you are still on vSphere 5.0 or tried 5.1 in the past and ran into problems, you definitely need to check out vSphere 5.1 Update 1. If you want a complete vSphere 5.1 installation guide, check out my 15-part blog series here. I will be updating it in the near future for Update 1. If you are running vSphere 5.1, there are a number of security vulnerabilities addressed in the update so start planning your upgrade.

VMwareKnown Issues with vSphere 5.1 Update 1

Today VMware posted a new KB warning about a vSphere 5.1 Update 1 bug, which may affect customers. The problem prevents you from logging into the vSphere Web Client using an AD account, if you AD account is a member of approximately 19 or more domain groups and the SSO service is configured with multiple domains. The KB states until a hotfix is released, DO NOT upgrade to vSphere 5.1 Update 1. In many enterprise environments a vSphere administrator may be in dozens of groups, depending on how access is controlled within the domain. Fewer customers will probably have SSO configured for multiple domains, so the impact of this issue is probably limited to larger enterprises. Additional issues include:

  • If you are using the vSphere Storage Appliance, you MUST upgrade to vSA 5.1.3 after you upgrade the rest of your infrastructure to vSphere 5.1 Update 1. vSA 5.1.1 is NOT compatible with vSphere 5.1 Update 1.
  • You can NOT use the simple installer to upgrade from prior 5.1 versions to 5.1 Update 1. You must utilize the individual installers.
  • Windows Server 2012 failover clusters are NOT supported on ESXi 5.1 Update 1. The cluster validation wizard gets stuck in an endless loop and you are unable for form the cluster.

What got updated in vCloud Suite 5.1 Update 1?

  • ESXi 5.1 Update 1 Build 1065491
  • vCenter Server 5.1 Update 1 Build 1065152
  • vSphere Data Protection
  • vSphere Replication 5.1.1
  • vSphere Storage Appliance 5.1.3
  • vCenter Orchestrator Appliance 5.1.1
  • vCloud Director 5.1.2
  • vCenter Site Recovery Manager 5.1.1
  • vSphere 5.1 Update 1 Virtual Disk Development Kit
  • vSphere CLI 5.1 Update 1
  • VMware Converter Standalone 5.1 (Download here)
  • VMware vCenter Server Heartbeat 6.5 Update 1
  • VMware vSphere Management Assistant ( – April 4, 2013)
  • HP Custom Image for ESXi 5.1.0 Update 1 Install CD

You can find all of these downloads in the usual place, My VMware. You can download the updated documentation archive ZIP bundle here. The full documentation page is here.

vCenter Server 5.1 Update Release Notes

vCenter 5.1 Update 1 is more than just bug and security fixes, it incorporates a number of newly supported operating systems and database back-ends. You can find the full release notes here. Below is just a tiny faction of the new features and bugfixes.

What’s New?

  • vCenter Server can be installed on Windows Server 2012
  • vCenter can use Microsoft SQL Server 2012 and SQL Server 2008 R2 SP2
  • Guest operating customization support for Windows 8, Windows Server 2012, Ubuntu 12.04 and RHEL 5.9
  • Removed vRAM usage limit of 192GB on vSphere Essentials and Essentials Plus

Resolved Issues

A lot of bug fixes are included, but a few highlights include:

  • Better error reporting when accidentally updating the Admin or STS service with incorrect protocol parameters. It will now tell you what you botched up.
  • Number of security patches including Java, tcServer, vCSA remote code vulnerability
  • Upgrade issues from 5.1.0a to 5.1.0b

VMware ESXi 5.1 Update 1 Release Notes

  • Mirrors the new guest OS support in vCenter 5.1 Update 1. Full 200+ page OS compatibility matrix is here.
  • Contains several security patches (glibc, libxslt, libxml2)
  • Resolved: Long running vMotion operations might result in unicast flooding
  • Windows Server 2012 failover clustering is not supported

You can find the ESXi 5.1 Update 1 full release notes here.

vCloud Director 5.1.2 Release Notes

Like vCenter 5.1 Update 1, vCloud Director has some new features and many resolved issues. Full release notes is here. The full vCloud Director 5.1.2 documentation set is here.

What’s new?

  • Ability to delegate creating, reverting, and removing snapshots
  • You can install vCloud Director on Red Hat Enterprise Linux 6.3
  • You can install vClould Director using Microsoft SQL Server 2012 databases
  • Supports customization of Windows Server 2012 guest operating systems

Resolved Issues

  • Security vulnerabilities addressed by updating Java to 1.6.0_37
  • Multiple bug fixes, see full release notes

vCenter Converter Standalone 5.1 Release Notes

The new version of Converter has added a number of great new features and broader operating system support. You can find the full release notes here.

  • Supports VM hardware version 9
  • Guest operating system support for Windows 8 and Windows Server 2012
  • Guest operating system support for Red Hat Enterprise Linux 6
  • Support for machine sources that use GPT partition tables
  • Support for systems that use UEFI
  • Support for EXT4 file system

vCenter Server Heartbeat 6.5 Update 1 Release Notes

No major changes here, but incremental support for the latest VMware products. Full release notes are here.

  • Support for vCenter 5.1 Update 1
  • Support for View Composer 5.2

vSphere Data Protection 5.1.20 Release Notes

More than just bug fixes, VMware added many new features to this build. Full release notes are here. A subset of the new features:

  • Integration with vCenter alarms and alerts notification system
  • Ability to clone backup jobs
  • New filters to restore tab
  • Expands capacity up to 8TB per appliance
  • Supports the ability to expand existing datastores
  • Supports guest-level backups of Microsoft SQL Servers
  • Supports guest-level backups of Microsoft Exchange Servers

vSphere Storage Appliance 5.1.2 Release Notes

Like vSphere Data Protection, the vSphere Storage Appliance has many new features. The full release notes are here.

  • Support multiple VSA clusters managed by a single vCenter Instance (about time)
  • Ability to run vCenter Server on a subnet different from the VSA cluster
  • Support for running the VSA on one of the ESXi hosts in the VSA cluster
  • Ability to install the VSA on an existing ESXi host that has running VMs
  • Ability to increase the storage capacity of a VSA cluster
  • Up to 24TB of storage per node
  • Multiple RAID types (RAID 5, RAID 6, RAID 10)


vSphere 5.1 Update 1 will be a welcomed upgrade to customers already running vSphere 5.1. After a rocky start of vSphere 5.1 GA, VMware has clearly been working on stability, bug fixes, and supporting the latest Microsoft operating systems and SQL databases. The vCloud Suite is ever expanding, so when you go to download all the components you will see over two dozen downloads you can choose from. If you’ve been hesitant to move up to vSphere 5.1, give 5.1 Update 1 a whirl in your lab and see if it’s stable enough for you.

vSphere 5.1 Suite

vCenter Certificate Automation Tool: Part 4 (Web Client and Log Browser)

Continuing on from Part 3 of the VMware vCenter Certificate Automation Tool series, we are now ready to replace the Web Client and Log Browser SSL certificates. If you want to start at the beginning, check out Part 1.

1. Pressing 5 takes us back to the main menu. Now we press 6 to enter the web client and log browser update process. Pre the pre-planning guide we need option 1. I enter the SSO administrator username and password.

Several minutes later the process was a success.

Step 11 of the pre-planning guide is complete. Check!

2. Now we need to press 2, to trust the inventory service.

Several minutes later the process was a success.

Step 12 of the pre-planning guide is complete. Check!

3. Now we need to press 3, to trust the vCenter server.

Step 13 of the pre-planning guide is complete. Check!

4. Now we need to press 4, to update the web client SSL certificate. Again, the presented paths and files were correct. Enter the SSO administrator username and password.

Step 14 of the pre-planning guide is complete. Check!

5. Next up is pressing 5, to enable the log browser service to trust SSO.

Step 15 of the pre-planning guide is complete. Check!

6. Now press 6, to update the log browser SSL certificate. Again, the certificate and paths looked good. Enter the SSO username and password.

Step 16 of the pre-planning guide is complete. Check!

At this point, since I’m using the vCenter FQDN for the VUM configuration, I am not able to use the v1.0 of this tool to update the VUM certificates. You can check out Part 12 of my vSphere 5.1 Install series for the manual method to update the VUM SSL certificate.

7. To validate some of the certificates I launch the vSphere web client. Using my web browser I view the SSL certificate and validate that my new certificate is being used. I also open the log browser and pull down the logs from an ESXi host to verify that works as well.

Minus the VUM “known issue”, the tool worked flawlessly for me and certainly helped ease the SSL burden. I’m hoping future versions of the tool have the following enhancements:

  • Automated execution of the pre-planning steps, so I don’t have to keep referring back the 18 step list and checking off each one (assuming an all-in-one server).
  • Ability to create CSRs, submit to a Microsoft online CA, and download the certificates.
  • Ability to create CSRs for an offline/commercial CA, and use the resulting certs
  • Automatically build and verify the CA chain files, to reduce human error and confusion
  • Provide a full GUI with detailed logging, to make the processes even easier
  • Perform full certificate validation to ensure unique DNs
  • Fix the VUM FQDN “known issue”
  • Back-port the tool to work with vSphere 5.0
  • Perform SQL database password validation
  • Cache in memory all required passwords (flushed upon error or exiting)
  • Configure all ODBC/JDBC SQL connection strings to use SSL (if SQL supports SSL)

I think the tool is a decent first stab at helping with the SSL configuration nightmare that 5.1 unleashed on the community. The process could be more fully automated, so I hope future versions can improve on this useful utility.

vCenter Certificate Automation Tool: Part 3 (vCenter and Orchestrator)

Continuing from Part 2 of my VMware vCenter Certificate Automation tool series, we are now ready to replace the vCenter server and vCenter Orchestrator certificates. If you want to start at the beginning, check out Part 1.

1. Per the pre-planning guide step 4 I exit back to the main menu by pressing 5, then press 4. vCenter needs to trust the SSO certificate, so I press 1. The default path and file are correct, so I press enter. Success!

Step 4 of the pre-planning guide is complete. Check!

2. From the same menu I press 2, to update the vCenter SSL certificate. Again, the default paths and files were correct so I accepted them. Now I’m prompted for the vCenter administrator name and password. Next I’m asked to enter the original vCenter server database password, with all kinds of scary warnings if I input the wrong password since no validation is done. I’m also asked to enter the SSO administrator username and password.

After several minutes of chugging away I see a successful message.

Step 5 of the pre-planning guide is complete. Check!

3. Per the pre-planning guide I now must select option 3, to trust the inventory service SSL certificate.

Step 6 of the pre-planning guide is complete. Check!

4. Pressing 5 I get back to the main menu. And I need to go back into the inventory service, so I press 3.  Finally, we now configure the inventory service to trust vCenter by pressing 2.

Step 7 of the pre-planning guide is complete. Check!

5. Pressing 5 I get back to the main menu. I now press 5, to update vCO. Per the pre-planning guide I need to configure vCO to trust SSO, so I press 1. The default SSO filename is correct so I press enter.

Step 8 of the pre-planning guide is complete. Check!

6. Now vCO needs to be told to trust vCenter server, so I press 2 and validate the path is right.

Step 9 of the pre-planning guide is complete. Check!

7. Next up is updating the vCO SSL certificate, so I press 3 and validate the path.

Step 10 of the pre-planning guide is complete. Check!

Check out Part 4 where we update the Web Client and Log Browser SSL certificates.

vCenter Certificate Automation Tool: Part 2 (SSO and Inventory)

Continuing from Part 1 of my VMware vCenter Certificate Automation tool, we are finally at the point where we can review what the built-in planner advises we do, and then replace our certificates. If you missed Part 1, go back and execute all of the steps or you have a better chance of a pig flying by your window and waiving at you than getting new SSL certificates working.

1. In case things go Tango Uniform, I strongly urge you do a full backup of all vCenter databases (SSO, vCenter, and VUM), plus snapshot/backup your vCenter VM(s). If you hose up the certificate replacement process you may be left with a smoking vCenter hole. Backup before proceeding!

2. On your vCenter server run the ssl-updater.bat script. They have a built-in planner which tells you which steps to perform and in what order, depending on what services you want to update. To access the planner type 1.

3. Since we want to update all our services, I pressed 8.

The result of pressing 8, was the following text:

1. Go to the machine with Single Sign-On installed and – Update the Single Sign-On SSL certificate.
2. Go to the machine with Inventory Service installed and – Update Inventory Service trust to Single Sign-On.
3. Go to the machine with Inventory Service installed and – Update the Inventory  Service SSL certificate.
4. Go to the machine with vCenter Server installed and – Update vCenter Server trust to Single Sign-On.
5. Go to the machine with vCenter Server installed and – Update the vCenter Server SSL certificate.
6. Go to the machine with vCenter Server installed and – Update vCenter Server trust to Inventory Service.
7. Go to the machine with Inventory Service installed and – Update the Inventory  Service trust to vCenter Server.
8. Go to the machine with vCenter Orchestrator installed and – Update vCenter Or chestrator trust to Single Sign-On.
9. Go to the machine with vCenter Orchestrator installed and – Update vCenter Or chestrator trust to vCenter Server.
10. Go to the machine with vCenter Orchestrator installed and – Update the vCenter Orchestrator SSL certificate.
11. Go to the machine with vSphere Web Client installed and – Update vSphere Web  Client trust to Single Sign-On.
12. Go to the machine with vSphere Web Client installed and – Update vSphere Web  Client trust to Inventory Service.
13. Go to the machine with vSphere Web Client installed and – Update vSphere Web  Client trust to vCenter Server.
14. Go to the machine with vSphere Web Client installed and – Update the vSphere  Web Client SSL certificate.
15. Go to the machine with Log Browser installed and – Update the Log Browser trust to Single Sign-On.
16. Go to the machine with Log Browser installed and – Update the Log Browser SSL certificate.
17. Go to the machine with vSphere Update Manager installed and – Update the vSphere Update Manager SSL certificate.
18. Go to the machine with vSphere Update Manager installed and – Update vSphere Update Manager trust to vCenter Server.

As you can see, we have to perform 18 steps to fully update all SSL certificates. Due to the “Known Issues” with VUM and using a FQDN, I shall not be performing steps 17-18 since that is not a supported configuration.

4. Getting back to the main menu by pressing 9, I now want to start updating the SSL certificates in the prescribed order per the pre-planner. So I press 2 to start with SSO.

To perform the certificate update I press 1. At this point you can opt to sacrifice a chicken over your vCenter VM to appease the SSL gods and make this go smoother.

After pressing 1 it then asks me where my SSO SSL chain file is stored. And it also wants to know where the SSO private key is, as well. Since we previously configured the environment script, the paths and files it listed were correct. I then typed in my SSO master password (you do remember it, right?). My install did not involve load balancers, so I told the installer no.

At this point the black magic starts, and my heart was thumping hoping that my chicken sacrifice worked. And a minute later….all seems to be well. Chicken worked!

Step 1 of the pre-planning guide is complete. Check!

5. Now that the SSO certificate appears to be successfully updated, it’s time to march on to the inventory service. So I press 3 to return to the main menu. On the main menu I press 3 to update the inventory service. I’m now presented with a plethora of options.

Per the pre-planning guide I need to select option 1. After 30 seconds of disk activity, I get a successful message.

Step 2 of the pre-planning guide is complete. Check! 16 left to go.

6. Slightly illogically the next step is to select option 3, per the pre-planning guide. Again, the certificate paths and files are pre-populated and are correct. Now it wants to know the SSO administrator user. If you aren’t sure what this is, open the Web Client and login. If you can access and modify the Sign-On and Discovery settings, you probably have the right username. In my case this is “sysadmin”, but it will surely be different for you.

A little whirring of my disk drive, and I get a successful message.

Step 3 of the pre-planning guide is complete. Check! 15 left to go.

Next up in Part 3 is continuing the march towards completing all 18 steps by updating the vCenter and Orchestrator certificates.

vCenter Certificate Automation Tool: Part 1 (Pre-reqs and Config)

For those of you that have installed vSphere 5.1 and tried to use your own trusted SSL certificates, you will probably find the experience extremely tedious, cumbersome, error prone, and vastly harder than any product you’ve used before. Now out is the VMware vCenter Certificate Automation tool v1.0, to make the process somewhat easier.

My 15-part vSphere 5.1 series goes into all of the gory details of the manual replacement process, and closely follows the associated VMware KB articles but in a more understandable format that people seem to appreciate. But, now the process can be made easier and less painful.

So what’s new on the vSphere 5.1 SSL front? Late last week VMware released their first stab at easing the SSL certificate replacement torture in vSphere 5.1 with a basic command line tool which helps automate the process. I covered the announcement here. Since I’m pretty familiar with the pain and suffering vSphere 5.1 certificates has caused me, I wanted to see if this tool would make life easier.

Since the process is a bit long and only semi-automated, I’ve broken down the process into a series of posts:

Part 2 (SSO and Inventory)
Part 3 (vCenter and Orchestrator)
Part 4 (Web client and Log Browser)

UPDATE 5/18/2013: I’ve included some certificate and chain.pem validation steps. The certificate tool is very picky about the formatting and contents of the chain.pem files. If you get the certs in the wrong order or have too many certs, the tool produces cryptic error messages. I strongly urge you validate your certificates and chain.pem files or you may be opening a support case with VMware.

Before you begin this process, you MUST read through all of the limitations (“Known Issues”) of the v1.0 tool. The list is pretty extensive, and there was one biggie that jumped out at me. If during the vCenter VUM installer you elect to use the server FQDN instead of the IP address (which I would argue is a best practice), then you can’t use this tool to replace the VUM certificate. Really? Ouch. The tool also doesn’t help you generate any of the certificates, or a new requirement of certificate chain files for each of the seven services. So you still have a lot of pre-work to get to the point of even trying to use the tool. This is not a fully automated end-to-end tool that is wizard driven, which we desperately need.

Going into this with my eyes wide open, and somewhat tempered expectations of what it will do for me, I decided to give it a whirl. VMware’s KB article on the tool guides you through the process, but as usual, I think the process can use a bit more elaboration and screenshots.

In my case I have a Windows Server 2012 CA, and all of the vCenter services are installed on a single Windows Server 2008 R2 VM. My vCenter databases are on an external SQL 2008 R2 VM.


Since this tool doesn’t help you create the certificate request files, generate the certificates, or the new PEM chain files we must do that prior to using the tool. I’ve updated my vCenter 5.1 U1 Installation: Part 2 (Create vCenter SSL Certificates) to address the new requirements and the addition of the Orchestrator certificate. So open up that post and follow through the entire certificate generation process, except for creating the JKS keystore. The script at the end of that post has been updated to create the newly required chained PEM files for each service. So if you’ve used that script before, grab the updated version and run it.

If you did not use my script which creates the chain.pem files, don’t worry. I’ve included steps later in this article on how to manually create them. Now that you’ve run through that long post to create all the certificate files, you should have a directory structure that looks like the screenshot below. I have these folders residing under D:\certs.

Inside each of the seven folders you should have the same set of files, as shown below with the appropriate configuration file. Again, if your chain.pem file is not present, I’ve included steps below on how to create and validate them.

You will also need the following accounts and passwords handy to complete the process:

  • SSO administrator and password
  • vCenter administrator and password
  • Original vCenter database password

Configuring the Tool

1. Download the SSL Certificate Automation Tool from My VMware.

2. Copy it to your vCenter server and unzip it to a safe place, such as D:.

3. Open the ssl-environment.bat  file and fill in all of the missing paths. In my case I set the following:

set sso_cert_chain=D:\certs\sso\chain.pem
set sso_private_key=D:\certs\sso\rui.key
set sso_node_type=single
set sso_admin_is_behind_lb=
set sso_lb_certificate= set sso_lb_hostname=

set is_cert_chain=D:\certs\inventory\chain.pem
set is_private_key_new=D:\certs\inventory\rui.key

set vc_cert_chain=D:\certs\vCenter\chain.pem
set vc_private_key=D:\certs\vCenter\rui.key

set ngc_cert_chain=D:\certs\WebClient\chain.pem
set ngc_private_key=D:\certs\WebClient\rui.key

set logbrowser_cert_chain=D:\certs\LogBrowser\chain.pem
set logbrowser_private_key=D:\certs\LogBrowser\rui.key

set vco_cert_chain=D:\certs\Orchestrator\chain.pem
set vco_private_key=D:\certs\Orchestrator\rui.key

set vum_cert_chain=D:\certs\UpdateManager\chain.pem
set vum_private_key=D:\certs\UpdateManager\rui.key

set sso_admin_user=admin@system-domain
set vc_username=contoso\svc-vctr02-001

set last_error=
set LOGS_FOLDER=%~dp0logs

4. Open an elevated command prompt and run the ssl-environment.bat script.

Creating PEM Files

If you used my script to automate the minting of your certificates then you already have the required chain.pem files. If you decided to go the more manual route, then you probably don’t have the required chain.pem files. They are easy to create, but also easy to screw up as the tool is very picky. Inside each service directory you need to have a chain.pem file. That file is comprised of the service’s certificate, intermediate CA (if you have one) and root CA certificates.

The VMware vCenter Certificate automation tool is very picky about whether your certificates are minted from a root CA or a subordinate CA. To find out which situation applies to you, open one of the certificates you’ve minted. If you only see one CA and the name of your vCenter server then then you don’t have a subordinate CA. If you see two or more CAs, then the one at the top is the root and the second one is the subordinate.

5-18-2013 3-49-48 PM

From inside each service directory you can use the following command to create the chain.pem file (assuming a subordinate CA):

copy /B rui.crt + D:\certs\root64-2.cer + D:\certs\root64-1.cer chain.pem

Repeat this process for every service directory. If you only have a root CA then the command would look like:

copy /B rui.crt + D:\certs\root64.cer chain.pem

Validate Chain.pem Files

You really should validate that the chain.pem files are properly configured. Having the certificates in the wrong order, or including an intermediate CA when your certificates are issued from a root CA will all cause the tool to #fail. Below is a truncated example of my root CA certificate file. I like to look at the last few characters of the cert, since they are very likely unique. Yours will of course be different.

Root64-1.cer (Root)

5-18-2013 4-26-36 PM

Root64-2.cer (Intermediate)

5-18-2013 4-28-46 PM

Opening one of the chain.pem files, if you are using an intermediate CA, shows that the service certificate is at the top, intermediate is in the middle, and the root is at the bottom. This is the correct order, and there must not be any blank lines before or after any certificate.

The certificate headers should also look exactly the same as below, with no extra information present. Even if your environment has an intermediate CA, if your certificates are issued from the root, do NOT include the intermediate CA. In that case you’d only have the service certificate at the top and the root at the bottom. Only include the CAs shown in your minted certificate, no more, no less.

5-18-2013 4-21-16 PM

If you jack up the order of the certs in the file you will get:

ERROR: One or more required parameters are not set or have invalid values: Certificate chain is incomplete: the root authority certificate is not present and cannot be detected automatically. The presence of the root certificate is required so the other service can establish trust to this service. Try adding the authority certificate manually.

Or if you include an intermediate CA but your certs are issued from the root:

ERROR: One or more required parameters are not set or have invalid values: The certificate chain file does not contain a valid certification path. PKIX path validation failed with: Could not validate certificate signature. (at certificate #1)

So please double check that your files are properly formatted, as the tool’s error messages are not that enlightening and don’t deal well with extra CA certificates.

Now that we have generated all of the required certificate files and set the environmental variables, we can walk through the planner and then actually replace the certificates. Replacing the SSO and Inventory SSL certificates are covered in Part 2.

VMware vCenter Certificate Automation Tool 1.0 Released

Today VMware announced their first stab at helping customers manage the SSL certificate replacement challenge that we face with vSphere 5.1: VMware vCenter Certificate Automation Tool v1.0. For anyone that has followed my 15-part series on vSphere 5.1 installation, you will know the certificate portions are quite a challenge and a source of major headaches and hair loss.

The new tool is called vCenter Certificate Automation 1.0, and will replace the certificates for:

  • vCenter Server
  • vCenter Single Sign On
  • vCenter inventory service
  • vSphere web client
  • vCenter log browser
  • VMware Update Manager
  • vCenter Orchestrator

VMware has a KB article which goes into great detail about how to use the tool and the known issues. It’s critical you read the Know Issues section, as there’s a long list of issues to be aware of. One of the biggies to me is the unsupported case of registering VUM to vCenter using the FQDN. This is standard practice in all of my configurations, so for now v1.0 of this tool won’t be a complete solution. There are also some roll-back issues as well, so just to be safe I would make sure you have a complete backup your server and related databases, in case things go sideways.

It’s great to see VMware try and ease the pain they’ve created in the methodology they’ve employed to use SSL certificates. I hope in future versions that under the covers they do some major re-work of the SSL architecture to not require such complex and tedious steps or specialized tools to implement what I consider basic modern security. The Horizon View team got certificates “right” starting with 5.0.

You can find a four part series on using the tool in the real world here. I encourage everyone to check out that series, so you can get a feeling for how the process works.

VMware Posts new vCenter SSO KB Article

For those of you trying to install and maintain vSphere 5.1, you probably will run into some vCenter SSO issues at one point or another. One of the big problems with SSO not using the very friendly Microsoft ODBC connector, is post-install reconfiguration of database parameters. The JDBC database interface is, shall we say, far from user friendly and a source of frustration.

VMware has added a new KB article, among the dozens for the SSO service, to address more issues related to changing the database configuration. Specifically, the KB article tells you how to:

  • Modify the SQL server port
  • Move the SSO database to another server
  • Modify the RSA_USER password

You can check out the full KB article here. If you haven’t dabbled with vSphere 5.1 and want a weekend of entertainment, check out my 15-part install guide series here.