vSphere 5.5 Install Pt. 4: ESXi 5.5 Upgrade

ESXi 5.5 upgradeIn this installment of the vSphere 5.5 installation how-to series we cover upgrading ESXi hosts, VMs, and VMFS. As stated in my vCenter 5.5 upgrade post, I’m not going to do a step-by-step screenshot filled posts for upgrades. Why? Too many different deployment types for that to be widely useful. But you do need to understand ESXi/VM/VMFS upgrade best practices, recommended order, and gotchas. That’s what this post is for.

Blog Series

SQL 2012 AlwaysOn Failover Cluster for vCenter
vSphere 5.5 Install Pt. 1: Introduction
vSphere 5.5 Install Pt. 2: SSO 5.5 Reborn 
vSphere 5.5 Install Pt. 3: vCenter Upgrade Best Practices and Tips
vSphere 5.5 Install Pt. 4: ESXi 5.5 Upgrade Best Practices and Tips
vSphere 5.5 Install Pt. 5: SSL Deep Dive
vSphere 5.5 Install Pt. 6: SSL Certificate Template
vSphere 5.5 Install Pt. 7: Install SSO
vSphere 5.5 Install Pt. 8: Online SSL Minting
vSphere 5.5 Install Pt. 9: Offline SSL Minting
vSphere 5.5 Install Pt. 10: Update SSO Certificate
vSphere 5.5 Install Pt. 11: Install Web Client 
vSphere 5.5 Install Pt. 12: Configure SSO
vSphere 5.5 Install Pt. 13: Install Inventory Service
vSphere 5.5 Install Pt. 14: Create Databases
vSphere 5.5 Install Pt. 15: Install vCenter
vSphere 5.5 Install Pt. 16: vCenter SSL
vSphere 5.5 Install Pt. 17: Install VUM
vSphere 5.5 Install Pt. 18: VUM SSL
vSphere 5.5 Install Pt. 19: ESXi SSL Certificate

Permalink to this series: vexpert.me/Derek55
Permalink to the Toolkit script: vexpert.me/toolkit55

Upgrade Overview

First of all, planning is key. Even in a lab environment you want to settle on an upgrade strategy and understand the order. Order is huge!  At a high level the order is:

1) vCenter
2) VUM
3) ESXi hosts
4) VMs

But don’t just plow ahead full steam ahead and forget about things like vCenter plug-ins, VDI dependencies, backup software support, SRM, and the plethora of other VMware and third-party products. Once you get vCenter and VUM updated it is fully supported to do rolling ESXi host upgrades. Now you have to think about VM hardware versions, VM tools, and VDS configuration.

Bottom line: Think through and plan the ENTIRE upgrade before starting any part of it, including vCenter.

VIBs and Image Profiles

Understanding how VMware packages ESXi is important to better understand the upgrade path. Vendors like HP, Cisco, Dell, and others provide customized ESXi ISO media. VMware packages software (drivers, agents, etc.) as VIBs (vSphere Installation Bundle). It’s similar to a zip file or tarball. VIBs can be bundled into an ISO file (such as the ESXi installer), or as a zip depot file.

An image profile defines the VIBs which will be installed. A “standard” profile contains VMware tools and a “no-tools” profile has no VMware tools (mostly for autodeploy). You can use the image builder CLI to create a custom profile. In fact, I have a blog article here about how to build a custom ESXi ISO for Cisco UCS here.

9-29-2013 2-45-06 PM

If you want to view the VIBs on your ESXi host use the following command:

esxcli software vib list

There are many third party custom ISOs, bundles, and online depots. VMware recommends that you use a vendor customized ISO for your hardware. Some vendors are extremely timely (HP 5.5 ISO here), while others lag or nearly non-existent (Cisco). I know from personal experience the HP install ISOs are heavily customized, while the Cisco ones only have a handful of drivers. Tip: Do NOT use the HP ISO on non-HP hardware. The core software packaged on VMware ISOs and vendor ISOs is the same.

Upgrading vSphere Hosts

The big question is: Should I upgrade the host or do a fresh install? Unlike vCenter where VMware recommends to do a fresh install, if possible, they recommend upgrading ESXi hosts. You can leverage features like HA, DRS, storage vMotion, and host profiles to quickly roll through hosts. Fresh installs should be limited to a small number of hosts, maybe for test purposes. Or if you are really bored at work, then knock yourself out.

Before you upgrade check the VMware Compatibility Guide. Just because your host works with 5.0 or 5.1, does NOT mean it will work with 5.5. For example, historically HP BladeSystem has needed newer firmware to address gotchas with new ESXi builds. Don’t just blow this step off and think you have a tier-1 vendor so all is good. Likely specific firmware versions will be required/approved. Also, with 5.5 VMware removed some drivers like RealTek NICs. So if you do a fresh install you may suddenly be missing your NICs on a whitebox server. Doh!

Release Notes

The vSphere 5.5 release notes are quite lengthy. A number of support calls can be avoided by getting a heads up of issues. That’s why planning is so important. Get a cup of coffee or Five Hour Energy and read every issue in the release notes. It can pay dividends! The vSphere 5.5 release notes are here.

ESXi Upgrade Methods

  • ESXi Installer – Boot from ISO, choose upgrade
  • vSphere Update Manager – Import ISO, create upgrade baseline, remediate
  • ESXCLI – Stage ZIP, execute ‘esxcli system profile update’
  • Scripted Upgrades – Update/customize upgrade script

The most popular and automated method is using VUM. It will orchestrate host maintenance modes, respect DRS directives, and generally make it seamless. You can directly upgrade from ESX/ESXi 4.x and ESXi 5.x. No stairstep upgrade is needed.

Upgrading Clusters

Rolling upgrades within clusters are supported and highly recommended. You can mix ESX/ESXi 4.x and ESXi 5.x hosts in the same cluster. Be careful with VM hardware compatibility in such situations though. Be sure to leverage HA, DRS, vMotion and storage vMotion to enable minimal/zero downtime upgrade. If you are using Enterprise Plus, leverage host profiles. It minimizes configuration drift and enables stricter configuration control.

Upgrading ESXi Hosts

The boot disk is not re-partitioned during the upgrade process. However, the contents ARE overwritten. If there’s a VMFS datastore on the boot volume it will be preserved. Same for scratch. Absolute minimum is 1GB of space on your boot volume. Here’s a good KB on boot volume sizing. I personally use 5-6GB LUNs for boot-from-SAN configurations. The figure below shows the basic partition layout of an ESXi installation.

9-29-2013 3-42-30 PM

VM Upgrades

VMware has changed their nomenclature in how they refer to VM hardware compatibility. Previously they always called out the specific “hardware” version such as 4, 7, 9, etc. But that didn’t obviously relate to a specific release, and people got confused. Plus they thought on my gosh I’m on HW 4 and they are up 9, I’m way out of date…upgrade!

Now VMware calls out the “Compatibility” level and ties that to a release of ESXi. For example, if under the covers the VM is HW v7 it will show ESX 4.x and later in the web GUI. Do NOT feel pressure to always upgrade the compatibility level. Sometimes you need to, such as provisioning a monster VM that wasn’t supported on older versions of ESXi. But if your VM is running perfectly fine in ESX 4.x compatibility mode, you really don’t need to upgrade. I’ve fallen into the HW upgrade trap, but after hearing VMware tell us not to worry, I’ll worry less about it.

9-29-2013 3-49-24 PM

Upgrading tools and VM hardware is OPTIONAL, and VMware officially supports N-4 versions. VM hardware versions are NOT backwards compatible, though. You won’t be running HW version 10 VMs on anything but vSphere 5.5.

Important Note: Any VM’s that are only compatible with vSphere 5.5 and later (hardware version 10) can NOT be modified by the Windows VI client. No adding memory, no changing networks, nothing. This poses a problem if you want to do things like add memory to your vCenter server and hot-add is not enabled. Also if you are in an emergency situation and need to change VM properties (networking, etc.) while vCenter is down you are out of luck. While I understand the Windows VI client will probably go away entirely in vSphere 6.0, today’s situation is not optimal. Unless you are pushing the boundaries of a VM’s size and REQUIRE vHW 10, I would strongly advise to cap the VMs at vHW 9. Don’t rush into vHW 10 mode.

VMware tools is a different story,thankfully. VMware tools are backward and forward compatible to a very large degree. Don’t freak out if your VM isn’t running the latest tools. VMware recommends you DO keep up (performance, security, compliance checking, etc.), but you have wide latitude. Backup software, HA, heartbeats and other functions rely on VMware tools so if they have problems, verify the tools version matches your host. VUM is excellent for verifying compliance.

9-29-2013 3-55-59 PM

For those of you that heard starting with vSphere 5.1 that upgrading VMware tools would no longer require a reboot, that’s not actually the case. The low-down is that VMware did make changes to VMware tools to leverage Windows hot-swap of some kernel modules. However, some modules like keyboard/mouse/USB still require reboots. VMware includes those non-hot-plug modules in each tools update. So the net result is still needing to reboot when doing VMtools updates. Perhaps in the future they will change that behavior, but that’s not in 5.1 or 5.5.

VMFS Upgrades

VMFS upgrades are simple, and completely non-disruptive. You can upgrade a VMFS datastore from VMFS-3 to VMFS-5 with running VMs. However, while this may sound perfect, keep reading as the reality is more complicated. The table below shows the differences between the two filesystem versions.

9-29-2013 4-02-44 PM

Ok so you are thinking, why is an upgrade not ideal? The problem is that an upgraded volume does NOT look the same under the covers from a freshly formatted VMFS-5 volume. The table below shows the differences. The most impacting can be the block size. In vSphere 4.x and earlier you had a choice of block sizes that ranged from 1MB to 8MB. If your array supports VAAI extensions the VMFS volumes must have the same block size if you are doing operations such as copying VMs. Otherwise the disk operations revert back to legacy mode and will run slower.

9-29-2013 4-05-15 PM

The VMware recommendation is to create a fresh VMFS datastore then storage vMotion your VMs into the datastore. After the datastore is evacuated re-format or decommission it. If you aren’t licensed for storage vMotion, then during your vCenter upgrade don’t input a product key. This gives you 60 days of the ‘enhanced’ license features.


  • Understand the vSphere Upgrade Process
  • Understand how ESXi is packaged and distributed
  • Understand patches vs. updates vs. upgrades
  • Know the different upgrade methods
  • Stay current on VMware tools
  • Upgrade VM HW compatibility only when needed
  • Freshly format VMFS5 volumes; don’t upgrade from VMFS3

Again, don’t feel pressure to immediately upgrade all of your VMs to hardware version 10 (vSphere 5.5 compatibility). As mentioned above, in vSphere 5.5 the only way to modify a VM that’s at HW version 10 is via the web client/vCenter. The Windows VI client will NOT let you modify VM properties. Makes it challenging to add more CPU/memory to your vCenter VM or recover from emergency situations where vCenter is down.

Next up in Part 5 is a deep dive on vCenter SSL Certificate requirements.

vSphere 5.5 Install Pt. 1: Introduction

9-29-2013 5-00-14 PMAt VMworld 2013 in San Francisco VMware unveiled vSphere 5.5, the successor to vSphere 5.1. Customers are now chomping at the bit for vSphere 5.5 upgrade information. Of great anticipation was what VMware did to the vSphere 5.1 SSO service to address common complaints.

If you’ve been living under a rock, or haven’t tried vSphere 5.1, you’ve been missing out. While vSphere 5.1 brought a great number of new features to the table, it also brought a wee bit of frustration surrounding the new SSO (Single Sign-On) service and SSL certificates.

Exactly one year later we now have vSphere 5.5 dropped in our lap. Last year I put countless hours into writing a 15-part vCenter 5.1 install series, which earned me the nickname “SSL Guy” at VMworld. The amount of traffic that series continues to get floors me, and I’m glad the community has found it useful.

This year I will endeavor to one up myself, and do a better job with 5.5. Now that SSO and SSL are bit better understood (by myself and VMware), and massively improved, I shouldn’t have to revise the articles on a weekly basis like I did for the 5.1 series. I hope this series proves even clearer, more accurate, and fills in the gaps that VMware documentation sometimes has.

I’ve created a shortened permalink that you can use for quick reference: vexpert.me/Derek55 for this series. Feel free to use however you like…PowerPoint slides, email, etc. If you find this series helpful, please spread the word.

UPDATE November 3, 2013: VMware released vCenter 5.5a, to correct some bugs with SSO and other services. Please use the 5.5a media and not the 5.5 GA media. Find out more here.

Series Agenda

The exact number of installments and what I’ll cover is a bit fluid at the moment. But at a minimum it will cover the following topics:

  • Upgrade or fresh install?
  • Deep dive on what’s new in SSO 5.5
  • vCenter upgrade best practices and tips
  • ESXi upgrade best practices and tips
  • Right sizing your WS2012 vCenter VM (think big)
  • Creating vCenter SSL certificates (remember the 5.1 days)
  • VMware vCenter Certificate Automation Tool (hair loss reduction tool)
  • Manual SSL certificate replacement process
  • Using a SQL 2012 AlwaysOn Failover Cluster for the vCenter database
  • Installing the full vCenter stack of software on Windows Server 2012 (not R2)
  • Configuring VUM
  • ESXi host SSL certificate replacement
  • Deploying the vCenter Server Appliance (VCSA)
  • ..and possibly more…such as VSAN or vFlash Read Cache

I am fully anticipating the vSphere 5.5 installation will be easier, quicker, and cause less hair loss. If you are lucky it might even re-grow hair lost during your vSphere 5.1 deployment. I shall report my results after this series is complete.

While I have two entire blog posts dedicated to upgrade best practices and tips, the step-by-step instructions will assume a fresh install. This is the VMware recommended approach, but doesn’t work for everyone. Upgrade how-to’s are not very valuable, IMHO, since customer configurations will wildly vary. This is particularly true with SSO 5.1 and the many deployment options, coupled with little VMware best practices around SSO 5.1.

Database Support

VMware now officially supports SQL 2012 AlwaysOn failover clusters (using shared storage) for the vCenter database. It does NOT support AlwaysOn Availability groups or database mirroring. To that end I recently wrote a soup to nuts guide (12 parts) on how to install a SQL 2012 Failover Cluster on Windows Server 2012. If that’s something you want to do, you can dive head first into that while waiting on me to post the next vCenter installation installments. Many of you may not be clustering experts, so it should be enough to get you all the way up, with a ton of best practices incorporated. Here’s a quick reference chart for all of the SQL 2012 HA/DR options.

9-29-2013 5-44-04 PM

Best Practices Video

You can also check out this 20 minute video from VMware on vCenter 5.5 best practices. There’s also an accompanying KB article that has some good details as well. You can find that here.

Derek’s Toolkit Script

This year I’m doing something a little different to hopefully make life easier for you all. Last year in the vCenter 5.1 series I had numerous scripts and configuration files to help you through the SSL replacement process. While those worked, it still wasn’t as clear cut and easy to use as I would have liked.

So this year I’ve written a PowerShell script that I cover in-depth in Part 8, which takes most of the pain away in creating your certificate requests and making the files the VMware certificate automation tool needs. As I go through the series it will also do tasks like creating your ODBC connectors. The script will be updated on a regular basis. If there are any PowerShell experts out there that have ideas for making it better, I’m all ears. A sample screenshot of v1.56 is below.

1-11-2014 2-27-28 PM

You can also download the latest version at: vexpert.me/toolkit55


As I add new installments to the series this landing page will be updated with links to each part. The flow will be somewhat different than last year, due to the re-ordering of components and some SSL lessons learned. If you have other areas that you think I should cover, please leave a comment or Tweet me. I’m very active on Twitter, so if you use that medium, be sure to follow me @vDerekS. I’ll tweet about new posts as they get published. If you like the posts, sharing on social media and with co-workers is appreciated.

Feedback is always welcome, so leave comments about your experiences. This can help other people that may have the same problem. The 5.1 series literally had hundreds of reader comments. Some of which had to be censored due to expressed high frustration with 5.1.

One last comment…and I can’t stress this enough. You must, must, must read the vSphere 5.5 release notes. You can find the long document here.

Blog Series

SQL 2012 AlwaysOn Failover Cluster for vCenter
vSphere 5.5 Install Pt. 1: Introduction
vSphere 5.5 Install Pt. 2: SSO Reborn
vSphere 5.5 Install Pt. 3: vCenter Upgrade Best Practices and Tips
vSphere 5.5 Install Pt. 4: ESXi Upgrade Best Practices and Tips
vSphere 5.5 Install Pt. 5: SSL Deep Dive
vSphere 5.5 Install Pt. 6: SSL Certificate Template
vSphere 5.5 Install Pt. 7: Install SSO
vSphere 5.5 Install Pt. 8: Online SSL Minting
vSphere 5.5 Install Pt. 9: Offline SSL Minting
vSphere 5.5 Install Pt. 10: Update SSO Certificate
vSphere 5.5 Install Pt. 11: Install Web Client 
vSphere 5.5 Install Pt. 12: Configure SSO
vSphere 5.5 Install Pt. 13: Install Inventory Service
vSphere 5.5 Install Pt. 14: Create Databases
vSphere 5.5 Install Pt. 15: Install vCenter
vSphere 5.5 Install Pt. 16: vCenter SSL
vSphere 5.5 Install Pt. 17: Install VUM
vSphere 5.5 Install Pt. 18: VUM SSL
vSphere 5.5 Install Pt. 19: ESXi SSL Certificate

Permalink to this series: vexpert.me/Derek55
Permalink to the Toolkit script: vexpert.me/toolkit55

VMworld 2013: What’s new in vSphere 5.5


This session was a fire hose of the top vSphere 5.5 features. There’s a lot that’s new in this release, and they’ve addressed many of the vSphere 5.1 SSO headaches. So if you skipped vSphere 5.1 (like I did) for production environments, then get ready for the vSphere 5.5 train and jump on board. This is a release that you won’t want to miss. Also learn why vCloud Director will be going the way of the Windows c# vSphere client (hint, think dodo bird).

Cloud Management Offerings

  • vSphere with Operations Management – new SKU in March 2013 -vSOM Enterprise + is $4245 per socket
  • vCloud Suite per CPU: Enterprise Plus is $11,495
  • Operations management – A large customer found 90% of VMs were over provisioned

What’s new in vSphere 5.5


  • vSphere Big Data Extensions – Optimize Hadoop workloads and extend project Serengeti
  • Pivotal and VMware vSphere – Building PaaS on-Prem
  • Latest chip set support – Intel E5 V2, Intel Atom C2000
  • OpenStack – Delivering architecture choices

Performance and Scale

  • 2x in configuration maximums
  • Up to 62TB VMDKs
  • Low latency application configuration 31% latency improvement
  • 320 pCPUs, 4TB RAM, 16 NUMA nodes, 4096 vCPUs
  • 4GB ESXi minimum RAM (e.g. for labs)

vSphere App HA

  • Detect and recover from application or OS failure
  • Supports most common packages apps (Exchange, SQL, Oracle, SharePoint, etc.)
  • vCloud Extensibility – APIs and ecosystem
  • Deployed as two virtual appliances
  • Tier 1 application protection at scale

vSphere Flash Read Cache

  • Virtualized flash resource managed just like CPU and memory
  • Per-VM hypervisor based read caching using server flash
  • Compatible with vMotion, DRS and HA
  • Accelerates performance for mission critical apps by up to 2x
  • Enables efficient use of server flash in virtual environments
  • Fully transparent to VMs

vSphere Big Data Extensions

  • Elastic scaling
  • Easy to use interface
  • Enhanced HA/FT leveraging vSphere
  • Higher cluster utilization

vSphere Replication

  • Still 15 minute RPO
  • Multiple point in time copies
  • Multiple replication appliances per vCenter
  • Support storage vMotion and storage DRS

vSphere Data Protection

  • 4x greater scalability – Advanced SKU (more $$)
  • Agent-based application awareness of Exchange and SQL – Advanced SKU only (extra $$)
  • Direct recovery – can recover VMs without vCenter
  • Restore individual VMDKs
  • Can restore with a different VADP appliance
  • 6x faster recovery
  • 4x more storage efficient
  • Managed from vSphere web client

vCenter Server 5.5

  • SSO: Improved user experience. SSO no longer requires SQL database.
  • vCenter Appliance supports 500 vSphere hosts and 5000 VMs
  • vCenter Databases – Official support for database clustering – Oracle RAC, SQL cluster
  • Added support for OS X vSphere web client
  • VM console access, deploy OVF templates
  • Drag and drop

Best of the Rest

  • Hardware version 10
  • MSCS support enhancements
  • VMFS heap enhancements
  • Enhanced LACP support
  • Enhanced SR-IOV
  • QoS tagging
  • Packet capture
  • 40G support
  • Support “reliable memory”
  • Hot-plug SSD PCie Devices
  • Expanded vGPU and GP-GPU support

License SKUs

  • Enterprise: Adds big data extensions and reliable memory
  • Enterprise Plus: Flash read cache and App HA

vSphere 5.5 Support Lifecycle

  • Normal 5 year support would end 2016 (based on vSphere 5 starting in 2011)
  • Support will be extended to 2018
  • Only applies to ESXi and vCenter 5.5

Reduce Complexity

  • vCloud Director is GOING AWAY post vSphere 5.5. Functionality migrated to vCAC and the virtualization platform
  • vCloud Automation Center – vCAC
  • vCloud director will also have extended support period like vSphere 5.5