In this installment of the vSphere 5.5 installation how-to series we cover upgrading ESXi hosts, VMs, and VMFS. As stated in my vCenter 5.5 upgrade post, I’m not going to do a step-by-step screenshot filled posts for upgrades. Why? Too many different deployment types for that to be widely useful. But you do need to understand ESXi/VM/VMFS upgrade best practices, recommended order, and gotchas. That’s what this post is for.
SQL 2012 AlwaysOn Failover Cluster for vCenter
vSphere 5.5 Install Pt. 1: Introduction
vSphere 5.5 Install Pt. 2: SSO 5.5 Reborn
vSphere 5.5 Install Pt. 3: vCenter Upgrade Best Practices and Tips
vSphere 5.5 Install Pt. 4: ESXi 5.5 Upgrade Best Practices and Tips
vSphere 5.5 Install Pt. 5: SSL Deep Dive
vSphere 5.5 Install Pt. 6: SSL Certificate Template
vSphere 5.5 Install Pt. 7: Install SSO
vSphere 5.5 Install Pt. 8: Online SSL Minting
vSphere 5.5 Install Pt. 9: Offline SSL Minting
vSphere 5.5 Install Pt. 10: Update SSO Certificate
vSphere 5.5 Install Pt. 11: Install Web Client
vSphere 5.5 Install Pt. 12: Configure SSO
vSphere 5.5 Install Pt. 13: Install Inventory Service
vSphere 5.5 Install Pt. 14: Create Databases
vSphere 5.5 Install Pt. 15: Install vCenter
vSphere 5.5 Install Pt. 16: vCenter SSL
vSphere 5.5 Install Pt. 17: Install VUM
vSphere 5.5 Install Pt. 18: VUM SSL
vSphere 5.5 Install Pt. 19: ESXi SSL Certificate
First of all, planning is key. Even in a lab environment you want to settle on an upgrade strategy and understand the order. Order is huge! At a high level the order is:
3) ESXi hosts
But don’t just plow ahead full steam ahead and forget about things like vCenter plug-ins, VDI dependencies, backup software support, SRM, and the plethora of other VMware and third-party products. Once you get vCenter and VUM updated it is fully supported to do rolling ESXi host upgrades. Now you have to think about VM hardware versions, VM tools, and VDS configuration.
Bottom line: Think through and plan the ENTIRE upgrade before starting any part of it, including vCenter.
VIBs and Image Profiles
Understanding how VMware packages ESXi is important to better understand the upgrade path. Vendors like HP, Cisco, Dell, and others provide customized ESXi ISO media. VMware packages software (drivers, agents, etc.) as VIBs (vSphere Installation Bundle). It’s similar to a zip file or tarball. VIBs can be bundled into an ISO file (such as the ESXi installer), or as a zip depot file.
An image profile defines the VIBs which will be installed. A “standard” profile contains VMware tools and a “no-tools” profile has no VMware tools (mostly for autodeploy). You can use the image builder CLI to create a custom profile. In fact, I have a blog article here about how to build a custom ESXi ISO for Cisco UCS here.
If you want to view the VIBs on your ESXi host use the following command:
esxcli software vib list
There are many third party custom ISOs, bundles, and online depots. VMware recommends that you use a vendor customized ISO for your hardware. Some vendors are extremely timely (HP 5.5 ISO here), while others lag or nearly non-existent (Cisco). I know from personal experience the HP install ISOs are heavily customized, while the Cisco ones only have a handful of drivers. Tip: Do NOT use the HP ISO on non-HP hardware. The core software packaged on VMware ISOs and vendor ISOs is the same.
Upgrading vSphere Hosts
The big question is: Should I upgrade the host or do a fresh install? Unlike vCenter where VMware recommends to do a fresh install, if possible, they recommend upgrading ESXi hosts. You can leverage features like HA, DRS, storage vMotion, and host profiles to quickly roll through hosts. Fresh installs should be limited to a small number of hosts, maybe for test purposes. Or if you are really bored at work, then knock yourself out.
Before you upgrade check the VMware Compatibility Guide. Just because your host works with 5.0 or 5.1, does NOT mean it will work with 5.5. For example, historically HP BladeSystem has needed newer firmware to address gotchas with new ESXi builds. Don’t just blow this step off and think you have a tier-1 vendor so all is good. Likely specific firmware versions will be required/approved. Also, with 5.5 VMware removed some drivers like RealTek NICs. So if you do a fresh install you may suddenly be missing your NICs on a whitebox server. Doh!
The vSphere 5.5 release notes are quite lengthy. A number of support calls can be avoided by getting a heads up of issues. That’s why planning is so important. Get a cup of coffee or Five Hour Energy and read every issue in the release notes. It can pay dividends! The vSphere 5.5 release notes are here.
ESXi Upgrade Methods
- ESXi Installer – Boot from ISO, choose upgrade
- vSphere Update Manager – Import ISO, create upgrade baseline, remediate
- ESXCLI – Stage ZIP, execute ‘esxcli system profile update’
- Scripted Upgrades – Update/customize upgrade script
The most popular and automated method is using VUM. It will orchestrate host maintenance modes, respect DRS directives, and generally make it seamless. You can directly upgrade from ESX/ESXi 4.x and ESXi 5.x. No stairstep upgrade is needed.
Rolling upgrades within clusters are supported and highly recommended. You can mix ESX/ESXi 4.x and ESXi 5.x hosts in the same cluster. Be careful with VM hardware compatibility in such situations though. Be sure to leverage HA, DRS, vMotion and storage vMotion to enable minimal/zero downtime upgrade. If you are using Enterprise Plus, leverage host profiles. It minimizes configuration drift and enables stricter configuration control.
Upgrading ESXi Hosts
The boot disk is not re-partitioned during the upgrade process. However, the contents ARE overwritten. If there’s a VMFS datastore on the boot volume it will be preserved. Same for scratch. Absolute minimum is 1GB of space on your boot volume. Here’s a good KB on boot volume sizing. I personally use 5-6GB LUNs for boot-from-SAN configurations. The figure below shows the basic partition layout of an ESXi installation.
VMware has changed their nomenclature in how they refer to VM hardware compatibility. Previously they always called out the specific “hardware” version such as 4, 7, 9, etc. But that didn’t obviously relate to a specific release, and people got confused. Plus they thought on my gosh I’m on HW 4 and they are up 9, I’m way out of date…upgrade!
Now VMware calls out the “Compatibility” level and ties that to a release of ESXi. For example, if under the covers the VM is HW v7 it will show ESX 4.x and later in the web GUI. Do NOT feel pressure to always upgrade the compatibility level. Sometimes you need to, such as provisioning a monster VM that wasn’t supported on older versions of ESXi. But if your VM is running perfectly fine in ESX 4.x compatibility mode, you really don’t need to upgrade. I’ve fallen into the HW upgrade trap, but after hearing VMware tell us not to worry, I’ll worry less about it.
Upgrading tools and VM hardware is OPTIONAL, and VMware officially supports N-4 versions. VM hardware versions are NOT backwards compatible, though. You won’t be running HW version 10 VMs on anything but vSphere 5.5.
Important Note: Any VM’s that are only compatible with vSphere 5.5 and later (hardware version 10) can NOT be modified by the Windows VI client. No adding memory, no changing networks, nothing. This poses a problem if you want to do things like add memory to your vCenter server and hot-add is not enabled. Also if you are in an emergency situation and need to change VM properties (networking, etc.) while vCenter is down you are out of luck. While I understand the Windows VI client will probably go away entirely in vSphere 6.0, today’s situation is not optimal. Unless you are pushing the boundaries of a VM’s size and REQUIRE vHW 10, I would strongly advise to cap the VMs at vHW 9. Don’t rush into vHW 10 mode.
VMware tools is a different story,thankfully. VMware tools are backward and forward compatible to a very large degree. Don’t freak out if your VM isn’t running the latest tools. VMware recommends you DO keep up (performance, security, compliance checking, etc.), but you have wide latitude. Backup software, HA, heartbeats and other functions rely on VMware tools so if they have problems, verify the tools version matches your host. VUM is excellent for verifying compliance.
For those of you that heard starting with vSphere 5.1 that upgrading VMware tools would no longer require a reboot, that’s not actually the case. The low-down is that VMware did make changes to VMware tools to leverage Windows hot-swap of some kernel modules. However, some modules like keyboard/mouse/USB still require reboots. VMware includes those non-hot-plug modules in each tools update. So the net result is still needing to reboot when doing VMtools updates. Perhaps in the future they will change that behavior, but that’s not in 5.1 or 5.5.
VMFS upgrades are simple, and completely non-disruptive. You can upgrade a VMFS datastore from VMFS-3 to VMFS-5 with running VMs. However, while this may sound perfect, keep reading as the reality is more complicated. The table below shows the differences between the two filesystem versions.
Ok so you are thinking, why is an upgrade not ideal? The problem is that an upgraded volume does NOT look the same under the covers from a freshly formatted VMFS-5 volume. The table below shows the differences. The most impacting can be the block size. In vSphere 4.x and earlier you had a choice of block sizes that ranged from 1MB to 8MB. If your array supports VAAI extensions the VMFS volumes must have the same block size if you are doing operations such as copying VMs. Otherwise the disk operations revert back to legacy mode and will run slower.
The VMware recommendation is to create a fresh VMFS datastore then storage vMotion your VMs into the datastore. After the datastore is evacuated re-format or decommission it. If you aren’t licensed for storage vMotion, then during your vCenter upgrade don’t input a product key. This gives you 60 days of the ‘enhanced’ license features.
- Understand the vSphere Upgrade Process
- Understand how ESXi is packaged and distributed
- Understand patches vs. updates vs. upgrades
- Know the different upgrade methods
- Stay current on VMware tools
- Upgrade VM HW compatibility only when needed
- Freshly format VMFS5 volumes; don’t upgrade from VMFS3
Again, don’t feel pressure to immediately upgrade all of your VMs to hardware version 10 (vSphere 5.5 compatibility). As mentioned above, in vSphere 5.5 the only way to modify a VM that’s at HW version 10 is via the web client/vCenter. The Windows VI client will NOT let you modify VM properties. Makes it challenging to add more CPU/memory to your vCenter VM or recover from emergency situations where vCenter is down.
Next up in Part 5 is a deep dive on vCenter SSL Certificate requirements.