In this installment of the vSphere 6.0 installation how-to series we cover upgrading ESXi hosts, VMs, and VMFS. You do need to understand ESXi/VM/VMFS upgrade best practices, recommended order, and gotchas. That’s what this post is for.
vSphere 6.0 Install Pt. 1: Introduction
vSphere 6.0 Install Pt. 2: Platform Services Controller
vSphere 6.0 Install Pt. 3: Certificate Management
vSphere 6.0 Install Pt. 4: vCenter Upgrade Best Practices
vSphere 6.0 Install Pt. 5: ESXi Upgrade Best Practices
vSphere 6.0 Install Pt. 6: Install Windows PSC
vSphere 6.0 Install Pt. 7: Config SQL DBs
vSphere 6.0 Install Pt. 8: Toolkit Configuration
vSphere 6.0 Install Pt. 9: SSL Templates
vSphere 6.0 Install Pt. 10: Install VCSA PSC
vSphere 6.0 Install Pt. 11: VMCA as Subordinate
vSphere 6.0 Install Pt. 12: PSC Machine Certificate
vSphere 6.0 Install Pt. 13: Directory Services Certificate
vSphere 6.0 Install Pt. 14: Windows vCenter Install
First of all, planning is key. Even in a lab environment you want to settle on an upgrade strategy and understand the order. Order is huge! If you are running the basic vSphere stack and no other products like SRM, vCAC, etc, the order looks like this:
3) ESXi hosts
But don’t just plow ahead full steam ahead and forget about things like vCenter plug-ins, VDI dependencies, backup software support, SRM, and the plethora of other VMware and third-party products. Once you get vCenter and VUM updated it is fully supported to do rolling ESXi host upgrades. Now you have to think about VM hardware versions, VM tools, and VDS configuration. For a great summary of the upgrade order if your use other VMware products check out this KB.
Bottom line: Think through and plan the ENTIRE upgrade before starting any part of it, including vCenter. Many times third party products like backup software can lag significantly in vSphere support. So you may be waiting a while before you can upgrade.
VIBs and Image Profiles
Understanding how VMware packages ESXi is important to better understand the upgrade path. Vendors like HP, Cisco, Dell, and others provide customized ESXi ISO media. VMware packages software (drivers, agents, etc.) as VIBs (vSphere Installation Bundle). It’s similar to a zip file or tarball. VIBs can be bundled into an ISO file (such as the ESXi installer), or as a zip depot file.
An image profile defines the VIBs which will be installed. A “standard” profile contains VMware tools and a “no-tools” profile has no VMware tools (mostly for autodeploy). You can use the image builder CLI to create a custom profile.
If you want to view the VIBs on your ESXi host use the following command:
esxcli software vib list
There are many third party custom ISOs, bundles, and online depots. VMware recommends that you use a vendor customized ISO for your hardware. Some vendors are extremely timely, while others lag or nearly non-existent. I know from personal experience the HP install ISOs are heavily customized, while the Cisco ones only have a handful of drivers. Nutanix, for example, goes through a thorough testing process and bakes the ESXi install into our Foundation product. So no need to deal with custom ISOs or VIBs, as Foundation will deploy everything needed in an automated fashion.
Upgrading vSphere Hosts
The big question is: Should I upgrade the host or do a fresh install? Unlike vCenter where VMware recommends to do a fresh install, if possible, they recommend upgrading ESXi hosts. You can leverage features like HA, DRS, storage vMotion, and host profiles to quickly roll through hosts. Fresh installs should be limited to a small number of hosts, maybe for test purposes.
Before you upgrade check the VMware Compatibility Guide. Just because your host works with 5.0 or 5.5, does NOT mean it will work with 6.0. For example, historically HP BladeSystem has needed newer firmware to address gotchas with new ESXi builds. Don’t just blow this step off and think you have a tier-1 vendor so all is good. Likely specific firmware versions will be required/approved. Also, with 6.0 VMware removed some drivers like RealTek NICs. So if you do a fresh install you may suddenly be missing your NICs on a whitebox server. Good news is that if you are using a Mac Mini, many models come with out of the box support in 6.0!
If you are Nutanix customer, you can do a one-click hypervisor upgrade once we have qualified vSphere 6.0. This means you don’t need to use VUM, as the Nutanix PRISM GUI fully automates the upgrade process for you. Keep an eye out for the Nutanix announcement of vSphere 6.0 support. Our stated SLA is 90 days from GA.
The vSphere 6.0 release notes are quite lengthy. A number of support calls can be avoided by getting a heads up of issues. That’s why planning is so important. Get a cup of coffee or Five Hour Energy and read every issue in the release notes. It can pay dividends! The vSphere 6.0 release notes are here.
ESXi Upgrade Methods
- ESXi Installer – Boot from ISO, choose upgrade
- vSphere Update Manager – Import ISO, create upgrade baseline, remediate
- ESXCLI – Stage ZIP, execute ‘esxcli system profile update’
- Scripted Upgrades – Update/customize upgrade script
- Nutanix – One click upgrades
The most popular and automated method is using VUM. It will orchestrate host maintenance modes, respect DRS directives, and generally make it seamless. You can directly upgrade from ESXi 5.x to 6.0.
Rolling upgrades within clusters are supported and highly recommended. Do take note that vCenter 6.0 does not support ESX/ESXi 4.x hosts, so upgrade them to 5.x prior to upgrading vCenter. Be careful with VM hardware compatibility in such situations though. ESXi 6.0 has wide latitude in virtual hardware support, so there’s no critical rush to upgrade to v10 or later hardware. Be sure to leverage HA, DRS, vMotion and storage vMotion to enable minimal/zero downtime upgrade. If you are using Enterprise Plus, leverage host profiles. It minimizes configuration drift and enables stricter configuration control.
Upgrading ESXi Hosts
The boot disk is not re-partitioned during the upgrade process. However, the contents ARE overwritten. If there’s a VMFS datastore on the boot volume it will be preserved. Same for scratch. Absolute minimum is 1GB of space on your boot volume. Here’s a good KB on boot volume sizing. I personally use 5-6GB LUNs for boot-from-SAN configurations. The figure below shows the basic partition layout of an ESXi installation. This scheme has not changed for 6.0.
VMware has changed their nomenclature in how they refer to VM hardware compatibility. Previously they always called out the specific “hardware” version such as 4, 7, 9, etc. But that didn’t obviously relate to a specific release, and people got confused. Plus they thought on my gosh I’m on HW 4 and they are up 9, I’m way out of date…upgrade!
Now VMware calls out the “Compatibility” level and ties that to a release of ESXi. For example, if under the covers the VM is HW v7 it will show ESX 4.x and later in the web GUI. Do NOT feel pressure to always upgrade the compatibility level. Sometimes you need to, such as provisioning a monster VM that wasn’t supported on older versions of ESXi. And sometimes there are performance gains to be had when using new vHW versions. My advice is to upgrade the vHW as part of your overall upgrade plan. Do realize that some new VM features can’t be edited in the Windows C# client, but basic properties like RAM and vCPUs can be modified. Click on the graphic to expand and see the various upgrade paths.
Upgrading tools and VM hardware is OPTIONAL, and VMware officially supports N-4 versions. VM hardware versions are NOT backwards compatible, though. You won’t be running HW version 11 VMs on anything but vSphere 6.0.
VMware tools are backward and forward compatible to a very large degree. Don’t freak out if your VM isn’t running the latest tools. VMware recommends you DO keep up (performance, security, compliance checking, etc.), but you have wide latitude. Backup software, HA, heartbeats and other functions rely on VMware tools so if they have problems, verify the tools version matches your host. VUM is excellent for verifying compliance. My recommendation is to keep your VMware tools up to date, specially after a big upgrade such as going to 6.0.
For those of you that heard starting with vSphere 5.1 that upgrading VMware tools would no longer require a reboot, that’s not actually the case. The low-down is that VMware did make changes to VMware tools to leverage Windows hot-swap of some kernel modules. However, some modules like keyboard/mouse/USB still require reboots. VMware includes those non-hot-plug modules in each tools update. So the net result is still needing to reboot when doing VMtools updates. Perhaps in the future they will change that behavior, but that’s not in 5.1 through 6.0.
VMFS upgrades are simple, and completely non-disruptive. You can upgrade a VMFS datastore from VMFS-3 to VMFS-5 with running VMs. However, while this may sound perfect, keep reading as the reality is more complicated. The table below shows the differences between the two filesystem versions. Now that VMFS-5 has been around for a while, I hope you don’t have too many VMFS-3 datastores around.
Ok so you are thinking, why is an upgrade not ideal? The problem is that an upgraded volume does NOT look the same under the covers from a freshly formatted VMFS-5 volume. The table below shows the differences. The most impacting can be the block size. In vSphere 4.x and earlier you had a choice of block sizes that ranged from 1MB to 8MB. If your array supports VAAI extensions the VMFS volumes must have the same block size if you are doing operations such as copying VMs. Otherwise the disk operations revert back to legacy mode and will run slower.
The VMware recommendation is to create a fresh VMFS datastore then storage vMotion your VMs into the datastore. After the datastore is evacuated re-format or decommission it. If you aren’t licensed for storage vMotion, then during your vCenter upgrade don’t input a product key. This gives you 60 days of the ‘enhanced’ license features.
VMFS will play less of a role in vSphere 6.0 and beyond with the advent of VVols. VVols does not use a filesystem, so there’s no VMFS to deal with. Once your storage array supports VVols and you migrate VMs to vVols you can forget about VMFS. I have no insider knowledge here, but I’d be surprised if VMware released any major new VMFS versions given the VVols future.
New to vSphere 6.0 are different SSL certificate options. They are:
- VMware Certificate Authority mode – VMCA automatically provisions host certificates
- Custom Certificate mode – Enabled you to use your own certificates
- Thumbprint mode – Can be used to retain vSphere 5.5 certificates during upgrade
Which mode you use depends on your business requirements. VMCA mode is the easiest, as it automates ESXi certificate deployment. I would recommend this mode. You could use custom certificate mode and then use my vCenter 6.0 toolkit to replace the certificates, but I’d only recommend that if you can’t use the VMCA and need to use trusted certificates.
Smart Card Authentication
Also new to vSphere 6.0 is the ability to use smart card authentication to your ESXi host. They support US DoD CAC cards as well as traditional industry standard smart cards. See the vSphere 6.0 Security guide for additional details on how to configure your ESXi hosts to use smart cards. I will not be covering that in this series.
- Understand the vSphere Upgrade Process
- Understand how ESXi is packaged and distributed
- Understand patches vs. updates vs. upgrades
- Know the different upgrade methods
- Stay current on VMware tools
- Freshly format VMFS5 volumes; don’t upgrade from VMFS3
- Consciously pick which certificate deployment model you will use
- Investigate smart card authentication, if you have a business requirement for it
Now that we’ve gotten the upgrade and best practices out of the way, in the next installment we will start installing the vSphere 6.0 PSC. You can check out that installment here.