Upgrades can be scary times with any enterprise product. The more your critical infrastructure relies on a particular solution, or set of solutions, the more imperative it is you fully understand and test the new product. vSphere 5.1 taught us that thorough testing cannot be skipped and you should not rush a new product into production.
Normally for my vSphere installation series I do NOT cover upgrades, or go through an upgrade process in the series. Why? Customer environments wildly vary and a simple lab upgrade will likely not look like or behave like YOUR environment. That’s why its so critical for you to test in your environment. My upgrade would not look like your upgrade.
But, what I am doing in this post and the next installment is covering upgrade best practices to help you understand your road ahead and things to keep in mind. It contains information from VMworld 2013 vSphere 5.5 upgrade sessions, plus links to resources that have been published post-GA. This post covers vCenter only, and the next installment covers VMs, VMFS, ESXi hosts, and other products.
SQL 2012 AlwaysOn Failover Cluster for vCenter
vSphere 5.5 Install Pt. 1: Introduction
vSphere 5.5 Install Pt. 2: SSO 5.5 Reborn
vSphere 5.5 Install Pt. 3: vCenter Upgrade Best Practices and Tips
vSphere 5.5 Install Pt. 4: ESXi Upgrade Best Practices and Tips
vSphere 5.5 Install Pt. 5: SSL Deep Dive
vSphere 5.5 Install Pt. 6: SSL Certificate Template
vSphere 5.5 Install Pt. 7: Install SSO
vSphere 5.5 Install Pt. 8: Online SSL Minting
vSphere 5.5 Install Pt. 9: Offline SSL Minting
vSphere 5.5 Install Pt. 10: Update SSO Certificate
vSphere 5.5 Install Pt. 11: Install Web Client
vSphere 5.5 Install Pt. 12: Configure SSO
vSphere 5.5 Install Pt. 13: Install Inventory Service
vSphere 5.5 Install Pt. 14: Create Databases
vSphere 5.5 Install Pt. 15: Install vCenter
vSphere 5.5 Install Pt. 16: vCenter SSL
vSphere 5.5 Install Pt. 17: Install VUM
vSphere 5.5 Install Pt. 18: VUM SSL
vSphere 5.5 Install Pt. 19: ESXi SSL Certificate
Permalink to this series: vexpert.me/Derek55
Permalink to the Toolkit script: vexpert.me/toolkit55
vSphere 5.5 Upgrade Overview
- Plan your upgrade – Extremely important. KB on update sequence is here.
- Five major steps: vCenter, VUM, ESXi, VMs, VMFS
- Key VMware Sites to bookmark: Documentation Center, Compatibility Guide, Interop matrix
- KB article here for vCenter 5.5 Upgrade using the Simple Installer
- KB article here for vCenter 5.5 Upgrade using the Custom Installer
- If you upgrade Windows with a service pack or other system changes and get locked out of SSO, read this KB to regain access
- Upgrade vCenter to 5.5 before vSphere replication is upgraded (blog post here)
Prior to 5.1 life was simple. You had vCenter Server, vCenter Database server, and vSphere web client (introduced in 5.0, but rarely used). The vCenter server is NOT stateless, meaning the database is not all inclusive. The local vCenter server has SSL certificates and the ADAM database. ADAM is not just for linked mode but holds data such as licenses, roles, and permissions. So don’t stand up a fresh VM, install the “old” version on that VM then do an upgrade to 5.5 and expect everything to be there. It won’t be and further complicates your upgrade process. If you are using vSphere 5.1, then ‘tags’ are also stored locally on the vCenter server and thus not in the database.
- In-place upgrade supports vCenter 4.x, 5.0.x, 5.1.x (must be 64-bit host)
- VMware does NOT support directly migrating an existing 5.x or earlier vCenter Server to a new machine during the upgrade process
- vCenter Server 5.5 can manage ESX/ESXi 4.x, 5.0.x and 5.1.x hosts. It will NOT manage ESX 2.x or 3.x hosts.
- Strongly recommend installing ALL vCenter components on a single VM – Simplified model
- Simple install – 2 vCPUs, 12GB RAM, 100GB disk
- Recommended for 400 hosts or 4000 VMs: 4 vCPU, 24GB RAM, 200GB disk
- vCenter OS Support: Removes WS2003, only supports Windows Server 2008 SP2 and later (including WS2012 but NOT WS2012 R2)
New Install Vs. In Place Upgrade
VMware recommends a fresh install, but sometimes its not just possible. However, do check out the “Inventory Snapshot” Fling, which is a great (unsupported) tool to migrate hosts, VM, and permissions from one vCenter instance to another. It does NOT appear to support tags and currently has some vDS issues. Tags are not stored in the SQL database, so if you use tags then be sure to find a way to migrate them. If you are in a regulated industry and have strict audit requirements you may be legally required to maintain the historical data in your vCenter database and unable to start fresh.
If you have a sprawling 5.1 architecture, with different vCenter components on different VMs, strongly consider a fresh install and do not upgrade. As previously mentioned VMware now urges the “simple install” method where all components are on a single beefy VM. This is a great time to re-visit your architecture and make it easier to manage and follow 5.5 best practices. That’s not to say you can’t upgrade and consolidate at the same time, you can, and VMware has promised some blog posts on how to do just that.
I’ve read reports that upgrading a vCenter 5.1 instance with trusted SSL certificates to 5.5 had problems. I have not personally tried that yet, so I can’t report my own experience. So make sure you have full backups and a tested plan to revert back to 5.1 incase you experience problems.
VMware has stated that the vCenter Server appliance will be the ONLY deployment option sometime in the future. So if you are starting with a fresh install, do take a close look at the VCSA. It still has a few minor gotchas including no support for IPv6, Linked Mode or vCenter Heartbeat. Those features are probably not widely used, so if you aren’t using those features take a serious look at VCSA.
At this time an external SQL database is NOT supported for the VCSA, but in the future when Microsoft releases the ODBC driver for SUSE Linux (currently in tech preview), VMware will support it. VCSA is certified up to 100 hosts and 3000 VMs. If you need to scale beyond that, use Windows.
Installation – Then and Now
vSphere 5.5 features a new Install splash screen, and the component order is different from 5.1. Simple Install should only be used for the first vCenter. All subsequent vCenter/SSO installs should use the custom method. This is due to changes in SSO, and the new automatic replication among SSO servers. Even if you are doing a single vCenter install and want to customize it in ANY way, including directory paths, you must do the custom install.
For “typical” single server upgrades the path is fairly simple. You can do an in place upgrade and all of the required components and configuration settings will be retained. If you are going from pre-5.1, then the only database in play is the vCenter database.
If you are already running 5.1, then the upgrade path is ever so slightly different. Since the SSO database in 5.1 is no more, that data is migrated into the new SSO internal database. So post upgrade you are left with only the vCenter upgrade. Yes, no more SQL authentication required or impossible to configure JDBC SSL.
If you are one of those adventurous customers that implemented a load balancer with SSO, VMware is really discouraging you to continue with that model. Its complex, SSL creates additional headaches, and just not needed in most environments. Big changes could be coming in the future, but it’s not recommended for 5.5. As mentioned in my previous installment, SSO Reborn, VMware recommends local SSO instances for each site/vCenter. SSO uses multi-master replication to sync data such as identity sources, users, group, and policies. A geographically distributed example is shown below. Notice the local SSO and vCenter instances at each site.
Linked mode adds additional complications to the upgrade process. As you may recall you can’t link vCenters of different versions. So you first need to unjoin all vCenters from the linked mode group. Once you upgrade two vCenters to 5.5, you can then re-establish Linked Mode and add other 5.5 vCenters as they come online. The biggest problems with Linked Mode include DNS and NTP failures. It’s critical name resolution works (forward AND reverse) and that the server clocks are all synchronized. All vCenter servers that are linked must also be a part of the same SSO authentication domain.
Host Agent Pre-Upgrade Checker
A tool included on the vSphere 5.5 ISO is the Host Agent Pre-Upgrade checker. Personally I’ve never used it (slipped my mind that it existed). If you choose to use it some simple checks are done against your ESXi hosts to validate that an upgrade will be successful. It’s not exhaustive, so even if your hosts pass the check you could still run into issues. But it’s a little bit of insurance that major gotchas can be discovered ahead of time. It does check items such as sufficient disk space, functional network, file system consistency, required patches are applied.
The VCSA has undergone major scalability increases in 5.5. In 5.1 it was only rated for 5 hosts and 50 VMs when using the embedded database. With 5.5 that is increased to 100 hosts and/or 3000 VMs. So that makes it a much more viable solution for enterprise customers. You can NOT migrate from the Windows vCenter to the VCSA. As mentioned before, there’s also no Linked Mode, vCenter Heartbeat or IPv6. Again, the road map is an appliance only model for vCenter, so now is an excellent time to try it out. VMware said upgrades to future versions will be pretty easy, simplifying life.
You can upgrade VUM from 4.x, 5.0 and 5.1 versions. VUM is still Windows only, so if you do deploy the VCSA you will still need a Windows server to host VUM. The web client in 5.5 also has limited VUM functionality, so the C# is still needed to do things like pushing patches and configuring baselines. During the upgrade you can’t change the installation or download paths. Scheduled tasks remain, but patch baselines are removed.
VMware has hinted/stated that VUM is going the way of the dodo bird. I would expect its replacement to be very different, and probably incorporated into the VCSA. I’m hoping in vSphere 6.0 there’s a good story on the VUM successor.
You need to carefully plan your upgrades, and understand all of the moving components. Generally you would start by upgrading vCenter, then your ESXi hosts. But you may have other products that depend on vCenter which need upgrading first. Thoroughly map out all of your dependencies, read the VMware documentation, then plan in an organized fashion how you are going to upgrade. If you are already on 5.1, custom SSL certificates may trip you up. So really make sure you have a full backup and roll-back plan in case things go pear shaped.
Next up in Part 4 are practices and tip for upgrading ESXi hosts, VMs, and VMFS datastores.
[quote]VMware has hinted/stated that VUM is going the way of the dodo bird. I would expect its replacement to be very different, and probably incorporated into the VCSA. I’m hoping in vSphere 6.0 there’s a good story on the VUM successor[/quote] Dropping guest patching made sense in 5.0 and put the focus more on ESX/ESXi, so hopefully a more simplified approach to specifying and attaching baselines in 6. I'd be a little unsure as to disk space of the VCSA if you'd then up with both a built-in database for vCenter and then VUM data, but it makes sense if… Read more »
“VMware does NOT support directly migrating an existing 5.x or earlier vCenter Server to a new machine during the upgrade process”
Derek, please share your perspective on the following contradictory (seemingly) info about the that statement:
Migrate vCenter Server and Components from a Windows Server 2003 Host:
A couple of other vExperts discussed this topic here:
If the described manual migration technique works for a Windows Server 2003 host scenario, why wouldn’t it work for other scenarios where migration to a new vCenter server instead of in-place upgrade might be desirable?
During VMworld 2013 VMware stated that concurrent upgrades while rehosting on a new server was not supported. Complications arise from the data stored in ADAM, SSL certificates, etc. I have not used the migration process that you've linked to. If you did want to try that, I would do it in a lab and see if it appears to work. At VMworld they stated the preferred upgrade method is a fresh install, then migrating objects to the new instance. Clearly this won't work for everyone, and on a large scale, would be a large amount of work. Not to mention… Read more »
Thank you for sharing your insight. In particular, you’ve confirmed my concerns about one the things that gave me pause about the process I linked to: it makes no mention of ADAM considerations.
Thanks as well for contributing this excellent blog series to the VMware community. Excellent work!
2.The Inventory Service is a service, but also uses a database. This flat-file database contains: ◾vCenter tags (version 5.1+ only)
◾Index of which VMs have which Storage Profile assigned