For many years home labs, and even in some production environments, booting VMware ESXi from a SD card or USB stick was an easy and simple way to deploy ESXi. For example, up until today I booted my Dell Precision 7920 ESXi 7 host from a high endurance SD card via a USB adapter.Â
Starting with some fairly recent releases of ESXi 7, the I/O load on the bootable SD/USB media has increased. This increase causes wear on the SD card or USB stick, and could cause the media to eventually fail. Specifically, the ESX-OSDATA partition has dramatically increased I/O.Â
Starting with the next major release of vSphere, VMware is pulling support for booting from SD cards and USB media. You can find the full explanation here in VMware KB85685. You can NOT use SAS, SATA or NVMe through any converters hanging off the USB bus. The storage media must use a native bus.Â
VMware is now recommending booting from high quality internal M.2 media, and migrating off SD cards or other USB media. If you don’t, then trouble will arise when the next major version of vSphere is released. At a minimum the M.2 media should be rated for 128TBW (TB written) and 100MB/s I/O. Thankfully, 128TBW is not a super high bar even for consumer grade NVMe media.Â
Â
My M.2 Media Shopping List
My heavily nested home vSphere environment runs on a single Dell Precision 7920 tower, with very beefy specs. However, I was booting from a very high endurance SD card via a USB adapter. So, I thought now is a good time to migrate to internal M.2 media, and share the build list and process. Thankfully I had a free internal PCIe slot to use for this upgrade.
Direct from Amazon I ordered:
Samsung 970 Pro 512GB for $159.99
Glowtrends PCIe M.2 Adapter $11.99
Thermalright M.2 heatsink $16.99
Now, I could have gone with a cheaper and smaller NVMe SSD. The Samsung 970 Pro has a 500TBW rating, which far exceeds VMware’s minimum recommendation of 128TBW. For example, the 256GB Samsung 980 Pro only costs $89.07 on Amazon. However, the TBW rating dramatically drops to 150TBW. So the Samsung 980 Pro would probably work fine, for $70 more I got over 3x the endurance rating and twice the space, which I think is money well spent. I plan to keep my new lab gear for years, and possibly VMware’s TBW floor will be raised over that time.
Update: Amazon has a Hynix 500GB 500TBW drive for $60 after coupon, here (normally $75). This is a much better value than the Samsun 970 Pro, with the same warranty and ware specs.Â
Â
USB SD Migration to M.2 Process
The migration process from booting ESXi 7 from SD card to internal NVME M.2 media was actually very straight forward and could be done in an afternoon for one host.Â
Here’s the process I followed:
- Take note of the ESXi version and build number for your SD/USB booted host.Â
- Locate the VMware ESXi ISO for the exact same build number (VERY important) and write it to a spare USB key.Â
- Shutdown your ESXi host, and install the NVMe M.2 card.
- Insert the spare USB key with your freshly written ISO on it.
- Boot your ESXi host from the spare USB key.
- Install ESXi to the NVMe m.2 drive. Configure the network settings and test.
- Remove the spare USB key, and boot back into your production SD/USB ESXi environment. Make sure SSH is enabled.
- SSH into your production ESXi environment and follow the “ESXi Command line” section in VMware KB2042141. Follow steps 1-3 to generate and download the ESXi configuration backup to your local computer.
- SCP/SFTP into your production ESXi host, and locate an available VMFS volume to put the ESXi backup file. I put it on the VMFS volume that the ESXi installer created on my Samsung drive. The volumes are under /vmfs/volumes. Copy the config file to this VMFS volume.
- Shutdown your production ESXi host and remove the SD/USB media.
- Boot into your NVMe ESXi environment and enable SSH.
- SSH into the new ESXi environment and follow steps 1-4 under “Restoring ESXi host configuration data” in KB2042141.
- After the ESXi host reboots, the installation configuration should match that of your SD/USB environment.
Â
Summary
VMware is now telling customers that future major versions of vSphere will NOT support booting from SD cards or USB-attached media. So, customers will need to migrate their ESXi boot environment onto different media. The ESXi backup/restore process worked flawlessly for me. As far as I can tell, it retained all my ESXi settings including distributed switches, network info, VM registrations, license key, etc. Color me impressed.Â
Bottom line, I suggest moving away from booting from SD/USB media ASAP. If you have a spare PCIe slot in your server, the solution is not too expensive and pretty easy to do. If you have NUC hosts, or don’t have a free PCIe slot, then your path forward may be more complicated or expensive.Â
Thanks Derek. For home I’ll surely migrate now. My problem is at work. Servers almost all of them come from the vendor pre-loaded on a usb drive and today those servers are on the hcl. We are running 7 and seeing issues already “bootbank”. You can’t just switch many servers nor change mid-stream of a lifecycle. Problem for VMware IMHO.
This was very helpful. Thanks for the post
Awesome! Thanks for the great write up. I have to perform the same process on a Dell PER6415 server soon and I’m hoping it wont give me problems. The only difference is that Dell server requires me using their BOSS card with the NVME SSD.