vSphere 5.5 U1 NFS APD Fix out

A couple of months ago when upgrading one of my Nutanix clusters to vSphere 5.5 U1 I started to see what appeared to be random loss of connection to my NFS datastores. All VMs on the datastores would become inaccessible, then a few minutes later, access would be restored. As it turns out, this All Path Down (APD) bug introduced in vSphere 5.5 U1 and affected most any storage vendor using NFS. VMware wrote a KB article about the problem, which you can read here.

I am now pleased to announce that with vSphere 5.5 express patch 04, VMware has said they resolved the issue. You can download the patch here. I haven’t yet tried out the patch, but I’m very glad it is now out. For those of you still on vSphere 5.5 GA and using NFS, I would encourage you in a *lab* environment to thoroughly test express patch 04 and validate for your environment the issue has been resolved.

A big thanks to VMware for staying on top of this issue and coming out with a patch. I’ll be testing this in my lab very shortly.

vSphere 5.5 Install Pt. 19: ESXi SSL Certificate

12-22-2013 2-49-19 PMAfter a bit of a delay, I’m finally publishing how to update your ESXi SSL certificate. The process is pretty much unchanged from the 4.x era, but what is new is my Toolkit script. It has been updated to include ESXi certificate support. This is accomplished all in PowerShell, and does NOT require SSH be enabled on the host. It uses the HTTP PUT command to upload the two required files. This should also run successfully against older ESXi hosts, but I haven’t tested it.

Blog Series

SQL 2012 AlwaysOn Failover Cluster for vCenter
vSphere 5.5 Install Pt. 1: Introduction 
vSphere 5.5 Install Pt. 2: SSO 5.5 Reborn 

vSphere 5.5 Install Pt. 3: vCenter Upgrade Best Practices and Tips
vSphere 5.5 Install Pt. 4: ESXi 5.5 Upgrade Best Practices and Tips 
vSphere 5.5 Install Pt. 5: SSL Deep Dive
vSphere 5.5 Install Pt. 6: SSL Certificate Template
vSphere 5.5 Install Pt. 7: Install SSO
vSphere 5.5 Install Pt. 8: Online SSL Minting
vSphere 5.5 Install Pt. 9: Offline SSL Minting 
vSphere 5.5 Install Pt. 10: Update SSO Certificate
vSphere 5.5 Install Pt. 11: Install Web Client
vSphere 5.5 Install Pt. 12: Configure SSO
vSphere 5.5 Install Pt. 13: Install Inventory Service
vSphere 5.5 Install Pt. 14: Create Databases
vSphere 5.5 Install Pt. 15: Install vCenter
vSphere 5.5 Install Pt. 16: vCenter SSL
vSphere 5.5 Install Pt. 17: Install VUM
vSphere 5.5 Install Pt. 18: VUM SSL
vSphere 5.5 Install Pt. 19: ESXi SSL

Permalink to this series: vexpert.me/Derek55
Permalink to the Toolkit script: vexpert.me/toolkit55

Introduction

Download my vCenter 5.5 Toolkit v1.5 or later from here. It was just updated for this post, so if you’ve been following along and already have the script, you will need the updated version for the ESXi features. At this point I’ll make the following assumptions: 1) You’ve installed ESXi 5.x 2) Created a DNS entry for your ESXi host 3) ESXi host is reachable on the network via the FQDN (e.g. ESX01.yourdomain.com) 4) The host you run the Toolkit script on has HTTPS access to the ESXi host(s).

VMware HA can get confused with updated thumbprints when you replace the SSL certificate if the host was already added to vCenter. For this reason I recommend either updating the host SSL certificate prior to adding to vCenter, or disconnect the host in vCenter, update the certificate, and reconnect in vCenter. Should you run into SSL thumbprint errors, you may see a message like the one below. For more information you can checkout KB 2006210.

12-22-2013 4-02-36 PM

The script is fairly agile, and can produce ESXi certificates via a variety of methods. You can also feed it a CSV file of hosts, and mass produce certificates as well. Each method has its own section below. I suggest reading through all methods so you know what options you have and get the complete picture.

Online Minting Method

1. Open an elevated PowerShell prompt and run the Toolkit script. You will see the following menu items. Version 1.5 and later has a new “ESXi Hosts” section with several options. Like the other modules, the script supports an online Microsoft CA, an online Microsoft CA that requires manual approval, or creating the CSRs to use with a non-Microsoft CA.

In this first example let’s assume you have an online Microsoft CA that does not require manual certificate approval. Select Option 11.

12-22-2013 3-03-31 PM

2. You are now presented with a second menu. Here you can manually enter the ESXi hostnames, or read in a CSV if you have lots of hosts to prepare certificates for. Let’s first go for the manual host entry and select Option 1.

12-22-2013 3-13-05 PM

3. After selecting Option 1 you are prompted to enter the hostname(s) of the ESXi servers. Be sure to use the FQDN of your ESXi host(s). Comma separate the hosts if you input more than one. Assuming you haven’t run the script before it will then ask you for the root credentials of the ESXi host. The credentials must be the same for all hosts, and you only need to enter it once no matter how many hosts you are updating.

12-22-2013 3-15-58 PM

4. After you enter the root credentials you should get yellow status messages for each ESXi host that the certificates were successfully uploaded to. There is some error trapping, so errors like incorrect credentials or invalid hostnames will throw an exception message, but continue with other hosts.

5. At this point I recommend rebooting the ESXi host. Yes, technically you can restart the ESXi management agent but I feel better with a full reboot. After the reboot open your favorite browser and go to the FQDN of your ESXi host. You should not get any SSL errors. You can also open the certificate properties and verify it came from your trusted CA.

12-22-2013 4-18-16 PM

You can now add your ESXi host to vCenter. Find your cluster in vCenter, then right click on it and select Add Host. Enter the FQDN of your ESXi host and run through the rest of the wizard. It should now be added, and the proper thumbprint stored in the database.

12-22-2013 3-33-25 PM

Offline Minting Method

1. Run my Toolkit script but this time select option 12. This will only create the CSRs, which you will then submit to your own CA and download the minted certificate. Again here I selected option 1 to manually enter the ESXi hostname.

12-22-2013 4-25-56 PM

2. If you look in the certificate directory path (configurable in the script), you will see a folder with the FQDN of your ESXi host. If you open that folder you will see three files. Take the CSR and submit it to your CA. Download a BASE64 encoded certificate (not a certificate chain) and save it as rui.crt in the same folder.

12-22-2013 4-27-07 PM

3. Re-run the Toolkit script but this time select option 13. Re-enter the hostname(s) that you created certificates for. Enter the root credentials if they are not already cached. You should get a yellow status message for each ESXi host if it is successful.

12-22-2013 4-31-44 PM

4. Go back to Step 5 in the Online Minting section to reboot your host, validate the certificate is correct, and reconnect the ESXi host to vCenter.

Manual Approval Method

1. Run my Toolkit script using option 11. This will proceed just like the online method, but it will display RequestIDs that your CA administrator must approve. I selected option 1 on the sub-menu to manually enter the ESXi hostname.

12-22-2013 4-59-17 PM

2. Note in yellow the RequestID(s) and have your CA administrator approve them.  After it is approved, re-run the Toolkit script but select option 13 from the main menu. Re-enter the same ESXi hostname(s) or CSV file that you used for the original request. You should see a status message showing the certificate(s) were downloaded and then successfully uploaded to each ESXi host.

12-22-2013 5-02-27 PM

3. Go back to Step 5 in the Online Minting section to reboot your host, validate the certificate is correct, and reconnect the ESXi host to vCenter.

CSV Input File

If you have several hosts that you need to update certificates on, then you don’t really want to be typing in all the hostnames. So this script will also accept a formatted text file of ESXi hosts names. The input file can be used with ALL minting methods (online, offline, manual approval). The file is very simple. Each line should have the FQDN of a single ESXi host. There is no limit to the number of hosts you can put in the file.

12-22-2013 5-07-13 PM

To use this file merely select Option 2 on the sub-menu (Read ESXi hosts from CSV file) and input the path to your text file. You can see a sample use case below, where I’m using manually approved certificates.

12-22-2013 5-09-59 PM

Summary

As you can see, the Toolkit script is now fairly complete, although delayed a bit longer than I had originally planned. It certainly is not foolproof, but does have some error checking. No doubt there will be some circumstances where it will fail. Should you need to manually copy the certificate files to the host using something like WinSCP, copy the rui.crt and rui.key files to /etc/vmware/ssl on the ESXi host. Be sure to use ASCII/text mode to avoid translation issues. Also, if you wish to delete the cached root credentials (I would suggest this after completing the certificate upgrades), then remove the root-credentials file from your Certificate directory.

VMware ESXi 5.1 Patches Released

VMwareHot off the presses are some ESXi 5.1 patches. This build of ESXi 5.1 (1157734) fixes several bugs and more importantly addresses some security issues. As always in any environment, please test out the patches thoroughly before putting them into production. Each environment is unique, and issues may surface that could cause you some headaches. These bug fixes aren’t earth shattering, so I would not suggest rushing them out to production systems.

ESXi 5.1 Build 1157734

Highlights of the patch bundle included in this release are:

  • Black frames might appear around text boxes in an application running on Virtual Machine Hardware Version 8 or later. This issue occurs on virtual machines with Windows 7 guest operating system and View 5.0 PCoIP.
  • For two ESXi hosts with different host names, identical machine names are generated in the domain controller under certain conditions. As a result, the ctive Directory functionality is lost for one of the two ESXi hosts.
  • After you upgrade to ESXi 5.1 from an earlier version, attempts to power on a virtual machine with static MAC address outside the allowed range (00:50:56:[00-3f] or 00:50:56:[80-BF]) fail with the following error message: The MAC address entered is not in the valid range.
  • If a physical NIC is named using non-standard naming conventions (other than vmnic#) and is added to a vSwitch, host profile creation fails with the following error message: Invalid value chosen for active NICs.
  • ESXi 5.1 hosts might get disconnected randomly from the vCenter Server system. This issue might occur if the heartbeat thread in the vpxa agent does not receive a response from the futex_wait system call. As a result, the heartbeat thread stops responding, and the vCenter Server does not receive heartbeat messages from the ESXi hosts for several hours.
  • Upon reboot, ESXi 5.1 hosts configured to obtain DNS configuration and host name from a DHCP server displays its host name as localhost in syslog rather than displaying the host name obtained from the DHCP server. As a result, for a remote syslog collector, all ESXi hosts appear to be the same, with the same host name.
  • To prevent buffer overflow, the HPSA proc node truncates LUN details on an ESXi host.
  • This patch updates the esx-base VIB to resolve a stability issue.

As always, you can down the ESXi patches from here. The full KB article for the patch bundle is here.

VMware vSphere 5.1 Update 1..is it for you?

VMware vSphere 5.1 update 1 is probably one of the most recently anticipated updates of the VMware stack and it has finally hit the streets. For those of you following the release of vSphere 5.1, you have seen the GA release last fall, followed by 5.1.0a then a couple of months later 5.1.0b, all addressing bugs and ironing out critical installation issues.

VMware vSphere 5.1 Update 1 has a laundry list of improvements, support for new Microsoft products, and a lot of bug fixes. If you are still on vSphere 5.0 or tried 5.1 in the past and ran into problems, you definitely need to check out vSphere 5.1 Update 1. If you want a complete vSphere 5.1 installation guide, check out my 15-part blog series here. I will be updating it in the near future for Update 1. If you are running vSphere 5.1, there are a number of security vulnerabilities addressed in the update so start planning your upgrade.

VMwareKnown Issues with vSphere 5.1 Update 1

Today VMware posted a new KB warning about a vSphere 5.1 Update 1 bug, which may affect customers. The problem prevents you from logging into the vSphere Web Client using an AD account, if you AD account is a member of approximately 19 or more domain groups and the SSO service is configured with multiple domains. The KB states until a hotfix is released, DO NOT upgrade to vSphere 5.1 Update 1. In many enterprise environments a vSphere administrator may be in dozens of groups, depending on how access is controlled within the domain. Fewer customers will probably have SSO configured for multiple domains, so the impact of this issue is probably limited to larger enterprises. Additional issues include:

  • If you are using the vSphere Storage Appliance, you MUST upgrade to vSA 5.1.3 after you upgrade the rest of your infrastructure to vSphere 5.1 Update 1. vSA 5.1.1 is NOT compatible with vSphere 5.1 Update 1.
  • You can NOT use the simple installer to upgrade from prior 5.1 versions to 5.1 Update 1. You must utilize the individual installers.
  • Windows Server 2012 failover clusters are NOT supported on ESXi 5.1 Update 1. The cluster validation wizard gets stuck in an endless loop and you are unable for form the cluster.

What got updated in vCloud Suite 5.1 Update 1?

  • ESXi 5.1 Update 1 Build 1065491
  • vCenter Server 5.1 Update 1 Build 1065152
  • vSphere Data Protection 5.1.10.32
  • vSphere Replication 5.1.1
  • vSphere Storage Appliance 5.1.3
  • vCenter Orchestrator Appliance 5.1.1
  • vCloud Director 5.1.2
  • vCenter Site Recovery Manager 5.1.1
  • vSphere 5.1 Update 1 Virtual Disk Development Kit
  • vSphere CLI 5.1 Update 1
  • VMware Converter Standalone 5.1 (Download here)
  • VMware vCenter Server Heartbeat 6.5 Update 1
  • VMware vSphere Management Assistant (5.1.0.1 – April 4, 2013)
  • HP Custom Image for ESXi 5.1.0 Update 1 Install CD

You can find all of these downloads in the usual place, My VMware. You can download the updated documentation archive ZIP bundle here. The full documentation page is here.

vCenter Server 5.1 Update Release Notes

vCenter 5.1 Update 1 is more than just bug and security fixes, it incorporates a number of newly supported operating systems and database back-ends. You can find the full release notes here. Below is just a tiny faction of the new features and bugfixes.

What’s New?

  • vCenter Server can be installed on Windows Server 2012
  • vCenter can use Microsoft SQL Server 2012 and SQL Server 2008 R2 SP2
  • Guest operating customization support for Windows 8, Windows Server 2012, Ubuntu 12.04 and RHEL 5.9
  • Removed vRAM usage limit of 192GB on vSphere Essentials and Essentials Plus

Resolved Issues

A lot of bug fixes are included, but a few highlights include:

  • Better error reporting when accidentally updating the Admin or STS service with incorrect protocol parameters. It will now tell you what you botched up.
  • Number of security patches including Java, tcServer, vCSA remote code vulnerability
  • Upgrade issues from 5.1.0a to 5.1.0b

VMware ESXi 5.1 Update 1 Release Notes

  • Mirrors the new guest OS support in vCenter 5.1 Update 1. Full 200+ page OS compatibility matrix is here.
  • Contains several security patches (glibc, libxslt, libxml2)
  • Resolved: Long running vMotion operations might result in unicast flooding
  • Windows Server 2012 failover clustering is not supported

You can find the ESXi 5.1 Update 1 full release notes here.

vCloud Director 5.1.2 Release Notes

Like vCenter 5.1 Update 1, vCloud Director has some new features and many resolved issues. Full release notes is here. The full vCloud Director 5.1.2 documentation set is here.

What’s new?

  • Ability to delegate creating, reverting, and removing snapshots
  • You can install vCloud Director on Red Hat Enterprise Linux 6.3
  • You can install vClould Director using Microsoft SQL Server 2012 databases
  • Supports customization of Windows Server 2012 guest operating systems

Resolved Issues

  • Security vulnerabilities addressed by updating Java to 1.6.0_37
  • Multiple bug fixes, see full release notes

vCenter Converter Standalone 5.1 Release Notes

The new version of Converter has added a number of great new features and broader operating system support. You can find the full release notes here.

  • Supports VM hardware version 9
  • Guest operating system support for Windows 8 and Windows Server 2012
  • Guest operating system support for Red Hat Enterprise Linux 6
  • Support for machine sources that use GPT partition tables
  • Support for systems that use UEFI
  • Support for EXT4 file system

vCenter Server Heartbeat 6.5 Update 1 Release Notes

No major changes here, but incremental support for the latest VMware products. Full release notes are here.

  • Support for vCenter 5.1 Update 1
  • Support for View Composer 5.2

vSphere Data Protection 5.1.20 Release Notes

More than just bug fixes, VMware added many new features to this build. Full release notes are here. A subset of the new features:

  • Integration with vCenter alarms and alerts notification system
  • Ability to clone backup jobs
  • New filters to restore tab
  • Expands capacity up to 8TB per appliance
  • Supports the ability to expand existing datastores
  • Supports guest-level backups of Microsoft SQL Servers
  • Supports guest-level backups of Microsoft Exchange Servers

vSphere Storage Appliance 5.1.2 Release Notes

Like vSphere Data Protection, the vSphere Storage Appliance has many new features. The full release notes are here.

  • Support multiple VSA clusters managed by a single vCenter Instance (about time)
  • Ability to run vCenter Server on a subnet different from the VSA cluster
  • Support for running the VSA on one of the ESXi hosts in the VSA cluster
  • Ability to install the VSA on an existing ESXi host that has running VMs
  • Ability to increase the storage capacity of a VSA cluster
  • Up to 24TB of storage per node
  • Multiple RAID types (RAID 5, RAID 6, RAID 10)

Summary

vSphere 5.1 Update 1 will be a welcomed upgrade to customers already running vSphere 5.1. After a rocky start of vSphere 5.1 GA, VMware has clearly been working on stability, bug fixes, and supporting the latest Microsoft operating systems and SQL databases. The vCloud Suite is ever expanding, so when you go to download all the components you will see over two dozen downloads you can choose from. If you’ve been hesitant to move up to vSphere 5.1, give 5.1 Update 1 a whirl in your lab and see if it’s stable enough for you.

vSphere 5.1 Suite

HP ESXi 5.0 U1 Updated ISO Image

A few days ago VMware released an updated custom HP ISO image for ESXi 5.0 Update 1, which includes all of the HP specific drivers and agents. The big difference between this release and prior HP ESXi 5.0 releases is that you can download them directly from VMware and not from HP’s depot. This should make the process less cumbersome. HP states the ISO image will support G5-Gen8 servers, and it also is devoid of the HP custom license file that has caused problems in the past for some people.

To review the contents of the updated image (June 2012), you can find the version details here. To download the ISO from VMware’s web site, go here. In case you want to build your own HP ESXi 5.0 image with all the latest security patches and drivers, check out my how-to blog article here. The VMware custom ISO is not using the latest build, so for those that are super security conscious and need to load a fully patched image, you will need to refer to my blog article. Otherwise you can of course patch after the fact.

New HP ESXi 4.1 U2 and 5.0 U1 Custom ISO Media

UPDATE 10/26/12: A new blog post with updated links is here.

HP just released their June 2012 VMware ESXi 4.1 U2 and ESXi 5.0 U1 ISO installation media. Nearly every driver in this custom ISO has been updated from the previous release. You can find the entire driver set versions here. HP did NOT roll in the latest security updates, so they are still just shipping ESXi 5.0 build 623860. As of the date of this writing the latest build is 721882, which addresses a number of security issues.

You can download the ESXi 5.0 custom image here and the ESXi 4.1 custom image here. As always, these updates are free to download but you do need to register. If you wish to build your own HP custom ISO with the security patches rolled in, check out my article here. However, use the latest drivers listed in the HP media release notes, not the older versions referenced in my previous blog article.

Give the plethora of driver updates, I would recommend you download the updated drivers from the driver set page here and push them out via VUM to your production servers, after adequate testing.

VMware Security Patches Released for Several Products

VMware released a security advisory on June 14, 2012 and patch for a variety of virtualization products. Details of the affected products and the vulnerabilities are below. You can download the ESX(i) patches from here.

VMware Product Running Replace with/
Product Version on Apply Patch
============= ======= ======= =================
vCenter any Windows not affected
Workstation 8.x any 8.0.4 or later
Workstation 7.x any 7.1.6 or later
Player 4.x any 4.0.4 or later
Player 3.x any 3.1.6 or later
Fusion 4.x Mac OS/X 4.1.3 or later
ESXi 5.0 ESXi ESXi500-201206401-SG
ESXi 4.1 ESXi ESXi410-201206401-SG
ESXi 4.0 ESXi ESXi400-201206401-SG
ESXi 3.5 ESXi ESXe350-201206401-I-SG
ESX 4.1 ESX ESX410-201206401-SG
ESX 4.0 ESX ESX400-201206401-SG
ESX 3.5 ESX ESX350-201206401-SG

VMware Host Checkpoint File Memory CorruptionCertain input data is not properly validated when loading checkpoint files. This might allow an attacker with the ability to load a specially crafted checkpoint file to execute arbitrary code on the host.
The Common Vulnerabilities and Exposures project (cve.mitre.org) has assigned the name CVE-2012-3288 to this issue.

The following workarounds and mitigating controls might be available to remove the potential for exploiting the issue and to reduce the exposure that the issue poses.

Workaround: None identified.
Mitigation: Do not import virtual machines from untrusted sources.

VMware Virtual Machine Remote Device Denial of Service
A device (for example CD-ROM or keyboard) that is available to a virtual machine while physically connected to a system that does not run the virtual machine is referred to as a remote device. Traffic coming from remote virtual devices is incorrectly handled. This might allow an attacker who is capable of manipulating the traffic from a remote virtual device to crash the virtual machine.
The Common Vulnerabilities and Exposures project (cve.mitre.org) has assigned the name CVE-2012-3289 to this issue.

The following workarounds and mitigating controls might be available to remove the potential for exploiting the issue and to reduce the exposure that the issue poses.
Workaround: None identified.

Mitigation:

  • Users need administrative privileges on the virtual machine in order to attach remote devices.
  • Do not attach untrusted remote devices to a virtual machine.

Create Custom HP ESXi 5.0 ISO Media

Similar to my post on how to create custom Cisco UCS ESXi 5.0 installation media, I thought I would tackle the same problem for HP ProLiant servers. Yes, HP does provide regularly updated ESXi installation media that bundles in their drivers, but it doesn’t always have the latest security patches. In some circumstances you may want the very latest ESXi build when you do the base install, and not rely on VUM or manually patching after the installation.

To find out which drivers are bundled in the HP ISO images, you can find their official list here. You should also be aware that HP tests specific driver versions with certain firmware versions and considers it a “supported recipe”. To find the supported recipes, you must look at the Service Pack for ProLiant release notes (300+ pages) and they provide a table, such as the one shown below for their 2012.02.0 release.

To start building your own HP custom ISO for ESXi 5.0 follow these steps:

1) Add the HP software depot (no drivers, just HP specific packages).

Add-EsxSoftwareDepot http://vibsdepot.hp.com/index.xml

2) Download the complete driver set that HP includes in their image:

Broadcom NetXtreme I (net-tg3)
Broadcom NetXtreme II (misc-cnic-register, scsi-bnx2i, net-cnic, net-bnx2x, scsi-bnx2fc, net-bnx2)
QLogic Fibre Channel and CNA (scsi-qla2xxx)
HP SAS SCSI Driver (scsi-hpsa)
Emulex Network (net-be2net)
Emulex iSCSI (be2iscsi)
Emulex HBA (lpfc820)
QLogic Network (net-qlcnic)
QLogic Network (net-nx-nic)
LSI SAS (scsi-mpt2sas)
Brocade HBA and CNA (scsi-bfa)
Mellanox (net-mlx4-en)

3) Unzip each of the files that you downloaded, which will reveal another ZIP file and a VIB file, among others. We will be using the embedded bundle ZIP files. If you downloaded all of the drivers, unpacked them, and moved the bundled ZIPs to a single directory it should look like:

4) Add each software ZIP bundle to your depot using the following command, changing the ZIP filename for each bundle.

add-esxsoftwaredepot E:hpsa-500-5.0.0-offline_bundle-537239.zip

5) List all of the packages so we know which ones to add to our image profile. Note, that we haven’t added the online VMware depot, so the only packages shown here are the ones from the HP online depot and the manually downloaded packages.

Get-EsxSoftwarePackage | select Name,Version,ReleaseDate | sort ReleaseDate


6) Add the VMware software depot so we can use the latest VMware image profile:

Add-EsxSoftwareDepot https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml

7) We want to use the latest VMware profile, that includes all of the latest patches, so let’s list all the available profiles. The profile highlighted in yellow is the May 2012 release with several critical security patches, so we want that image (“standard” means it includes VMware tools). Choose the latest when you create your image.

Get-EsxImageProfile | Sort-Object “ModifiedTime” -Descending | format-table -property Name,CreationTime



8)  Clone the standard VMware profile so we can modify it with our driver set:

new-esximageprofile -cloneprofile ESXi-5.0.0-20120504001-standard
-name “ESXi-5.0.0-HP-05132012”



9) Using the output in step 5, we add all of those packages to our image profile:

add-esxsoftwarepackage -imageprofile ESXi-5.0.0-HP-05132012 hpbootcfg, char-hpcru, hpnmi, char-hpilo, hponcfg, hp-smx-provider, hp-ams, hpacucli, misc-cnic-register, scsi-bnx2i, net-cnic, net-bnx2x, scsi-bnx2fc, net-bnx2, net-tg3, scsi-qla2xxx, scsi-hpsa, net-be2net, scsi-be2iscsi, scsi-lpfc820, net-qlcnic, scsi-mpt2sas, scsi-bfa, net-mlx4-en, net-nx-nic, ima-be2iscsi


10) To validate that your new profile in fact has the updated and new HP drivers, use the following command:

compare-esximageprofile -comparisonprofile ESXi-5.0.0-HP-05132012 -referenceprofile ESXi-5.0.0-20120504001-standard
As you can see in the screenshot below HP drivers were added to our custom image (hpnmi, hpcru, etc.) while others were upgraded (note the output does not show the full list of upgraded drivers). 

11) To create a customized bundle that you can use later, issue the following command:
export-esximageprofile -imageprofile ESXi-5.0.0-HP-05132012 -exporttobundle -filepath e:ESXi-5.0.0-HP-05132012.zip
12) To create a customized bootable ISO image, issue the following command:
export-esximageprofile -imageprofile ESXi-5.0.0-HP-05132012 -exporttoISO -filepath e:ESXi-5.0.0-HP-03132012.ISO

During the boot process of the custom ISO image you will see the profile name that you configured:

vSphere 5.0 Hardening Guide Draft Release

A few days ago VMware released a public draft of the vSphere 5.0 Hardening Guide. Unlike the vSphere 4.0 Hardening Guide that I’ve talked about before, this version only comes in an Excel Spreadsheet form. Personally, I think the PDF format of the 4.0 Guide is easier to read, but the spreadsheet is good for tracking and sorting of the various parameters. Since the hypervisor is the underpinings of your environment, it is extremely critical that it be secured. A “click next” installation of ESXi, vCenter, network configuration, and VM deployment is just waiting to be compromised.

You can find the vSphere 5.0 Hardening Guide – Public Draft here.

Creating Cisco UCS Customized vSphere 5.0 U1 Bootable ISO

UPDATE 2 5/15/2012: Looks like VMware/Cisco pulled the 5.0 U1 custom ISO installation media. So follow my blog post below to create your own.


UPDATE 1 4/23/12: Cisco released a customized vSphere 5.0 U1 installation ISO with all of their latest drivers. You can download it here under OEM Customized Installer CDs. The instructions below are still valid, and would be good for incorporating future updates in your ISO image. 

Some vendors, like HP, produce customized VMware installation ISOs that have all of their drivers integrated. This is a great time saver, but unfortunately Cisco does not provide customized vSphere 5.0 installation media with the very latest drivers. Starting with vSphere 5.0 VMware gave users a method to build their own installation media and include updated packages, such as drivers. The procedure below creates a bootable ISO image using the very latest ESXi build (5.0 U1 plus the latest patches). Your machine must be connected to the internet, as it will pull down the latest bundles in real time. You do NOT need to start with an offline depot.

1) Open a PowerCLI window with Administrator rights and type the following command:

Add-EsxSoftwareDepot https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml

2) At this point you can list all of the packages in the depot with the following command. A partial listing is shown below.
Get-EsxSoftwarePackage | select Name,Version,ReleaseDate | sort ReleaseDate
3)  Download the driver packages for your hardware from the following VMware URLs. Personally I would suggest you download all of them, so you don’t have to rebuild the image if you get a different server model.
4) You need to unzip each of the files that you downloaded, which will reveal another ZIP file and a VIB file, among others. We will be using the embedded bundle ZIP files. If you downloaded all of the drivers, unpacked them, and moved the bundled ZIPs to a single directory it should look like:
5) Repeat step 1 from above, but substitute the bundle zip files from the above screenshot. A sample is below:
add-esxsoftwaredepot E:\enic_driver_2.1.2.22-offline_bundle-564611.zip
add-esxsoftwaredepot E:\fnic_driver_1.5.0.7-offline_bundle-563432
6) Now you want to create a copy of the “latest” VMware profile and give it a unique name. To list all of the standard ESXi profiles use the following command:
Get-EsxImageProfile | Sort-Object “ModifiedTime” -Descending | format-table -property Name,CreationTime
7) You will notice that the latest profile has a date of 4/16/2012, but the build number is only 469512, which is far from the latest build. The latest build is actually ESXi-5.0.0-20120404001-standard. You can validate the latest patch build here. Update: Looks like the 4/16/2012 builds were a glitch, as the profile list on 4/17/2012 no longer showed the 4/16 builds and the latest was in fact the 3/16/2012 build.
8) Now you need to build a new profile based on the latest patch build. I called my new profile “ESXi-5.0.0-UCS-04152012“. The build profile name will be displayed during the boot selection process if you create an installable ISO file, so think about the name you use.
new-esximageprofile -cloneprofile ESXi-5.0.0-20120404001-standard
-name “ESXi-5.0.0-UCS-04152012”
9) After you create a new image profile, you now want to add the updated UCS drivers to the profile. To determine what software package name to use, look in your driver directory at the VIB filenames. The filename prefix (e.g. net-be2net) is what you will want to use when adding the driver files.

When I tried to update the scsi-megaraid-sas bundle it said it already existed, so I skipped that in example below.  To add the remaining drivers issue the following commands:

add-esxsoftwarepackage -imageprofile ESXi-5.0.0-UCS-04152012 net-enic
add-esxsoftwarepackage -imageprofile ESXi-5.0.0-UCS-04152012 scsi-fnic
add-esxsoftwarepackage -imageprofile ESXi-5.0.0-UCS-04152012 scsi-lpfc820
add-esxsoftwarepackage -imageprofile ESXi-5.0.0-UCS-04152012 net-ixgbe
add-esxsoftwarepackage -imageprofile ESXi-5.0.0-UCS-04152012 net-be2net
add-esxsoftwarepackage -imageprofile ESXi-5.0.0-UCS-04152012 scsi-qla2xxx
add-esxsoftwarepackage -imageprofile ESXi-5.0.0-UCS-04152012 net-qlcnic
add-esxsoftwarepackage -imageprofile ESXi-5.0.0-UCS-04152012 net-qlge
10) To validate that your new profile in fact has the updated and new UCS drivers, use the following command:
compare-esximageprofile -comparisonprofile ESXi-5.0.0-UCS-04152012 -referenceprofile ESXi-5.0.0-20120404001-standard
As you can see in the screenshot below two new drivers were added to our custom image (net-qlge and net-qlcnic) while four others were upgraded. So yes, our custom image did get injected with the new drivers.
11) To create a customized bundle that you can use later, issue the following command:
export-esximageprofile -imageprofile ESXi-5.0.0-UCS-04152012 -exporttobundle -filepath e:\ESXi-5.0.0-UCS-04152012.zip
12) To create a customized bootable ISO image, issue the following command:
export-esximageprofile -imageprofile ESXi-5.0.0-UCS-04152012 -exporttoISO -filepath e:\ESXi-5.0.0-UCS-04152012.ISO
13) If all goes well, and you use the exact same bundles that I did, when you install ESXi 5.0 you should see build 623860.