Archives for February 2013

New VMware Cisco UCS Drivers for vSphere 5.x

new VMware Cisco UCS Drivers for vSphere 5.x are now out for the fnic and enic drivers. The fnic and enic drivers are for the Cisco VIC cards, which come in a few different flavors. These are the most common FCoE cards used in the UCS blades. They are now at version and, respectively. The enic has the following enhancements:

  •     Improved RX checksum offload support
  •     Fixed PSOD in enic_reset in UPT mode
  •     In UPT mode, allow updating the provising info of a Virtual Interface.
  •     Add support for Cisco VIC 1240
  •     Improved WQ/RQ error recovery in UPT mode
The fnic has the following enhancements:
New Features:
  • FIP VLAN Discovery support
  • FC Event Tracing

Bugs Fixed (since fnic version

  • Corrected failure to login to SAN fabric which could randomly occur when FIP is enabled and the adapter is connected to an NPV-enabled switch advertising multiple FCFs (CSCtr73717, CSCtk14373).
  • Serialized interaction with adapter firmware to correct occasional fnic and/or device offline behavior when the connected Veth port on the switch is shut (disabled).  This was a general issue that could also occur in other scenarios where the driver was required to issue a reset command to the adapter firmware (CSCtz63473, CSCty31268).
New Hardware Supported:
  • Cisco UCS Virtual Interface Card 1240

The Cisco interoperability matrix that shows these drivers are validated with UCS 2.1(1) and 2.1(1a) firmware. You can find the whole matrix here. To download the new drivers, go to:

My VMware > Product & Downloads > All Downloads > VMware vSphere and click on the Drivers and Tools tab.

Cisco is very slow to release customized ESXi Installation media, so you can check out my post here to roll your own ESXi 5.0 updated media. This would allow you to build a ESXi 5.0 U2 media with the latest drivers baked in. As of this writing, Cisco has only released 5.0 U1 custom ISO media.

You can push the enic and fnic driver updates via VUM. To do that you will need to unzip the parent archive (the one you download) then import the “offline_bundle” ZIP into VUM. You can then create a custom baseline in VUM to push the enic/fnic drivers and validate compliance.

VMware Posts new vCenter SSO KB Article

For those of you trying to install and maintain vSphere 5.1, you probably will run into some vCenter SSO issues at one point or another. One of the big problems with SSO not using the very friendly Microsoft ODBC connector, is post-install reconfiguration of database parameters. The JDBC database interface is, shall we say, far from user friendly and a source of frustration.

VMware has added a new KB article, among the dozens for the SSO service, to address more issues related to changing the database configuration. Specifically, the KB article tells you how to:

  • Modify the SQL server port
  • Move the SSO database to another server
  • Modify the RSA_USER password

You can check out the full KB article here. If you haven’t dabbled with vSphere 5.1 and want a weekend of entertainment, check out my 15-part install guide series here.

Voting for Top Virtualization Blogs in Full Swing

For a few years now has been holding a poll for voting for the top virtualization blogs. Of course rock stars like Duncan and Frank get very high rankings, as they should. The sheer number of Virtualization blogs out there is really amazing. This year my blog made it on to the voting choices for independent bloggers. So I would encourage everyone to vote here for your favorite blogs. Voting goes until 3/1, so don’t wait and vote now!

February 2013 HP Service Pack for ProLiant released

For those of you with HP ProLiant servers in your environment, you know that on a quarterly basis HP releases a new “HP Service Pack for ProLiant”, which is a combo ISO of firmware, drivers, and tools for your ProLiant rackmount and blade servers. It’s very important to keep firmware up to date, as it can contain critical security updates, stability updates, and compatibility fixes for new operating systems, such as Windows Server 2012 or ESXi 5.0 U2.

What is also interesting, and maybe I just wasn’t paying attention before, is that HP offers non-bootable subsets for specific operating systems and OSes. For example, you can download the BladeSystem Microsoft Windows Server pack ISO image separately. Personally I’d just do the full ISO, but maybe you want something a bit more portable.

In VMware environments it’s particularly important to be using validated configurations, which HP calls a recipe. That’s a combination of HP firmware, HP drivers, HP agents, and ESXi base image. Drifting from the supported recipes can cause problems, particularly in the BladeSystem. I have a link below to the VMware recipe PDF. Also new to this release is enhanced online deployment of firmware on ESXi hosts.

Finally, HP has provided some support Windows Server 2012 on limited G6 models. Windows Server 2012 is very different from Windows Server 2008 R2, so don’t even try and shoehorn it on a non-supported HP server if you want everything to work such as iLO.

HP Service Pack for ProLiant

Full Documentation
Release Notes
Content Report
Server Support Guide
February 2013 VMware Firmware and Software Recipe

Enhancements to the 2013.02 SPP include:

  • Contains drivers and firmware for newest HP ProLiant Gen 8 options
    • HP Ethernet 1Gb 4-port 366FLR Adapter
    • HP Ethernet 10Gb 2-port 560M Adapter
    • HP Ethernet 10Gb 2-port 560FLR SFP+ Adapter
    • HP Ethernet 10Gb 2-port 530T Adapter
  • Provides online deployment of VMware ESXi 5.0 and VMware vSphere 5.1 firmware smart components for HP ProLiant servers so that only 1 reboot is required for updates to take effect; thus increasing uptime. Supported new components are:
    • HP ProLiant Gen 8 and G7 BIOS System ROMS
    • HP Integrated Lights-Out 3 and 4
    • HP Smart Array Controllers – P2XX, P4XX, P8XX
    • Select Hard Disk Drives for HP ProLiant Gen 8 and G7 servers
  • Contains latest operating system support for:
    • Red Hat Enterprise Linux 5.9
    • VMware ESXi 5.0 U2
  • Extends Microsoft Windows Server 2012 support to the following HP ProLiant G6 servers:
    • HP ProLiant BL460c G6 Server
    • HP ProLiant DL380 G6 Server
    • HP ProLiant DL360 G6 Server
    • HP ProLiant ML350 G6 Server
  • Contains PXE Directory information and sample files about PXE booting the SPP ISO for improved qualification cycles, resource usage, maintenance windows, and downtime
  • Contains the latest release of HP Smart Update Manager, HP SUM 5.3.5.
    • Added ability to allow users to use root credentials to deploy firmware components and rpms to Linux systems when logged in with a user account
    • Performance improvement to Add New Target screen: Added Server / Operating System selections to the Target Type drop down list
    • Updated reports to include driver and agent information in addition to NIC /CNA/FC HBA driver and firmware information for Windows and Linux operating systems
    • Reduced memory usage and improved performance when generating repositories

vCenter 5.1 Installation: Part 15 (ESXi SSL certificate)

Welcome to a post which will let you easily update your VMware ESXi host SSL certificates. Last year when I was writing my 15-part vSphere 5.1 installation and configuration series I didn’t include instructions on how to replace the ESXi SSL certificate. That process hasn’t changed for ages, so I put it on the back burner. Now the time has come to show you part 15 of the vSphere 5.1 install series, which is a semi-automated method to replace your ESXi SSL certificates.

vSphere 5.1 has relaxed the ESXi host certificate requirements a bit by not requiring the dataencipherment and nonrepudiation key properties, which previous versions required. However, I’ve included them in my script in case you have any 5.0 or 4.x hosts. It won’t hurt to have these properties enabled on a ESXi 5.1 certificate though.

Basic requirements for the script are:

  • ESXi 4.x or 5.x host(s)
  • OpenSSL installed (0.9.8 or higher, 32-bit or 64-bit)
  • Online Windows Enterprise Certificate Authority (2008 R2 or higher recommended)
  • vSphere CLI (I’ve tested with 5.x)
  • Properly configured Windows Certificate template (see blog post here)
  • DNS “A” record for your ESXi host
  • Existing D:\Certs directory

If your ESXi host is already managed by vCenter, the HA agent can get very confused by the new SSL certificate thumbprint. I would strongly suggest you first put your host in maintenance mode, remove it from the vCenter inventory, update the SSL certificate, reboot the ESXi host, then re-add it to the vCenter inventory.

Since the script includes the creation of the CSR, you will need to modify the basic attributes of the SSL certificate variables, as shown in red below. Once you’ve modified the variables for your environment, just open an elevated VMware vSphere CLI prompt (not just a regular command prompt) and type the script name followed by the FQDN of your ESXi server.

The script will create a CSR, submit the CSR to your MS online CA, download the new certificate, and upload it to your ESXi host. You will be prompted twice to enter the root credentials of your ESXi host. Now simply reboot your ESXi server, re-add it to your inventory, and you are done! Can’t get much easier than this folks. In my case the CA certificate life is shorter than what my certificate template requested, so I got a warning message.

The script has some error checking, but it’s not super robust. You might get tripped up on the Cert_Template and CA_Name variables, so let me explain them. The Cert_Template is “template name” NOT the “Template display name”. While they are the same in my example, the “Template name” usually has spaces removed.

The CA_Name is NOT simply the hostname of your CA, but is the hostname of the CA AND the CA name which was configured during the CA installation process. You can find the CA name by opening the Certification Authority MMC and looking at the left pane.

Congratulations! You have now made it through the whole vCenter 5.1 installation process using trusted SSL certificates. Probably took way longer than you expected, and much more tedious than it should be. I would hope in vSphere v.Next that they overhaul what seems like a complete mess of internal handing of certificates. How about certificate revocation? How about the ability to completely remove a compromised certificate from all keystores?

@Echo off
 REM Change these variables for your environment.
 REM Do not put spaces between the = sign
 REM SSL Certificate Properties
 REM Country name must be exactly two letters

 Set countryName=US
 Set state=CA
 Set locality=San Diego
 Set organization=Contoso

 REM Certifiate Authority Properties
 Set Cert_Template=VMware-SSL
 Set CA_Name=D001DC02\Contoso-D001DC02-CA

 REM Existing parent path for the ESXi certificate directory
 Set Cert_Path=D:\certs

 REM -- Don't change anything below here --

 set ESXiConfig=esxi.cfg
 if [%1]==[] ; GOTO :ERROR
 if not exist %Cert_Path%\ESXi mkdir %Cert_Path%\ESXi
 if exist "D:\program Files (x86)\VMware\Vmware vSphere CLI\bin" Set CLI=D:\program Files (x86)\VMware\Vmware vSphere CLI\bin
 if exist "C:\program Files (x86)\VMware\Vmware vSphere CLI\bin" Set CLI=C:\program Files (x86)\VMware\Vmware vSphere CLI\bin
 if exist "c:\OpenSSL-Win32\bin\openssl.exe" Set OpenSSL_BIN=c:\OpenSSL-Win32\bin\openssl.exe
 if exist "c:\OpenSSL-Win64\bin\openssl.exe" Set OpenSSL_BIN=c:\OpenSSL-Win64\bin\openssl.exe
 FOR /F "Tokens=1 delims=." %%A IN ("%1") DO SET Hostname=%%A
 Echo [ req ]
 Echo default_bits = 2048
 Echo default_keyfile = rui.key
 Echo distinguished_name = req_distinguished_name
 Echo encrypt_key = no
 Echo prompt = no
 Echo string_mask = nombstr
 Echo req_extensions = v3_req
 Echo [ v3_req ]
 Echo basicConstraints = CA:FALSE
 Echo keyUsage = digitalSignature, keyEncipherment, dataEncipherment, nonRepudiation
 Echo extendedKeyUsage = serverAuth, clientAuth
 Echo subjectAltName = DNS:%1, DNS:%hostname%
 Echo [ req_distinguished_name ]
 Echo countryName = %countryName%
 Echo stateOrProvinceName = %state%
 Echo localityName = %locality%
 Echo 0.organizationName = %organization%
 Echo commonName = %1
 ) >%Cert_Path%\ESXi\%ESXiConfig%
 %OpenSSL_BIN% genrsa 2048 > %Cert_Path%\ESXi\rui.key
 %OpenSSL_BIN% req -out %Cert_Path%\ESXi\rui.csr -key %Cert_Path%\ESXi\rui.key -new -config %Cert_Path%\ESXi\%ESXiConfig%
 certreq -submit -f -config "%CA_NAME%" -attrib "CertificateTemplate:%Cert_Template%" %Cert_Path%\ESXi\rui.csr %Cert_Path%\ESXi\rui.crt
 "%CLI%\" --server %hostname% --put %Cert_Path%\ESXi\rui.key /host/ssl_key
 "%CLI%\" --server %hostname% --put %Cert_Path%\ESXi\rui.crt /host/ssl_cert
 Exit /B
 Echo Please specify ESXi server FQDN (e.g.

Safely virtualizing Windows Server 2012 Active Directory via Generation-ID

Windows Server 2012 generation ID is a great new feature that will allow use to safely virtualize a domain controller, on specific hypervisors. One of the really great features that hypervisors have had for ages is the ability to perform snapshots, then roll back to a prior state with a click of a mouse. Invaluable feature in both the lab, and in production.

I know during all my (failed) vSphere 5.1 installs I practically wore out the revert to snapshot button in vCenter. But, there is at least one class of VMs that you almost NEVER want to roll back from a snapshot with, those which are vector-clock synchronized software such as Active Directory.

Why is rolling back AD bad? I mean why is rolling back AD *REALLY* bad? Microsoft has these little things called USNs, or Update Sequence Numbers. A USN is an Active Directory database instance counter which gets incremented each time an update to AD is made. USNs are unique to each DC, and use a monotonically increasing value. USNs are used to determine what changes need to be replicated to other DCs.

When you revert to a snapshot a USN rollback occurs. What can happen if a USN rollback occurs? Lots of bad things, such as missing AD objects, wrong security group memberships, passwords are reset, and re-appearing AD objects. Also, DCs that are rolled back may accumulate many changes which never get replicated to other DCs. In short, the AD consistency of your forest is SHOT.  Starting with Windows Server 2003 SP1 and later, an event log ID 2095 is generated if a USN roll-back is detected, but it’s up to you to fix the mess. Microsoft has a great KB article here that goes into a lot more detail.

What has Microsoft done in Windows Server 2012 (and Windows 8) to address this problem? They’ve introduced a safeguard called a VM-Generation ID, which can be implemented by any hypervisor. This generation ID can be used by applications and operating systems to detect if a virtual machine has been rolled back in time, and take appropriate measures.

So what happens when AD detects that the Generation IDs have changed? First, it dumps the RID pool, then does a non-authoritative synchronization of the SYSVOL folder. AD replication is then re-established to other DCs, to bring the reverted DC back into a consistent state with the rest of the forest.

Sounds great right? Well it is, but only a very limited number of hypervisors support VM-Generation ID. As of this writing the hypervisors are Hyper-V 3.0, vSphere 5.0 U2, and vSphere 5.1. Since a USN rollback is quite unpleasant, you of course want to verify that WS2012 and your hypervisor are playing nice and using the Generation-ID feature. If you look in the Directory Service event log, you will see event ID 2168 and 2172. In the screenshots below they have the same Generation-ID, since the VM was not reverted to a previous snapshot.

To test out this new feature I fired up my vCenter 5.1 web console and took a snapshot of my WS2012 domain controller. After the snapshot completed, I created a new group on another DC, then reverted the WS2012 DC back to my snapshot. Let’s look in the event viewer and see what happened:

Yes, AD realized it was reverted back to a prior snapshot…

Microsoft even tells you that snapshots are not backups, and silly, use an AD aware backup program to restore AD.

And now life is almost good…

Let’s freshen up FRS a little bit while we are at it…

Nothing like a new database to start off the day with…

A touch of USN cleanup…

And a few minutes later, everything is back in sync! As you can see from the screenshots, Microsoft is very verbose in the logs on exactly what is happening and why. In a very large forest with a lot of DCs the recovery process could take longer.

So under what circumstances does the Generation-ID change and not change? Here’s a list:

Generation-ID NOT changed when:
VM is paused or resumed
VM is rebooted
VM host reboots
VM is vMotioned/Live Migrated

Generation-ID IS changed when:
VM starts executing a snapshot
VM is recovered from a backup
VM is failed over in a disaster recovery environment
VM is imported, copied, or cloned

This feature alone should be a huge driver for deploying WS2012 based DCs on all of your hypervisors. Never thought I’d say this..but happy snapshotting your domain controllers! For even more detailed information on virtualized domain controllers, Microsoft has a great series of articles here you can read.

P.S. This feature does NOT work with array-based snapshots. The hypervisor tracks and creates the new Generation-IDs. So DO NOT revert a domain controller back to a prior state by reverting to a previous snapshot that your array created vice your hypervisor. With the forthcoming VVOLS in vSphere .Next, Generation-ID could be supported with hardware-snapshot offloads but we will have to wait and see if that’s the case.

VMware Whitepaper on CPU Scheduling Performance

In a virtual environment CPU scheduling is extremely important. You very likely will have more vCPUs than pCores, so the hypervisor has to play some “tricks” to fairly schedule VMs among the finite hardware resources. With each version of vSphere VMware has tuned and optimized CPU scheduler performance.

VMware design goals for CPU scheduling are fairness, throughput, responsiveness and scalability. So what has VMware done under the covers in vSphere 5.1 to be even more efficient? Glad you asked! A few days ago VMware released a brand new whitepaper titled The CPU Scheduler in VMware vSphere 5.1. In it, you get to read about:

  • Removing zombies
  • Proportional shared-based algorithms
  • Relaxed co-scheduling
  • Load balancing
  • Hyper-threading Policy
  • NUMA Scheduling Policy
  • Wide Virtual Machines
  • vNUMA Best Practices
  • Scheduler experiments and resulting data

In conclusion the whitepaper states:

The ESXi CPU scheduler achieves fair distribution of compute resources among many virtual machines without compromising system throughput and responsiveness. Relaxed co-scheduling is a salient feature that enables both correct and efficient execution of guest instructions with low overhead. The ESXi CPU scheduler is highly scalable and supports very big systems and wide virtual machine.

vSphere 5.1 optimizes the load-balancing algorithm introduced in 5.0. It results in noticeable reductions in CPU scheduling overhead. A policy change on hyper-threaded systems enables out-of-the-box performance of 5.1 exceeding that of a tuned version of vSphere 4.1. No special tuning is required to achieve the best performance for most common application workloads.

If you are uber geeky, then check it out here.