Archives for January 2011

XenDesktop 5 Machine Creation failure: VMware PVSCSI driver fix!

During my testing of Citrix XenDesktop 5 I ran across yet another bug, which set me back in my testing. Apparently if the VM template that you want machine creation services (MCS) to use has been configured with the VMware pvscsi controller, creating the VMs will fail when you generate a catalog.

The error that XenDesktop Desktop Studio will give you is:

The specified master VM snapshot could not be found. No machines have been created.

If you look in the Windows application log you see:

Provisioning scheme creation workflow operation failed : System.InvalidOperationException: VM not Found —> Citrix.HypervisorCommunicationsLibrary.InvalidVmConfigurationException: No disk controller found

As mentioned in my previous blogs, I always use the VMware pvscsi controller since it’s more efficient than the emulated legacy SCSI controllers. But, it seems that Citrix didn’t test this use case, since it fails miserably. The fix is to not use the VMware pvscsi controller, and use something like the LSI Logic SAS controller. But what if you already have a VM template built with the pvscsi controller, like me, and you don’t want to rebuild it because of a Citrix bug?

There’s an easy fix! While your VM template is running and using the pvscsi controller, open an elevated powershell and type the following:

Set-ItemProperty “HKLM:SYSTEMCurrentControlSetservicesLSI_SAS” -name “Start” -Value 0 -type “DWORD”

I then rebooted the VM (still with the pvscsi controller), shutdown the VM, then in vCenter changed the SCSI controller type to LSI Logic SAS. Next time the VM boots the LSI Logic SAS driver will be active at boot time and your VM won’t blue screen.

I hope Citrix can fix this bug in their next update for XenDesktop 5. It’s a bit disturbing that this scenario, just like the FIPS bug, wasn’t tested prior to GA.

Update: This issue is now fixed in XD5 SP1. Check out my post here.

Inject VMware drivers into Windows Install Discs

I like to perform unattended installations of my operating systems, like Windows Server 2008 R2 or Windows 7 using autounattend.xml so that requires that the image have the required drivers to recognize critical devices like mass storage hardware.

One of the performance optimizations that I always include in our Windows VM templates is the VMware paravirtual SCSI driver. This is a high performance mass storage driver that is optimized for virtual environments and gives you the best disk I/O performance. Unfortunately Microsoft does not include it out of the box on any OS install disk.

So you have two options:

1) Extract boot floppies from an ESXi update. I posted a blog on this a while back. Then you need to mount the virtual floppy image during the Windows install process and manually load the driver. This does not work for an unattended installation as Windows doesn’t automatically look for mass storage drivers.

2) Inject the mass storage drivers directly into the boot.wim file, so it is ‘baked in’ and then you can use an automated Windows install process all while using the high performance SCSI driver. I also inject the drivers into the main OS image (install.wim) so they are available to the operating system after installation.

Since option #2 is more automated, that is of course the option that I want to use. It’s a bit of a complicated process, but in the end in makes life easier. This process can work for other drivers as well, if you also want to use the ISO image on a physical server that has a unique mass storage controller, for instance.

Here is the basic process:

1) You will need to download and install the Windows Automated Installation Kit (WAIK). I used the latest version for Windows 7. Best practices is to install it on a x64 computer, so you can manipulate x64 images should you need to do that.

2) Perform a fully default installation of the WAIK. After the installation is complete, launch the Deployment Tools command prompt.

3) Mount the ISO image of your operating system. Navigate to the Sources directory and copy boot.wim to your computer, say on the D: drive.

4) Create a folder on your D: drive called Drivers. VMware provides both 32-bit and 64-bit pvscsi drivers, and you must use the right one depending on what CPU architecture you are injecting the drivers into. The easiest solution is to leverage an existing 32-bit or 64-bit VM running on vSphere and go into the C:\Program Files\VMware\VMware Tools\Drivers\pvscsi and copy the files in there to D:\drivers.

To verify the supported architecture of the drivers, open the pvscsi.inf file and scroll down to the [Manufacturer] section. If you see NTamd64, you have 64-bit drivers. If you see NTx86, you have 32-bit drivers. The 64-bit pvscsi.sys file is also larger than the 32-bit version (40K vs 35K for vSphere 4.1).

Do not inject both drivers into your image; only use the matching driver for the OS you are modifying. Server 2008 R2 is 64-bit only, whereas you have a choice with Windows 7.

5) Create a folder on the D:\ drive called Mount.

6) In the deployment tool command prompt type:

dism /get-wiminfo /wimfile:d:\boot.wim

And look at the index numbers. This is key!  You must select index 2, the Windows setup image. If you inject the drivers into index 1, the Windows setup routine will NOT see them and you will be banging your head against the wall.

Microsoft DISM

7) In the deployment tool command prompt type:

dism /Mount-Wim /WimFile:D:\boot.wim /Index:2 /MountDir:D:\mount

8) You should see an operation successful if the image mounted properly.

9) In the deployment tool command prompt type:

dism /image:D:\mount /Add-Driver /driver:d:\drivers\pvscsi.inf

10) You should see an operation successful if the driver was injected properly.

11) Commit the changes and unmount the image with this command:

dism /unmount-wim /mountdir:d:mount /commit

12) At this point I also inject the same drivers into the sourcesinstall.wim file, so the drivers are present before VMware tools gets installed. You follow the same procedure of mounting the WIM, adding the drivers, and then committing the changes. Most install.wim files have multiple images in them, so execute step #6 to list all of the images in the WIM and select the right one(s) to inject the drivers into.

13) Create a backup of your OS ISO file, and then use your favorite ISO editing tool (such as UltraISO) and replace the boot.wim and install.wim files in the Sources directory.

Now you can use the new ISO image to create a VM which uses the pvscsi controller for the boot drive, and you won’t need to manually mount a virtual floppy drive to use a pvscsi boot disk.

If you also want to inject the VMware vmxnet3 drivers, you can use the same procedures. But where do you find the drivers? On a VM running on vSphere navigate to C:\Program Files\VMware\VMware Tools\Drivers\vmxnet3. Copy all of the driver files to the same driver folder from above, and run step #9 a second time, but specifying the vmxnet3ndis6.inf file. Unlike the pvscsi driver, the VMXNET3 driver supports both 32-bit and 64-bit versions with the same set of files.

I would not recommend including the VMware mouse driver (vmmouse.inf) as it’s an unsigned driver and you will get an “Error 50 The request is not supported” when you try and commit the image changes, unless you use the /forceunsigned switch when adding the driver. VMware tools will of course install the mouse driver, so just leave this driver out.

I also wrote a blog about incorporating these drivers into the Windows Recovery environment. This can be very useful if a server goes belly up and you want to try and repair it. You can check that out here.

Finally…strong ESX 4.1 root passwords. SHA512 baby!

Historically VMware has not used the strongest hashing algorithms to store root passwords on ESXi or ESX hosts. And to make matters worse, ESX/i 4.1 had a major security hole that was open for over four months, which you can read about here. The short story is that ROOT passwords in ESX/i 4.1 were only authenticated up to 8 characters. The screw up on VMware’s part was only using DES (not even 3DES) for the password encryption. DES is a joke, and even 3DES is not considered secure. One workaround for this major hole was to use MD5 hashing, but even that is not considered secure.

A couple of days ago VMware pushed a KB article how to increase the password encryption strength by using SHA512. SHA512 is considered secure and is very well respected. So I applaud VMware in publishing an article on how to enable this feature. I am still shocked it took VMware four months to publish a patch to plug the 8 character password hole.

I can only hope in 4.1 U1 and future releases that SHA512 is used by default. Having to hack system files to increase security is not my idea of a fun time.

Free Microsoft security tool: EMET for you!

Yesterday I stumbled upon a blog post about a free Microsoft security tool. No matter how secure you think your system is, it can always be more secure. So I eagerly read Ed Bott’s article about Microsoft’s EMET (Enhanced Mitigation Experience Toolkit). I quickly installed it, configured it for maximum protection and rebooted my computer. So far, so good.

I won’t rehash Ed’s review and tutorial, but I would strongly encourage everyone to download the tool and experiment with it. The tool let’s you enforce application configuration options such as DEP, SEHOP, ASLR, EAF, NullPage and HeapSpray protections.

If my experimentation goes well, then I will probably recommend we test it at work possibly incorporate it into our Windows 7 image for additional system protection.

Download v2.0 here and it works on the following operating systems:

Client Operating Systems
• Windows XP service pack 3 and above
• Windows Vista service pack 1 and above
• Windows 7 all service packs

Server Operation Systems
• Windows Server 2003 service pack 1 and above
• Windows Server 2008 all service packs
• Windows Server 2008 R2 all service packs

Update: I found one program that crashes at launch time with the settings as configured above: UltraISO The solution is to configure the main DEP setting for ‘Application Opt Out’ then configure an application specific exclusion for the ultraiso.exe and uncheck only the DEP box. The application exclusions for DEP have no effect if the main DEP option is set to Always On.

Exchange 2010 and vSphere 4.x Best Practices

Virtualization has taken off like wild fire, and now organizations are in the process of virtualizing tier-1 applications like Exchange 2010. However, sizing and designing Exchange 2010 for a virtualized environment requires some additional care and thought versus a traditional physical server deployment. Critical sizing requirements like memory, vCPUs, and storage performance/type have unique guidelines when deployed on vSphere 4.x.

VMware has published a VERY lengthy guide for Exchange architects that covers all of the unique aspects for sizing and designing your Exchange 2010 environment for vSphere 4.x.  I highly encourage anyone doing such a deployment to thoroughly read the guide, found here. The whitepaper includes complex enterprise configurations supporting 16,000 users with 4-node DAG clustering.

Some example key recommendations include:

  • Only allocate multiple vCPUs to a virtual machine if the anticipated Exchange workload can truly take advantage of all the vCPUs.
  • If the exact workload is not known, size the virtual machine with a smaller number of vCPUs initially and increase the number later if necessary.
  • For performance-critical Exchange virtual machines (i.e., production systems), try to ensure the total number of vCPUs assigned to all the virtual machines is equal to or less than the total number of cores on the ESX host machine.
  • Don’t over commit memory
  • Spread the heavy I/O systems across several LUNs
  • Use eagerthickzero (EZT) VMDK files
  • Use VMXnet3 driver
  • Use PVSCSI adapter

Additional Exchange 2010 and vSphere resources:

Scale-Out Performance of Exchange 2010 Mailbox Servers
Scale-Up Performance of Exchange 2010
Exchange 2010 Disk I/O on vSphere
Dell Exchange 2010 on vSphere 4
Exchange 2010 on vSphere
DAG performance on vSphere 4.1
Mailbox VM I/O Sizes