Archives for November 2009

VMware vSphere 4.0U1 lacks broad Server 2008 R2 Support

I was very disappointed when I was looking over the vSphere 4.0 Update 1 compatibility matrix. I was hoping for wide spread Server 2008 R2 support for all roles. When in fact, the only support for Server 2008 R2 is as a guest VM. The matrix doesn’t even show vSphere client support for 2008 R2. Given that Update 1 was released months after R2 went gold, I’m quite disappointed.

VMware really needs to align their releases better with the latest Microsoft operating systems. My project is planning to upgrade from 2003 directly to 2008 R2, but companies like VMware throw monkey wrenches in the works when they lag by many months Microsoft releases. 95% of VMware runs Windows so it would make sense they put more effort into staying current. So now we will be forced to support three major operating systems instead of just two.

One change for Update 1 that is welcomed is that Orchestrator is now supported on SQL 2008, finally! You can check out the entire compatibility matrix for vSphere 4.0 Update 1 here.

Oh, and even their support for Windows Server 2008 SP2 is spotty. VUM and Orchestrator are not supported on SP2. Yes, they can’t even muster support for SP2 which was released in May, before the launch date of vSphere. Very poor support, IMHO.

VMware vCenter 4.0 Update Manager SSL Certificates

You can check out the improved, and officially supported method here. This works for vCenter 4.0 and 4.1.

After a significant effort of research and trial and error, it appears I have gotten VMware Update Manager (VUM) 4.0 Update 1 to use SSL certificates generated from an internal Microsoft CA. This completes my quest to replace all SSL certificates that vCenter 4.0 U1 and ESXi 4.0 hosts use. This method is somewhat of a ‘hack’, but so far everything seems to be working well. I haven’t tried this with the gold release of vCenter Update Manager 4.0, so I can’t comment if this procedure works or not.

In my scenario I have VUM installed on a separate server from vCenter. This is a recommended best practice in larger environments. But I’d think this method works equally well with vCenter and VUM co-located on the same server. In that case, you should be able to re-use the certificates you generated for your vCenter server since they have the same FQDN.

1. Read my article about vCenter SSL certificate generation.
2. Perform the exact same steps to generate a certificate (steps 1-9) but use the FQDN of the VUM server, if it’s on a dedicated server.
3. Find the SSL directory path for Update Manager on your system. In my case it’s located at:
D:\Program Files (x86)\VMware\Infrastructure\Update Manager\SSL
4. Compress all of the existing files in the SSL directory into a .ZIP for safe keeping.
5. Stop the VMware Update Manager Service.
6. Replace rui.crt, rui.key and rui.pfx with the new certificates.
7. De-Install VUM. Yes, remove it.
8. Re-install VUM using the exact same settings as your first install, and use the existing database.
9. Launch the vSphere client and open the vCenter Server Status window.
10. Verify everything has a green check, including all VMware Update Manager components.

If you see any errors about health service, or get weird login errors when launching the vSphere Client, something is broke. The key to this whole process is de-installing and re-installing VUM. This resets some credentials, the thumbprint in the ADAM instance, and uses the new certificates you installed. VMware should really make this easier!

You should also be able to pre-position the SSL certificates into the proper directory pior to ANY VUM installation, and it will use them. That would avoid a de-install and re-install. Depending on your installation parameters and whether you are x86 or x64, the directory path will vary.

vCenter Server 4.0 SSL Certificate Generation – Fixed

In the release notes of vCenter 4.0 Update 1 it mentions that the SSL thumbprint problem is solved. The bug in 4.0 caused various thumbprint entries buried deep in the ADAM LDAP database to not be updated when the certificates were replaced. That caused all kind of issues. Today I verified that vCenter 4.0 Update 1 solves the thumbprint problem.

Here’s the procedure I used to generate and install the vCenter certificates. Note that this doesn’t take care of the VUM SSL certificates. I’m still researching how to properly update those. Like my previous blog on updating ESXi SSL certificates, you need to install Open SSL. See this post, and follow the first three steps before proceeding.

  1. Execute: c:opensslbinopenssl req -new -nodes -out rui.csr
  2. At this point OpenSSL will prompt you for various parameters. Enter any information you wish, but make sure the Common Name is the FQDN of your vCenter server (.e.g. Do not set a password.
  3. Use NotePad and copy the contents of rui.csr to the clipboard.
  4. Navigate to your Microsoft CA and select the option called something like “Submit a certificate request by using a base-64-encoded CMC….”
  5. On the Saved Request screen paste the contents of the clipboard, and change the certificate template to Web Server.
  6. Submit the request, then download the Base-64 encoded certificate (not the certificate chain). I saved the file as rui.cer into the c:OpenSSLCerts diretory.
  7. Rename privkey.pem rui.key
  8. Rename rui.cer (from step 6) to rui.crt
  9. Note in the following command you must use testpassword, not your own password. C:opensslbinopenssl pkcs12 -export -in rui.crt -inkey rui.key -name rui -passout pass:testpassword -out rui.pfx
  10. In Explorer cut and paste the appropriate path into the address bar:

Server 2008/R2:

C:UsersAll UsersApplication DatavmwareVMware VirtualCenterSSL

Server 2003/R2:

C:Documents and SettingsAll UsersApplication DataVMwareVMware VirtualCenterSSL

11. Highlight all files, right click, and Send to a Compressed Folder named backup

12. Stop the VMware Virtual Center Server service.

13. From the C:Opensslcerts directory copy rui.key, rui.crt and rui.pfx to the SSL directory shown above and overwrite all existing files.

14. Restart the VMware VirtualCenter Server and Vmware VirtualCenter Management WebServices services. Verify they start.

15. Browse to the HTTPS FQDN of the vCenter Server and verify the new certificate is being used.

You should update the vCenter SSL certificate PRIOR to creating any customization specifications. If you update the certificate afterwards, you will need to re-do your customization specifications since they rely on encryption parameters that get changed when you update the SSL certificate.

vSphere ESXi SSL mystery solved

For quite a while I’ve been trying to get SSL certificates uploaded to an ESXi 4.0 host which were issued by our internal Microsoft CA. Unfortunately I ran into issues, the last being that adding an ESXi 4.0 host to vCenter 4.0 with the certificate would die at 80%.

After additional testing, I now have a procedure which seems to work perfectly for ESXi 4.0 and 4.0 Update 1 hosts. But you must follow the steps exactly as written, or it may not work. It even works with a certificate from a Windows Server 2008 R2 CA using the new sha512ECDSA (elliptic curve digital signature althorithm with secure hash algorithm 512) NSA Suite-B certificates.

1. Download the Windows OpenSSL binaries, either 32-bit or 64-bit. Remember to install the Visual C++ binaries on prior to OpenSSL.

2. I create a directory called Certs in c:OpenSSL just to keep certificates separate.

3. Cd c:opensslcerts

4. c:opensslbinopenssl genrsa 2048 > rui.key

5. c:opensslbinopenssl req -new -key rui.key > rui.csr

6. At this point OpenSSL will prompt you for various parameters. Enter any information you wish, but make sure the Common Name is the FQDN of your ESX server (.e.g. Do not set a password.

7. Use NotePad and copy the contents of rui.csr to the clipboard.

8. Navigate to your Microsoft CA and select the option called something like “Submit a certificate request by using a base-64-encoded CMC….”

9. On the Saved Request screen paste the contents of the clipboard, and change the certificate template to Web Server.

10. Submit the request, then download the Base-64 encoded certificate (not the certificate chain). I saved the file as rui.cer into the c:OpenSSLCerts diretory.

11. Optional: Perform verification of the certificates per my blog post here.

12. c:opensslbinopenssl x509 -in rui.cer -out rui.crt

13. Open a VMware vSPhere CLI command prompt (if you don’t have RemoteCLI installed, download it here.

14. –server ESXhostname –put c:opensslcertsrui.key /host/ssl_key

15. –server ESXhostname –put c:opensslcertsrui.crt /host/ssl_cert

16. Reboot the ESXi host and wait five minutes after the ESXi console appears. Use a web browser and navigate to your ESXi host. In the address bar of your browser open the properties of the SSL certificate and verify it was issued by your CA and is not the self-signed certificate.

17. Add your ESXi host to vCenter, and it should NOT get stuck at 80% and fail.

If you run into problems, make sure on the ESXi console that the hostname is configured with a FQDN. From the ESXi console you can also view the management agent logs and look for any SSL related errors.

Next up is changing the vCenter server SSL certificates, as well as VUM. This was broken in 4.0, so hopefully Update 1 has solved these problems. Expect a blog update on this and a procedure, if I find one that works.

OpenSSL Certificate Verification – for VMware

During my process of testing the SSL functionality in vSPhere 4.0 ESXi Update 1, I found a helpful set of OpenSSL commands. OpenSSL is used to generate the certificate signing request (CSR), which is then submitted to a CA which in turns provides you a certificate you can install on your ESXi host.

Unfortunately I was running into problems and when I was looking at the ESXi logs, as I was seeing a X509_check_private_key error 0B080074. Upon further investigation I found an OpenSSL command that verifies whether the private/public key pair you have are a match, and if they match your CSR.

To compare your private and public keys for ESXi, use the following OpenSSL commands. You run these commands on a Windows computer where you’ve downloaded and installed the OpenSSL binaries.

openssl x509 -noout -modulus -in rui.crt : openssl md5
openssl rsa -noout -modulus -in rui.key : openssl md5

Note: For some reason the pipe symbol was not being displayed properly in the blog. So change the : to the pipe symbol or the command won’t work.

If the MD5 hashes both commands return match, then you have matching key pairs. If you get different MD5 hashes, you have a mismatch of some type. If you are unsure which certificates go with a particular CSR, execute the following command and match the MD5 with the private/public key pairs from the previous commands.

openssl req -noout -modulus -in rui.csr : openssl md5

Note: Same information applies to the : in this command. Change it to the pipe symbol.

Rui.crt, rui.key and rui.csr are the typical names of the certificates and requests used for ESXi. But the file names don’t matter as long as contents are the proper types (private key, public key, signing request). A Windows CA will typically use a default extension of .cer for the certificate it issues, which I rename to .crt.

I would recommend that you run the x509 and RSA hash matching commands for every certificate you issue. Should you upload a mismatched pair to an ESXi host, you have to blow away the whole ESXi management configuration and start over.

VMware vSphere 4.0 Update 1 released

The much anticipated first update for VMware vSphere 4.0 is upon us. I’ve been awaiting this update with eager anticipation since we’ve uncovered some issues with 4.0 in our deployment. If you want to read the release notes for update 1, you can check them out here.

If you have already deployed 4.0 and just want to download the ESX/i update, you can find them here. You should also download vCenter 4.0 update 1, which you can find here. That same link has the full ESX/i installable CD ISO images. Remote CLI 4.0 Update 1 is here. And finally, PowerCLI 4.0 Update 1 is here. Whew!

If you are doing a fresh install of ESXi on HP, Dell or IBM hardware then I’d suggest you wait for the vendor to release their custom version with the hardware heatlh monitor providers built-in. HP usually has quick turnaround on releases, so I’d expect them to have an updated ESXi installable CD in the next two weeks, maybe less.

I’m going to do some testing this weekend and see if the major issues we’ve run into have been resolved. Look for an updated blog post in a couple of days.

Major features of Update 1 include:

  • Windows 7 and Server 2008 R2 guest OS suppport. vSphere client can now be installed on these systems as well.
  • Enhanced Clustering support for Microsoft Clusters – MSCS VMs can now be a part of a DRS/HA cluster if the MSCS VMs have HA/DRS disabled.
  • Paravirtualized SCSI controller is now supported for the boot volume of Server 2003/2008.
  • Improved vNetwork Distributed Switch performance.
  • Support for VMware View 4.0

The list of bug fixes is monumentally long, so I won’t post them here. My major issues with 4.0 were with SSL certificates, lockdown mode, and Windows 7 support.

Office 2010 Beta Downloads now available!

For all of you TechNet or MSDN subscribers, the latest beta of Office 2010 is now available for download. Just sign in to your TechNet/MSDN account and download! Remember, Office 2010 now comes in native 64-bit support, as well as legacy 32-bit versions. I’ve been running the pre-beta since the summer on my home PC and I’ve really enjoyed the enhancements.

More 3PAR Enhancements

A couple of days ago 3PAR made more announcements about their added features in the upcoming major release of their InServe operating system for their enterprise fibre channel and iSCSI storage arrays. If you missed their previous announcements, click on 3PAR on the right side of my blog for the other articles.

Of the three features announced recently, I’ll focus on autonomic groups since that is a feature my project will certainly be using. If you working in a highly virtualized environment with dozens of ESX hosts, then you know how complex storage provisioning can become. In an ESX cluster all servers must see the same set of LUNs on the storage array. This enables features such as vMotion, DRS, and HA.

But the problem becomes in presenting storage to several clusters and dozens or hundreds of hosts. Typically in a storage array with LUN masking you would need to individually present a LUN to each and every ESX host. If you forgot one host, then vMotion could crash and bomb. Or worse yet, if you present a LUN to the wrong server all of the data could be corrupted or a security breach occur. Human error in any environment, and particularly large environments, is a big problem.

What has 3PAR done about this? They have developed autonomic groups. An autonomic group is a collection of hosts and LUNs. Adding a new ESX host to an existing cluster with 30 LUNs? With autonomic groups now you just need to add the ESX host to a single autonomic group and a couple of clicks later all LUNs are presented to the new host. Or, if you have an existing ESX cluster and you add a new shared LUN, just add the LUN to the autonomic group and its exported to all ESX hosts with a couple of clicks. Massive reduction in the chances of human error, not to mention a significant reduction in provisioning time.

In addition to simple storage and host provisioning, they’ve also tied in snapshots. So you could snapshot an entire ESX cluster and the associated LUNs with a couple of clicks. Now all 3PAR needs to do is provide Windows VSS support for *ALL* ESX VMs on the LUNs in the autonomic group so that hardware based snapshots are 100% transactionally consistent when the autonomic group is snapshot. Let’s hope that will be in their Recovery manager for VMware product.

The other products they announced in the 3PAR scheduler, which is a scheduling engine to automate routine tasks. And finally, host explorer helps automate the detection of WWNs and other host based information.