You can check out the improved, and officially supported method here. This works for vCenter 4.0 and 4.1.
After a significant effort of research and trial and error, it appears I have gotten VMware Update Manager (VUM) 4.0 Update 1 to use SSL certificates generated from an internal Microsoft CA. This completes my quest to replace all SSL certificates that vCenter 4.0 U1 and ESXi 4.0 hosts use. This method is somewhat of a ‘hack’, but so far everything seems to be working well. I haven’t tried this with the gold release of vCenter Update Manager 4.0, so I can’t comment if this procedure works or not.
In my scenario I have VUM installed on a separate server from vCenter. This is a recommended best practice in larger environments. But I’d think this method works equally well with vCenter and VUM co-located on the same server. In that case, you should be able to re-use the certificates you generated for your vCenter server since they have the same FQDN.
1. Read my article about vCenter SSL certificate generation.
2. Perform the exact same steps to generate a certificate (steps 1-9) but use the FQDN of the VUM server, if it’s on a dedicated server.
3. Find the SSL directory path for Update Manager on your system. In my case it’s located at:
D:\Program Files (x86)\VMware\Infrastructure\Update Manager\SSL
4. Compress all of the existing files in the SSL directory into a .ZIP for safe keeping.
5. Stop the VMware Update Manager Service.
6. Replace rui.crt, rui.key and rui.pfx with the new certificates.
7. De-Install VUM. Yes, remove it.
8. Re-install VUM using the exact same settings as your first install, and use the existing database.
9. Launch the vSphere client and open the vCenter Server Status window.
10. Verify everything has a green check, including all VMware Update Manager components.
If you see any errors about health service, or get weird login errors when launching the vSphere Client, something is broke. The key to this whole process is de-installing and re-installing VUM. This resets some credentials, the thumbprint in the ADAM instance, and uses the new certificates you installed. VMware should really make this easier!
You should also be able to pre-position the SSL certificates into the proper directory pior to ANY VUM installation, and it will use them. That would avoid a de-install and re-install. Depending on your installation parameters and whether you are x86 or x64, the directory path will vary.