During my process of testing the SSL functionality in vSPhere 4.0 ESXi Update 1, I found a helpful set of OpenSSL commands. OpenSSL is used to generate the certificate signing request (CSR), which is then submitted to a CA which in turns provides you a certificate you can install on your ESXi host.
Unfortunately I was running into problems and when I was looking at the ESXi logs, as I was seeing a X509_check_private_key error 0B080074. Upon further investigation I found an OpenSSL command that verifies whether the private/public key pair you have are a match, and if they match your CSR.
To compare your private and public keys for ESXi, use the following OpenSSL commands. You run these commands on a Windows computer where you’ve downloaded and installed the OpenSSL binaries.
openssl x509 -noout -modulus -in rui.crt : openssl md5
openssl rsa -noout -modulus -in rui.key : openssl md5
Note: For some reason the pipe symbol was not being displayed properly in the blog. So change the : to the pipe symbol or the command won’t work.
If the MD5 hashes both commands return match, then you have matching key pairs. If you get different MD5 hashes, you have a mismatch of some type. If you are unsure which certificates go with a particular CSR, execute the following command and match the MD5 with the private/public key pairs from the previous commands.
openssl req -noout -modulus -in rui.csr : openssl md5
Note: Same information applies to the : in this command. Change it to the pipe symbol.
Rui.crt, rui.key and rui.csr are the typical names of the certificates and requests used for ESXi. But the file names don’t matter as long as contents are the proper types (private key, public key, signing request). A Windows CA will typically use a default extension of .cer for the certificate it issues, which I rename to .crt.
I would recommend that you run the x509 and RSA hash matching commands for every certificate you issue. Should you upload a mismatched pair to an ESXi host, you have to blow away the whole ESXi management configuration and start over.
