Enabling HTTP Strict Transport Security (HSTS) For WordPress

If you are a WordPress site administrator, one of the things you can do to improve SEO results and security is secure your site with SSL. Yes, even if you aren’t doing transactions like ecommerce, paypal, etc. using SSL is still recommended. Depending on your WordPress hosting company, they may even have free SSL certificates for you to use. But there are different flavors and configurations of SSL that can improve or detract from your security posture. One feature that was recently brought to my  attention is HTTP strict transport security, or HSTS.

HSTS, in short, tells your browser that you only want it to use (and enforce) SSL connections. Attempts to downgrade to non-encrypted communications are prohibited. HSTS is a flag that you configure on your WordPress site, and is not enabled by default (that I’m aware of). Since SSL configuration can be tricky, and you can end up with mixed mode content, I recommend a WordPress plug-in called Really Simple SSL.

As the plugin name implies, this makes configuring SSL (with HSTS) super easy and all from the GUI. It also scans your WordPress site for potential mixed content issues and brings them to your attention. My site had a couple of flagged issues that I fixed. The free version of the plug-in doesn’t configure/test HSTS for you, but their premium version does (and makes it 1-click easy).

However, it may still take a bit of configuration tweaking to fully enable HSTS. First, after you enable HSTS in the plugin, go to hstspreload.org and check your results. In my case, I had two errors. My site is currently error free, so I’m using aol.com as an example for what you may see.

First, ignore the no HSTS header error. That is likely caused by the second error and does not mean Really Simple SSL didn’t do its HSTS configuration. I use WP Engine as my provider, so I contacted their help desk and gave them a copy of the error. They did some back-end redirection magic and fixed up the redirection issue in about 15 minutes. My redirection issue was slightly different from AOL’s problem, but caused the same red failure message. After your redirection issue is fixed, re-try the scan. In my case, it came back with a green screen showing everything is good. Next, you can submit your site to be included on the global HSTS list, which I also did. Many browsers like Chrome and Firefox use the HSTS list for additional security measures.

And just to make sure my SSL is in top notch, I went over to SSL Labs and ran a test. And yes, my site is now rated A+, which is exceptionally good. It even catches the fact I’m successfully using HSTS.

And there you go! A simple, but not totally free, way to deploy and check HSTS on your WordPress site. Given the plug-in is just a few dollars, and helps fix up a variety of SSL issues besides HSTS, I think it’s money well spent.

vSphere 6.0 Toolkit Update

In my new role at Nutanix I’ve had the pleasure of working with end customers, and configuring their vSphere 6.0 environment. During this process, SSL certificates have come up. Surprisingly, thus far my clients have chosen the VMCA method of deploying certificates. This is great, as it automates certificate deployments in a vSphere 6.0 environment. Even with the VMware certificate tools, there are some manual steps for configuring the VMCA. My vSphere 6.0 toolkit automates most of those steps.

However, while going through the process we stumbled upon a slight bug in my Toolkit when using an intermediate certificate authority. I’ve since fixed that bug, and uploaded the latest vSphere 6.0 SSL Toolkit here.

I’ve been exceptionally busy the last few months, which is why blogging and updating the Toolkit script has taken a back seat. But I did want to get this script update pushed out so other customers don’t run into VMCA problems.

If you are unfamiliar with my vSphere 6.0 SSL Toolkit, then read up on my full vSphere 6.0 installation series here.

VMworld 2015: Certificates for Mere Mortals

Session INF4529

Note: Although not mentioned in this session, I have a SSL toolkit for vSphere 6.0 which makes the replacement process easier. Check out my vSphere 6.0 install guide here for all the details.

Certificate Lifecycle Management

  • VMCA: VMware certificate authority
  • VECS: VMware Endpoint Certificate store

VMCA

  • Dual Operational modes: Root CA and Issuer CA
  • Root CA: Automated, can issue other certs, all solutions and endpoint certificates are created and trusted to this root cert
  • Issuer CA: Can replace all default root CA certificate created during installation. Basically subordinate CA to your enterprise CA.

VECS

  • Repository for certificates and private keys
  • Mandatory component
  • Key stores: machine SSL certs, trusted roots, CRLs, solution users, others (e.g. VVOLS).
  • Managed through veccs-CLI
  • Does not manage SSO certificates

vSphere 6.0 Certificate Types

  • ESXi certificates – autogenerated post-install. New modes in 6.0, one of which can use VMCA certs. Can renew in webclient.
  • Machine SSL certificates – Creates server-side SSL (HTTPS, LDAP, etc.). Each node has its own machine SSL certificate.
  • Solution User certificates – Machine, vpxd, vpxd-extension, vsphere-webclient. Encapsulates one or more vCenter services.
  • Single-sign-on: Not stored in VECS. Stored in filesystem. STS certificate. Renew/update via GUI, not filesystem replacement.

Certificate Replacement Options

  • VMCA as root. Easiest deployment option.
  • VMCA as Enterprise CA subordinate – VMCA will issue certs on behalf of your enterprise CA
  • Custom CA – Only use custom certs all around. Not recommended except for Gov’t/Financial.
  • Hybrid – User facing certs replace, then let VMCA manage solution user and ESXi certs.

VMware vSphere 6.0 Certificate Manager

  • Available on both Windows and VCSA
  • Menu driven (GUI in 6.0 U1)

VMCA as Subordinate

  • RSA with 2048 bits
  • x.509v3
  • SHA256, 384 or 512
  • No wildcards in SubjectAltName
  • Cannot create subsidiary CAs of VMCA
  • Sync time for all nodes

Session videos, slides and scripts: http://vmware.com/go/inf4529

 

Ignite 2015: Encryption, Certificates and PKI

Session: BRK3130

Note: This was a great beginner level session for those not familiar with encryption, certificates or PKI. If you are in that boat, I would urge you to find the session video and watch the whole presentation. If you are a security professional and already know about these topics, then the content is probably too basic. I didn’t capture all the content below, but just took down some highlights what was covered.

Why am I here? Thanks to the NSA. Thanks to Edward Snowden. SharePoint, Lync, Exchange all  need to be secure.

Shows screens of RDP SSL warnings, and browser SSL warnings.

Are you still using passwords? Phishing and fraud, password fatigue, pass the hash attacks

IoT (Internet of things) is adding new concerns of authentication (connected cars, medical, industrial sensors)

Non-repudiation – Ability to bind a human to a digital document

Privacy – Hot topic over the last 2 years due to NSA and Snowden. Challenges are not new.

Encryption – Encryption at rest, in transit, challenges: weak algorithms

Encryption at rest – Bitlocker, EFS, SQL TDE

Encryption in transit – SSL/TLS, IPsec, Office 365 message encryption

Azure RMS – AD RMS for On-Premises. Protect documents from Birth to end of life. Protection regardless of location.

Speaker goes over symmetrical, asymmetrical encryption, hardware security modules (HSM) technologies such as AES and shows how they work.

What is hashing? Uniquely identify a stream of data. It’s a one way function.IMAG0425

Use the tool IIS Crypto to disable/enable and change the order that ciphers are use. FREE.

Good ideas: Remove RC4, reorder suites, Update to 2012 R2, research ECC vs. RSA

Talks about Certificate Authorities, certificates, and their basic properties. Also discusses path of trust, and where to find certificates in Windows.

CA Lifetime planning: End certs – 2 years, intermediate CA – 4 years, root CA – 8 years. Renew certificates when 50% of their life has expired.

S/MIME – For Email encryption and digital signatures

vSphere 6.0 Pt. 13: VMware Directory Svc Certificate

One of the lesser known SSL certificates in the vSphere 6.0 product is called the VMware Directory Service certificate. This is used by the built-in LDAP server for authentication and encryption. It’s most an internal use only certificate, and one that some customers may not worry about replacing. In fact, per VMware support, a lot of customers probably won’t replace this certificate. But, I’m a certificate whore, and wanted to be thorough in my coverage of vSphere 6.0. You will also see the certificate when you install vCenter 6.0 with an external PSC, and authenticate to the PSC. Even if you use the VMCA, the directory services certificate is not replaced by a trusted certificate.

In addition, the VMware certificate tool does not have a menu option to replace the VMware Directory Service certificate. But don’t fear, I’ve built it into my Toolkit script. What VMware doesn’t do, I do. So in this installment I will show you how to replace the VMdir certificate with either one trusted by your enterprise CA, or issued by the VMCA. The toolkit script will also automate the installation for you, on a Windows PSC. If you are using the VCSA, I’m sorry, but we have to use a manual process provided by VMware.

Blog Series

vSphere 6.0 Install Pt. 1: Introduction
vSphere 6.0 Install Pt. 2: Platform Services Controller
vSphere 6.0 Install Pt. 3: Certificate Management
vSphere 6.0 Install Pt. 4: vCenter Upgrade Best Practices
vSphere 6.0 Install Pt. 5: ESXi Upgrade Best Practices
vSphere 6.0 Install Pt. 6: Install Windows PSC
vSphere 6.0 Install Pt. 7: Config SQL DBs
vSphere 6.0 Install Pt. 8: Toolkit Configuration
vSphere 6.0 Install Pt. 9: SSL Templates
vSphere 6.0 Install Pt. 10: Install VCSA PSC
vSphere 6.0 Install Pt. 11: VMCA as Subordinate
vSphere 6.0 Install Pt. 12: PSC Machine Certificate
vSphere 6.0 Install Pt. 13: Directory Services Certificate
vSphere 6.0 Install Pt. 14: Windows vCenter Install

Permalink to this series: vexpert.me/Derek60
Permalink to my Toolkit script: vexpert.me/toolkit60

Mint VMDir Certificate (Online)

You should run the Toolkit script on your Windows external PSC, so you have all the files needed locally and it will also automate the installation. If you are using the VCSA PSC, then run this script from a Windows server that has PowerShell 3.0. Use this online procedure if your Microsoft CA will issue the subordinate certificate either with or without approval. This method also supports the VMCA.

If you are using a VMCA as a subordinate, then select menu option 3 to mint your certificates from the VMCA in step 1 below.

1. Run the Toolkit PowerShell script on your external PSC or a Windows server VM if using the VCSA PSC. Select the VMware Directory Service Certificate menu (option 3). Select the option to create a VMDir certificate with an online MS CA (option 1).2015-04-03_8-42-56

2. Enter the FQDN of the PSC, or press ENTER, if running from the PSC to accept the name. If no certificate approval is needed, the new VMDir certificate will be minted and downloaded.2015-04-07_11-03-09

 

If your MS CA is configured to require CA manager approval before issuing a certificate, you will see the following:

2015-04-02_8-06-50

Have the CA manager approve the request ID, then re-run my Toolkit script and select the “Resume a pending online request for VMDir certificate” (option 4). The script will show you the paths to the chained PEM file and the private key file. After the request is complete, all files are located in C:\Certs\VMDir.

Mint Machine Certificate (Offline)

Use this procedure if your issuing certificate authority is NOT a Microsoft online CA. It could be an offline Microsoft CA, or a non-MS CA as well.  This assumes you have the proper templates configured in your CA, per my Part 9 post.

1. Run the Toolkit script and in the VMware Directory Service menu (option 3) select the option to create a VMDir certificate with an offline or non-MS CA (option 2). The script will verify that you have downloaded the root chain certificates.

2. Because I was running this on the external PSC, I just pressed enter for the PSC name.2015-04-07_11-04-37

3. Navigate to C:\Certs\VMDir and upload the VMDir.csr file to your favorite CA and issue a certificate. Download the issued certificate in the base-64 format and save as VMDir.crt in the same folder. You MUST use this file name and it MUST be base-64 encoded. It should only contain the certificate, not a full chain.

4. Re-run the toolkit and from the menu select the option “Create VMDir PEM file from offline or non-Microsoft CA files” (option 5). No input is needed. This will properly create a PEM file with the full certificate chain.2015-04-07_11-06-54

 

Install VMDir Certificate (Windows PSC)

Note: For this procedure I am showing you how to use my Tookit script to install your VMDir certificate. VMware’s certificate tool does NOT support replacing the VMdir certificate, since not all customers feel the need to replace it. I feel the need. VMware did document the process in their vSphere 6.0 documentation, which is what I implemented in the script.

1. Re-run my Toolkit script and in the VMware Directory Service Menu (option 3) select the option “Install custom VMDir certificate on this computer” (option 6). 2015-04-03_8-44-00

2. Wait about 30 seconds, and the process will complete without any user input.

2015-04-02_21-04-48Install VMDir Certificate (VCSA)

By this point I’m assuming you have the BASH shell enabled, and know how to WinSCP and SSH into the VCSA. Those steps have been covered in pervious blog posts, so I’m not repeating them here.

1. Run my Toolkit script and on the main menu select VMware Directory Service Menu (Option 3). On the following menu select option 7 to rename the certificate files.

2. SSH into the VCSA and enter the following command:

/bin/service-control –stop VMWareDirectoryService

3. From the C:\certs\Machine directory copy the vmdircert.pem and vmdirkey.pem files to:

/usr/lib/vmware-vmdir/share/config/

4. Enter the following command:

/bin/service-control –start VMWareDirectoryService

Validate VMDir Certificate

In case you want to verify that the VMDir certificate actually was replaced and is using your trusted certificate, my toolkit can do that too!

1. Launch the Toolkit and from the main menu select VMware Directory Service menu (option 3). From there select Display VMDir Certificate (option 8).

2. OpenSSL is invoked to display in a somewhat unfriendly manner, the SSL certificate used for the LDAP services. Review the properties to ensure they contain those from your trusted CA.

Summary

While not a popular certificate to replace, replacing the VMware Directory Service certificate does its place. Since the Toolkit makes is so easy to do, I’d recommend replacing it as a matter of practice. This will eliminate a somewhat worrisome certificate validation pop-up during the vCenter installation process. Instead of seeing an untrusted certificate, you will see your freshly minted VMDir certificate.

vSphere 6.0 Pt. 12: PSC Machine Certificate

Back in Part 11 of this series we configured the VMCA to be a subordinate CA to our enterprise CA. This ensures that all certificates which get used by vCenter components are automatically trusted. But as previously mentioned, not all organizations can use the VMCA. The US Federal Government would be a prime example, where there’s no way you can stand up your own subordinate CA.

So if you are one of the organizations that can NOT use the VMCA and need to use custom SSL certificate throughout, this post is for you. In this post we will replace the PSC’s machine SSL Certificate with a certificate issued by your enterprise CA, not the VMCA. If you followed Part 11 and are using the VMCA, skip this post.

Just like Part 11, I’ll go through the same process of using a Microsoft online CA, offline CA, and updating the certificates for both Windows and the VCSA. This should cover most scenarios that people have to deal with. If that’s not exactly what your scenario is, you can probably figure out what to do between VMware documentation and my Toolkit posts.

As always, download the latest version of my Toolkit script, as it is rapidly changing as I add more blog posts about SSL and work through issues. The download permalink is below. For this post you will need at least version 0.75 (April 2, 2015) or later to follow along.

Ironically, the VMware supplied certificate tool in it’s GA form has a bug when you replace the machine certificate with multiple intermediate CAs. You can find the KB here. So I’d recommend using my Toolkit script for a Windows PSC, as it does not have the bug and is easier anyway. 🙂 I am told VMware is working on an updated script, but I have no ETA on a release date. If you are using the VCSA you will need to use the workaround, which I cover in my post below.

Blog Series

vSphere 6.0 Install Pt. 1: Introduction
vSphere 6.0 Install Pt. 2: Platform Services Controller
vSphere 6.0 Install Pt. 3: Certificate Management
vSphere 6.0 Install Pt. 4: vCenter Upgrade Best Practices
vSphere 6.0 Install Pt. 5: ESXi Upgrade Best Practices
vSphere 6.0 Install Pt. 6: Install Windows PSC
vSphere 6.0 Install Pt. 7: Config SQL DBs
vSphere 6.0 Install Pt. 8: Toolkit Configuration
vSphere 6.0 Install Pt. 9: SSL Templates
vSphere 6.0 Install Pt. 10: Install VCSA PSC
vSphere 6.0 Install Pt. 11: VMCA as Subordinate
vSphere 6.0 Install Pt. 12: PSC Machine Certificate
vSphere 6.0 Install Pt. 13: Directory Services Certificate
vSphere 6.0 Install Pt. 14: Windows vCenter Install

Permalink to this series: vexpert.me/Derek60
Permalink to my Toolkit script: vexpert.me/toolkit60

Mint Machine Certificate (Online)

You should run the Toolkit script on your Windows external PSC, so you have all the files needed locally and it will also automate the installation. If you are using the VCSA PSC, then run this script from a Windows server that has PowerShell 3.0. Use this online procedure if your Microsoft CA will issue the subordinate certificate either with or without approval.

1. Run the Toolkit PowerShell script on your external PSC or a Windows server VM if using the VCSA PSC. Select the Machine SSL Certificate menu (option 4). Select the option to create a Machine SSL certificate with an online MS CA (option 1).

2. Enter the FQDN of the PSC, or press ENTER, if running from the PSC to accept the name. If no certificate approval is needed, the new machine certificate will be minted and downloaded.2015-04-02_8-02-13If your MS CA is configured to require CA manager approval before issuing a certificate, you will see the following:

2015-04-02_8-06-50

Have the CA manager approve the request ID, then re-run my Toolkit script and select the “Resume a pending online request for Machine SSL certificate” (option 4). The script will show you the paths to the chained PEM file and the private key file.

2015-04-02_8-18-27

After the request is complete, you will see the following files in the C:\Certs\Machine directory.

2015-04-02_8-20-02

You have now minted your Machine SSL certificate, but it is not yet installed. Read on further in this post on how to install it.

Mint Machine Certificate (Offline)

Use this procedure if your issuing certificate authority is NOT a Microsoft online CA. It could be an offline Microsoft CA, or a non-MS CA as well.  This assumes you have the proper templates configured in your CA, per my Part 9 post.

1. Run the Toolkit script and in the Machine SSL menu (option 4) select the option to create a Machine SSL certificate with an offline or non-MS CA (option 2). The script will verify that you have downloaded the root chain certificates.

2. Because I was running this on the external PSC, I just pressed enter for the PSC name.

2015-04-02_8-53-19

3. Navigate to C:\Certs\Machine and upload the machine_ssl.csr file to your favorite CA and issue a certificate. Download the issued certificate in the base-64 format and save as new_machine.crt in the same folder. You MUST use this file name and it MUST be base-64 encoded. It should only contain the certificate, not a full chain.

4. Re-run the toolkit and from the menu select the option “Create Machine SSL PEM file from offline or non-Microsoft CA files” (option 5). No input is needed. This will properly create a PEM file with the full certificate chain.

2015-04-02_9-10-18

Install Machine SSL Certificate (Windows PSC)

Note: For this procedure I am showing you how to use my Tookit script to install your Machine SSL certificate. VMware provides a Certificate Management tool that can perform the same steps. I show you how to use the VMware tool in the next section, when using the VCSA. The tool is the same on Windows and the VCSA. So if you feel more comfortable using the VMware tool to install the cert, skip down to that section. On Windows you can find the tool at C:\Program Files\VMware\vCenter Server\vmcad\Certificate-manager. My tool uses the manual method as documented in the vSphere 6.0 security guide, so the results are the same.

1. Re-run my Toolkit script and in the Machine SSL menu (option 4) select the option “Install custom machine SSL certificate on this computer” (option 6).2015-04-02_9-13-45

2. Sit back and wait while the script stops services, installs the new certificate, and restarts the services. Keep an eye on the process, as mid way through you will need to confirm the deletion of the existing machine certificate. Simply press Y.

2015-04-02_9-29-56

Install Machine Certificate (VCSA PSC)

Note: For this procedure I am showing you how to use the VMware Certificate Manager tool to install the VMCA signing certificate. This assumes you used my Toolkit to generate the certificate files. There’s a bug documented in this VMware KB about the tool failing with multiple intermediate CAs. I’ll include the workaround here, so you have a one stop shop for replacing your certificates.

1. If you haven’t already enabled BASH on your VCSA let’s do that now. Open a console into the VCSA. Press F2 to customize the system. Login. Arrow down to “Troubleshooting Mode Options” then enable BASH shell. Exit the VCSA console.

2. Open a SSH session to the VCSA and type the following:

shell

chsh -s “/bin/bash” root

Make sure you run the ‘chsh’ command from the ‘shell’ prompt and not the VMware restricted shell…it won’t recognize the chsh command. Thanks to William Lam’s blog post here for this step!

2. Download and install your favorite SCP client. I like WinSCP. Connect via SCP using the VCSA credentials.2015-04-01_14-40-003. Create a folder to put your SSL certificates. I like the ‘/root/ssl’ directory.

4. In WinSCP navigate to the C:\Certs\machine folder. Upload the new_machine.cer and ssl_key.priv files to the SSL directory on the VCSA. The other files in the machine folder are not needed, so don’t upload them. From the C:\certs folder upload the chain.cer AND the root64.cer files to the /root/ssl directory on the VCSA. Note that all the options begin with a double dash, not a single dash. Cut/paste may mangle the dashes and cause the command to fail. Best to manually type the whole command instead of cut/paste.

5. SSH into the VCSA and ensure you get a ‘shell’ prompt. This will be in red, and have the short name of the VCSA. Type the following command.

Windows :

“C:\Program Files\VMware\vCenter Server\vmafdd\dir-cli.exe” trustedcert publish –chain –cert c:\certs\chain.cer

VCSA:

/usr/lib/vmware-vmafd/bin/dir-cli  trustedcert publish –chain –cert /root/ssl/chain.cer

6. In the VCSA shell run the following command:

/usr/lib/vmware-vmca/bin/certificate-manager

6. Choose Option 1 from the main menu. Enter the SSO password as requested.

7. From the new menu select Option 2, Import custom certificates. Input the root certificate file names when prompted. Use /root/ssl/new_machine.cer for the first prompt and /root/ssl/ssl_key.priv for the second. For the third and final prompt enter /root/ssl/root64.cer.

2015-04-02_14-42-00a

 

8. After you enter all the certificate paths you will be prompted to continue. The whole replacement process takes less than two minutes.

Inspecting the Machine Certificate

Now that we have installed a new machine SSL certificate, we want to make sure it was issued by our enterprise CA and is trusted. This can easily be done via any browser of your choosing.

1. Launch your favorite browser and go to https://PSC-FQDN/websso/. Open the certificate properties for the SSL site.

2015-04-01_19-07-392. Click on the Certification Path, and verify that all of your enterprise CAs are listed. If you only see a single entry in this list, and not the full chain, that likely means your Windows computer does NOT trust the full chain. See your CA administrator for getting all of your enterprise CAs published through Active Directory.

2015-04-02_15-25-05

Solution Warning

A reader pointed out the SRM and other solutions may fail when replacing the machine certificate on vCenter or the PSC. If you find yourself in this situation, check out this VMware KB article for remediation.

Summary

When you aren’t using the VMware VMCA, you must mint and install a machine SSL certificate for the PSC from your enterprise CA. This certificate is used for all reverse proxy services, such as those accessed by HTTP. You can elect to either use my Toolkit script to install the machine cert, or the VMware tool. Either way, you end up with a trusted machine SSL certificate on your PSC.

vSphere 6.0 Install Pt. 11: VMCA as Subordinate

As mentioned in my vSphere 6.0 install part 3, VMware has introduced a new certificate option in this release. You can now make a built-in VMware certificate authority (VMCA) a subordinate to your enterprise CA. This is great news, as it will allow some automation of the certificate deployment process. However, there are certainly some regulated industries like the US Federal Government where they’d laugh you out of the office if you asked to stand up your own subordinate CA. So this model won’t be used by everyone.

So if you are one of those organizations that CAN use the built-in VMware VMCA, then follow along in this article to see how you make it a subordinate in your enterprise environment. Later in this series we will cover the manual replacement process so don’t worry…alternatives to the VMCA are covered.

As a reminder, I’ve written the vCenter Toolkit script which helps in the SSL deployment process. In this version of the script I’ve added some VMCA features. See the download link below. In this series I’m assuming a Windows Enterprise CA, but you can use a non-Windows CA with a little bit more work (manual submission and download).

Important: You MUST use Toolkit version 0.7 or later. Please download the latest Toolkit from the URL below.

Blog Series

vSphere 6.0 Install Pt. 1: Introduction
vSphere 6.0 Install Pt. 2: Platform Services Controller
vSphere 6.0 Install Pt. 3: Certificate Management
vSphere 6.0 Install Pt. 4: vCenter Upgrade Best Practices
vSphere 6.0 Install Pt. 5: ESXi Upgrade Best Practices
vSphere 6.0 Install Pt. 6: Install Windows PSC
vSphere 6.0 Install Pt. 7: Config SQL DBs
vSphere 6.0 Install Pt. 8: Toolkit Configuration
vSphere 6.0 Install Pt. 9: SSL Templates
vSphere 6.0 Install Pt. 10: Install VCSA PSC
vSphere 6.0 Install Pt. 11: VMCA as Subordinate
vSphere 6.0 Install Pt. 12: PSC Machine Certificate
vSphere 6.0 Install Pt. 13: Directory Services Certificate
vSphere 6.0 Install Pt. 14: Windows vCenter Install

Permalink to this series: vexpert.me/Derek60
Permalink to my Toolkit script: vexpert.me/toolkit60

Mint VMCA Certificate (Online)

You should run the Toolkit script on your Windows external PSC, so you have all the files needed locally and it will also automate the installation. If you are using the VCSA PSC, then run this script from a Windows server that has PowerShell 3.0. Use this online procedure if your Microsoft CA will issue the subordinate certificate either with or without approval. Before you run the Toolkit, always download the latest version as new versions can come out rapidly. You can get the latest from the permalink above.

1. Run the Toolkit PowerShell script on your external PSC or a Windows server VM if using the VCSA PSC. Select the VMCA menu (option 2). Select the option to create a VMCA signing certificate with an online MS CA (option 1).

2. Press Enter to accept the PSC name, and the script will take care of the rest. After it quickly runs you have a C:\Certs\VMCA directory with all of the needed files.2015-04-01_14-21-25

 

If your MS CA is configured to require CA manager approval before issuing a certificate, you will see the following:

2015-03-29_18-53-10

Have the CA manager approve the request ID, then re-run my Toolkit script and select the “Resume a pending online request for VMCA certificate” (option 3).

2015-03-30_19-07-35

After the request is complete, you will see the following files in the C:\Certs\VMCA directory. As you will notice, I’ve purposely used the same file names as the VMware documentation and tools.

2015-03-30_19-08-41

You have now minted your VMCA certificates, but they are not yet installed. Read on in the blog post to find out how to install them.

Mint VMCA Certificate (Offline)

Use this procedure if your issuing certificate authority is NOT a Microsoft online CA. It could be an offline Microsoft CA, or a non-MS CA as well. Don’t try this procedure with a public CA provider, as they won’t let you issue subordinate CA certificates. Only use this for internal enterprise CAs which you control and can issue subordinate certificate authority certificates with.

1. Run the Toolkit script and in the VMCA menu (option 2) select the option to create a VMCA signing certificate with an offline or non-MS CA (option 2). The script will verify that you have downloaded the root chain certificates.

2. Because I was running this on the external PSC, I just pressed enter for the PSC name.2015-03-29_12-00-023. Navigate to C:\Certs\ and upload the root_signing_cert.csr file to your favorite CA and issue a certificate. Download the issued certificate in the base-64 format and save as root_signing_cert.crt in the same folder. You MUST use this file name and it MUST be base-64 encoded. It should only contain the certificate, not a full chain.

4. Re-run the toolkit and from the menu select the option “Create VMCA PEM files from offline or non-Microsoft CA” (option 4). No input is needed. This will properly create a PEM file with the full certificate chain. 2015-03-30_19-10-59

Now that we have our VMCA certificates, we need to install them. I will cover both the Windows and VCSA installation process. Windows is easier, since I’ve built that functionality into my Toolkit. For the VCSA we will leverage the VMware certificate tool.

Install VMCA Certificate (Windows PSC)

Note: For this procedure I am showing you how to use my Tookit script to install your VMCA signing certificate. VMware provides a Certificate Management tool that can perform the same steps. I show you how to use the VMware tool in the next section, when using the VCSA. The tool is the same on Windows and the VCSA. So if you feel more comfortable using the VMware tool to install the cert, skip down to that section. On Windows you can find the tool at C:\Program Files\VMware\vCenter Server\vmcad\Certificate-manager. My tool uses the manual method as documented in the vSphere 6.0 security guide, so the results are the same.

1. Re-run my Toolkit script and in the VMCA menu (option 2) select the option “Install VMCA signing certificate on Windows PSC” (option 5).

2015-04-01_18-24-46

2. Sit back and wait while the script stops services, installs the new certificate, restarts the services, then lists the certificate properties. Keep an eye on the process, and verify a ‘success’ message half way through. I’ve added a pause and notification when you should see the ‘success’ message. The entire replacement process takes about two minutes. 2015-03-29_12-20-293. Once the script completed, it will show the properties of the new VMCA. Validate that these match the issued certificate.2015-03-29_12-21-47

4. To follow in the footsteps of the VMware Certificate tool that also replaces the Machine SSL certificate by a VMCA issued certificate, re-run the Toolkit and select the Machine SSL certificate menu (option 4). Then select Option 3, Create Machine SSL certificate with VMCA. Press enter to accept the default hostnames (assuming you are running this on the PSC/VMCA).

2015-04-01_18-33-54 5. Re-Run the toolkit and select the Machine SSL certificate menu again. This time select option 6. It will stop the PSC services, install the certificate, and then re-start the services. You will also get prompted mid way through if you want to delete the existing machine certificate. Answer “Y” here to proceed. 2015-04-01_18-39-20

Install VMCA Certificate (VCSA PSC)

Note: For this procedure I am showing you how to use the VMware Certificate Manager tool to install the VMCA signing certificate. This assumes you used my Toolkit to generate the certificate files.

1. If you haven’t already enabled BASH on your VCSA let’s do that now. Open a console into the VCSA. Press F2 to customize the system. Login. Arrow down to “Troubleshooting Mode Options” then enable BASH shell. Exit the VCSA console.

2. Open a SSH session to the VCSA and type the following:

shell.set –enabled true

shell

chsh -s “/bin/bash” root

Make sure you run the ‘chsh’ command from the ‘shell’ prompt and not the VMware restricted shell…it won’t recognize the chsh command. Thanks to William Lam’s blog post here for this step!

2. Download and install your favorite SCP client. I like WinSCP. Connect via SCP using the VCSA credentials.2015-04-01_14-40-003. Create a folder to put your SSL certificates. I like the ‘/root/ssl’ directory.

4. In WinSCP navigate to the C:\Certs\VMCA folder. Upload the root_signing_chain.cer and root_signing_cert.key files to the SSL directory on the VCSA. The other files are not needed, so don’t upload them.

5. SSH into the VCSA and ensure you get a ‘shell’ prompt. This will be in red, and have the short name of the VCSA. Type the following command:

/usr/lib/vmware-vmca/bin/certificate-manager

6. Choose Option 2 from the main menu. Enter the SSO password as requested.

7. From the new menu select Option 2, Import custom certificates. Input the root certificate file names when prompted. Use root_signing_chain.cer for the first prompt and root_signing_cert.key for the second.

2015-04-01_17-35-298. If you haven’t run the tool before, you will be prompted for a series of default values for the certool.cfg. Use the same values here as you did when setting up the Toolkit variables. When prompted, use the FQDN of the PSC for the ‘hostname’.

9. Wait a couple of minutes for the tool to run. After it has completed it has made the VMCA a subordinate to your enterprise CA, and also updated the machine SSL certificate to one issued by the new VMCA.

Inspecting the SubCA Cert

If you wish to look at the properties of the newly minted VMCA subordinate signing certificate which was installed, just double click on the C:\Certs\VMCA\root_signing_chain.cer file. Go to the Certification Path tab, and you can see the full certificate chain. If you do NOT see the full certificate chain, then follow my blog post here to enable your enterprise to trust the VMCA.

2015-03-29_19-10-30

You can also check out the Details tab, and go down to the “Basic Constraints” and “Key Usage” fields. There you can see this certificate can be used as a subordinate CA.

2015-03-29_19-12-08

Inspecting the Machine Certificate

Now that we have installed a new machine SSL certificate, we want to make sure it was issued by the VMCA and is trusted. This can easily be done via any browser of your choosing.

1. Launch your favorite browser and go to https://PSC-FQDN/websso/. Open the certificate properties for the SSL site.

2015-04-01_19-07-392. Click on the Certification Path, and verify that all of your enterprise CAs are listed, as well as the FQDN of your PSC. There should be a certificate listed under the FQDN of your PSC. If you only see a single entry in this list, and not the full chain, that likely means your Windows computer does NOT trust the VMCA subordinate CA. If that’s the case, I would recommend publishing the VMCA public certificate via Active Directory so that your entire domain will trust the VMCA. As you can see in my screenshot below I have three enterprise CAs and the VMCA.

2015-04-01_19-11-42

Solution Warning

A reader pointed out the SRM and other solutions may fail when replacing the machine certificate on vCenter or the PSC. If you find yourself in this situation, check out this VMware KB article for remediation.

Summary

Using my toolkit script, minting and updating the VMCA certificate is a breeze. My toolkit supports online Microsoft CAs, or any offline CA that you wish you use. Remember you can’t use a public CA for issuing your VMCA subordinate certificate. This must be an internal CA which you own, and which has been configured to issue certificates for the subordinate authority template.

Because we are using the VMCA for all of our certificates, we’ve also re-issued the PSC’s machine SSL certificate to one issued by the VMCA. This means any of the web services the PSC is providing are now using a VMCA issued certificate. In the next installment we will assume you can NOT use the VMCA for compliance reasons, and will replace the PSC’s machine SSL certificate with a trusted one. That is covered in Part 12. This will bring our two installations up to the same level. After that, we will replace the Directory Services certificate on the PSC.

vSphere 6.0 Pt. 9: SSL Templates

VMware has provided new SSL template guidance for vSphere 6.0. New to vSphere 6.0 are machine SSL certificates, solution user certificates, and using the VMCA as a subordinate CA. If you are using an enterprise Microsoft CA, then this article is for you. I’ll show you how to create the new templates and publish them within your CA. You can then go into my vCenter Toolkit and change the template names to match. If you are not using a Microsoft CA, then you are on your own for creating the right templates in your particular CA. Again, you shouldn’t be using a public CA for these certificates. Use an internal enterprise CA.

April 2, 2015 Update: VMware has informed me that VUM 6.0 MUST use the old vSphere 5.5 certificate template. VUM 6.0 is NOT compatible with the new machine certificate template which debuted in 6.0. So jump to my 5.5 SSL template guide here and create the VMware-SSL template if it does not exist in your environment. If you followed my 5.5 guide and already have the template, then you are set.

Blog Series

vSphere 6.0 Install Pt. 1: Introduction
vSphere 6.0 Install Pt. 2: Platform Services Controller
vSphere 6.0 Install Pt. 3: Certificate Management
vSphere 6.0 Install Pt. 4: vCenter Upgrade Best Practices
vSphere 6.0 Install Pt. 5: ESXi Upgrade Best Practices
vSphere 6.0 Install Pt. 6: Install Windows PSC
vSphere 6.0 Install Pt. 7: Config SQL DBs
vSphere 6.0 Install Pt. 8: Toolkit Configuration
vSphere 6.0 Install Pt. 9: SSL Templates
vSphere 6.0 Install Pt. 10: Install VCSA PSC
vSphere 6.0 Install Pt. 11: VMCA as Subordinate
vSphere 6.0 Install Pt. 12: PSC Machine Certificate
vSphere 6.0 Install Pt. 13: Directory Services Certificate
vSphere 6.0 Install Pt. 14: Windows vCenter Install

Permalink to this series: vexpert.me/Derek60
Permalink to my Toolkit script: vexpert.me/toolkit60

Machine SSL and Solution User Certificates

1. Login to your issuing CA and launch the Certificate Authority MMC snap-in.

2. Locate the Certificate Templates folder, right click, and select Manage.

2015-03-30_10-44-143. Locate the “Web Server” template, right click, and duplicate it.

4. Click on the General tab and name it “vSphere 6.0”. You will use the “Template name” in my Toolkit script as the template name, FYI. 2015-03-30_10-57-025. Click on the Extension tab, click on Application Policies, then Edit. Remove Server Authentication and click OK.

2015-03-30_11-00-05

6. Select Key Usage, then click on Edit. Check the box next to nonrepudiation.

2015-03-30_11-00-517. Click on Subject name. Ensure that “Supply in the request” is selected.2015-03-30_11-02-558. Click on the Compatibility tab and ensure the Windows server 2003 is selected for both options. Even if you are running a newer CA, don’t select later CA options.2015-03-30_11-04-429. Close the Certificate Templates console window, right click on Certificate Templates, select New, then Certificate Template to Issue. Find the vSphere 6.0 template and select it. Click OK.

VMCA Subordinate Template

You only need this template if you will be using the VMCA as a subordinate CA to your enterprise CAs. If you are going to be using fully custom SSL certificates without the VMCA, you can skip this template.

1. Login to your issuing CA and launch the Certificate Authority MMC snap-in.

2. Locate the Certificate Templates folder, right click, and select Manage.

2015-03-30_10-44-14

3. Locate the “Subordinate Certificate Authority” template, right click, and select Duplicate.

4. On the General tab change the name to “vSphere 6.0 VMCA”. Also, it’s important to check the box to publish the certificate to Active Directory. This will ensure all computer trust your VMCA. For my Toolkit script you will use the template name of “vSphere6.0VMCA” (no spaces).2015-03-31_7-34-38

5. Click on the Compatibility tab and change both compatibility settings to Windows Server 2008. This enables hashing algorithms stronger than SHA1 to be used.

2015-03-30_11-25-02

6. Click on the Extensions tab. Select Key usage and click Edit. Verify that all the options shown below are checked.

2015-03-30_11-26-087. Close the Certificate Templates console window, right click on Certificate Templates, select New, then Certificate Template to Issue. Find the vSphere 6.0 VMCA template and select it. Click OK.

Summary

VMware has changed the security template requirements in vSphere 6.0. They’ve also introduced a new template requirement, if you are going to be using the VMCA as a subordinate CA. You need both templates if you are going to take full advantage of the new certificate features in vSphere 6.0. If you still have a VMware SSL template from prior versions, keep it around, in case you need to re-issue certs for your legacy environment. Remember to update the variables in my Toolkit script to match the new template names.

Next up in this series is installing a VCSA-based PSC, in case you want to go that route versus using a Windows PSC. You can find that article here.

vSphere 6.0 Install Pt. 8: Toolkit Configuration

Now that we have the PSC installed, it’s time to configure the variables for the Toolkit script, and also make sure we can download our root certificates. Depending on your configuration, you may need to manually download your root public certificates. VMware needs certificates in a specific format, and they need the full certificate chain. So in this installment I show you all the variables in the Toolkit script that you will need to change to make it successful. In subsequent installments we will then use the Toolkit to setup the VMCA and other certificate options.

April 2, 2015 Update: Per VMware, VUM 6.0 can NOT use the vSphere 6.0 SSL template. So I’ve added a new variable called $VUMTemplate for the old 5.5 SSL template name. You can find instructions for creating the vSphere 5.5 template here.

Blog Series

vSphere 6.0 Install Pt. 1: Introduction
vSphere 6.0 Install Pt. 2: Platform Services Controller
vSphere 6.0 Install Pt. 3: Certificate Management
vSphere 6.0 Install Pt. 4: vCenter Upgrade Best Practices
vSphere 6.0 Install Pt. 5: ESXi Upgrade Best Practices
vSphere 6.0 Install Pt. 6: Install Windows PSC
vSphere 6.0 Install Pt. 7: Config SQL DBs
vSphere 6.0 Install Pt. 8: Toolkit Configuration
vSphere 6.0 Install Pt. 9: SSL Templates
vSphere 6.0 Install Pt. 10: Install VCSA PSC
vSphere 6.0 Install Pt. 11: VMCA as Subordinate
vSphere 6.0 Install Pt. 12: PSC Machine Certificate
vSphere 6.0 Install Pt. 13: Directory Services Certificate
vSphere 6.0 Install Pt. 14: Windows vCenter Install

Permalink to this series: vexpert.me/Derek60
Permalink to my Toolkit script: vexpert.me/toolkit60

Derek’s Toolkit Script

My Toolkit PowerShell script performs several tasks and is menu driven. It’s an all in one script, meaning it handles online/offline CAs, Windows CA and non-Windows CAs, and will also do other install tasks like create your ODBC and SQL database files connectors. New for vSphere 6.0 are automation steps for the VMCA and added support for three tier CA hierarchy (root and two subordinates).

My Toolkit script does NOT replace the VMware certificate replacement tools, it only augments them. So you would normally use the combination of my Toolkit script plus the VMware certificate management tools for full SSL certificate replacement. I did this specifically so that customers would be fully supported by VMware, even if they use my tool. I just make the process easier, I don’t do any behind the scenes hacking or unsupported commands.

I am still in the process of developing the script, so some of the vCenter SSL features are disabled in the initial versions until I work through the full process. But much of the script is functional in this initial version.

The script has the following features:

  • Downloads and installs the proper version of OpenSSL if it’s not already installed
  • Creates 2048 bit RSA private keys in the proper format
  • Downloads both the root and up to two subordinate public certificates
  • Submits the CSRs to the online CA and downloads the certificates
  • Creates the needed service PEM files for the vCenter certificate tool
  • Creates the required root/subordinate PEM files
  • Does NOT require PowerCLI
  • Automatically uses the hostname of the server you run the script on for all certificates
  • Works with offline CAs
  • Creates customized SQL vCenter and VUM database creation script
  • Creates SQL ODBC DSNs for vCenter and VUM (SQL 2008 R2, 2012, 2014)
  • Automatically downloads and installs SQL 2008 R2 client package
  • Provides download URL for SQL 2012/2014 client
  • Support Microsoft CAs that require manual certificate approval
  • Requires PowerShell 3.0 or higher

Configure Toolkit Variables

1. Login to your external PSC and download my Toolkit script from here. You can run it from anywhere, but I think this is the optimal place for the first run.

2. My script will automatically download OpenSSL for you. Since OpenSSL versions change frequently, I put the download name up front for this version of the script. If you run the script and it errors out, it will display a friendly failure message. Just go to the URL shown, update the download filename and Voila! Unlike my vSphere 5.5 script, I won’t be releasing new versions every time OpenSSL is updated.

2015-03-31_7-11-21

 

 

3. Open the script in your favorite PowerShell editor and find the certificate details section. Modify the company name, organization, etc. for your environment.

2015-03-29_17-05-19

 

 

 

4. Modify the CA names as needed for your environment. My script now supports a root CA plus two subordinates. If you don’t have one or more subordinates, just add a # in front of the appropriate line.

2015-03-29_9-08-07

 

 

5. If you are  using a Microsoft CA with the certificate web enrollment service enabled, then select whether you will be accessing the CA web site via HTTP or HTTPS. HTTPS is recommended, but sometimes there are certificate errors that don’t allow that to work.

2015-03-29_9-10-57

6. Next up you need to configure your Issuing CA information. This can be a little confusing, due to the way Microsoft labels the CA. The best way to find the proper name is login to your issuing CA, launch the Certificate Authority snap-in. This could be called anything, depending on how your CA was setup. Look for the name next to the green check mark. In the script prepend that name with the hostname of your CA.

2015-03-29_9-22-13

2015-03-29_9-18-57

7. For VUM 6.0 we need to use the vSphere 5.5 SSL template. So enter the name of your vSphere 5.5 SSL template here. If you followed my 5.5 guide, then it will be called VMware-SSL. Do NOT use your vSphere 6.0 template name here, as it will NOT work.

2015-04-02_10-40-57

8. Now you need to configure your VMware SSL template name. These certificates will be used for vCenter services and ESXi host certificates. The steps for vSphere 6.0 are NOT the same, so refer to my blog article here in Part 9 for the template instructions. This template names assumes you will follow that article. You can NOT use your vSphere 5.5 template.

2015-04-04_18-52-05

9. Next up, you need to define the Subordinate template name. VMware requires using a custom template and not the Microsoft default. If you follow my blog post here, then your template name will be called vSphere6.0VMCA. If you won’t be using the VMCA subordinate CA feature, just ignore this section.

2015-03-30_11-38-47

If you have a custom template and need to know the “Template Name”, just open your CA MMC, go to “Certificate Templates”, right click and select “Manage”. Open the properties of the template in question and look for the “Template name” NOT the “Template display name”.

2015-03-30_11-36-39

 

 

 

 

 

10. To download the proper certificate chain, my script must download the public certificates from each of the CAs that are in the chain. Depending on the age of your CA, you may need to increment up the “renewal” numbers to get the latest certificate. If you increment too high it will download garbage and my script will alert you to that fact. “0” is the default, but you may find you need 1 or more here.

2015-03-29_9-13-52

Configure Windows CA

11. Next up we need to make sure your Windows CA can issue subordinate certificates if you will be using the VMCA as a subordinate CA. Ignore this section if you won’t be making the VMCA subordinate to your Windows CA. Go into your issuing CA, launch the Certificate Authority tool and look in the “Certificate Templates” folder. You should see a “vSphere 6.0 VMCA” template listed after you complete Part 9 of my guide. 2015-03-31_14-26-08

12. If you do not see this listed then you haven’t read Part 9 (sorry I didn’t blog about this before, but it was a last minute lesson learned) and created the template. Go to that part now, create the new vSphere 6.0 templates, then come back here.

Download Root Certs

If all of your CAs are serviced by an online Microsoft CA and you have correctly configured the Toolkit script variables, and you have web services enabled on the CA, then the script will automatically download the public certificates for you. However, if you have an offline CA or they aren’t web enrollment enabled, you will need to download them manually. Or if you are using a non-MS CA, then you need to get them manually as well. Sometimes the MS CA web services won’t cooperate so manual downloads are needed as well.

13.  Open a blank MMC, then add the Certificates snap-in for the Computer account.

14. Navigate to the “Intermediate Certification Authorities” folder and open the Certificates folder. If you don’t see your CAs there, poke around in the other folders until you find them.

15. Find the certificate authorities for your environment. Right click on each one, and export as a base-64 encoded x.509 certificate. Save the root certificate as C:\Certs\root64.cer. Save the first subordinate certificate (if applicable) as C:\Certs\interm64.cer. If you have a second subordinate, save that certificate as C:\certs\interm264.cer.

2015-03-29_11-19-12

In case you are unsure of the base-64 certificate format, it will look like the following graphic if opened in a text editor.

2015-03-29_11-44-50

 Summary

If you are familiar with my Toolkit script for vSphere 5.5, then you will be right at home in the 6.0 version. I’ve cleaned up the configurable variables, added a few new ones, and added full VMCA support. We will use the Toolkit to configure the remaining SSL certificates, which include vCenter and ESXi. Next up is configuring the SSL template in Part 9.

vSphere 6.0 Install Pt. 3: Certificate Management

Introduction

As long as I can recall certificate management in vSphere has been difficult, and for many customers, something they completely ignore. I’m surprised how many customer designs (even those done by VCDXs) I’ve seen where they feel it’s too difficult to deploy vSphere certificates so they accept the risk of using the non-trusted VMware provided certificates. While I don’t think untrusted SSL certificates are the biggest security threat to an enterprise, I do feel that using trusted certificates is the right thing to do and an add extra layer of security. If you work in a highly regulated industry like finance, healthcare or Government you may be mandated to use fully trusted certificate. Most of my career has been in the Government sector, so using trusted certs was not even a question and just a basic security requirement.

Starting in vSphere 5.1, SSL complexity really shot up and was pretty ‘cocked up’ to put it politely. In vSphere 5.5 VMware did address some of the complexity with a command line tool to help replace certificates. That was still complex, so I wrote the widely used vCenter 5.5 toolkit, which made the whole process super easy. Feedback on that effort has been super positive, and kept me motivated to do the same for 6.0. Now with vSphere 6.0 my toolkit script has to do less because VMware has made it easier, but I still want to make it even easier for customers. Fortunately or unfortunately, depending on how you look at it, vSphere 6.0 has new certificate management options which at first look make SSL more complex than in the past. We’ll dig into each option in this article.

Blog Series

vSphere 6.0 Install Pt. 1: Introduction
vSphere 6.0 Install Pt. 2: Platform Services Controller
vSphere 6.0 Install Pt. 3: Certificate Management
vSphere 6.0 Install Pt. 4: vCenter Upgrade Best Practices
vSphere 6.0 Install Pt. 5: ESXi Upgrade Best Practices
vSphere 6.0 Install Pt. 6: Install Windows PSC
vSphere 6.0 Install Pt. 7: Config SQL DBs
vSphere 6.0 Install Pt. 8: Toolkit Configuration
vSphere 6.0 Install Pt. 9: SSL Templates
vSphere 6.0 Install Pt. 10: Install VCSA PSC
vSphere 6.0 Install Pt. 11: VMCA as Subordinate
vSphere 6.0 Install Pt. 12: PSC Machine Certificate
vSphere 6.0 Install Pt. 13: Directory Services Certificate
vSphere 6.0 Install Pt. 14: Windows vCenter Install

Permalink to this series: vexpert.me/Derek60
Permalink to my Toolkit script: vexpert.me/toolkit60

Who cares about SSL?

Why should you go through the headaches of replacing all the VMware self-signed certificates? What’s the risk of using untrusted certificates? What can happen if the SSL connection is compromised?

Hypervisors are likely the underpinnings of your business critical apps and intellectual property. If your hypervisor is compromised then it’s just a few short commands to access your critical business data. Unless you like your infrastructure being p0wn3d, then you don’t want your VMware infrastructure compromised. If you don’t use trusted certificates, and just click through all the VI client SSL warnings (you have clicked Ignore and trust this certificate many times…haven’t you?) then you won’t know that a man-in-the-middle attack has taken place.

A man-in-the-middle attack is where a third party intercepts your “secure” communications and relays data between you, the attacker’s device, and the end host (like an ESXi server). This can be accomplished by ARP spoofing, or other means. If it’s second nature to ignore and click through all VI certificate warnings, you will have no idea your credentials have been intercepted….in clear text. No fancy brute force decryption required. Just sit back, grab a coke, and enjoy cleartext flowing across your screen. An interesting article on attacking VMware is here.

There are certainly many other ways to compromise your virtual infrastructure, like stealing the credentials of an administrator and gaining direct access to vCenter. Or using pass the hash, and gaining vCenter access that way. So ‘hacking’ SSL may not be the first choice for an attacker, but it is an attack vector you should consider and secure.

VMware Certificate Authority (VMCA)

This is a new an exciting component in vSphere 6.0 that will radically change how many will issue and deploy SSL certificates in their vSphere environment. SSL certificates are used extensively to secure communications in a vSphere environment. This ensures data confidentiality and integrity. Any attempt to modify data in transit is detected, such as man-in-the-middle attacks.

The VMCA is a built-in certificate authority, which is included in the Platform Services Controller (PSC) service. This is a full blown CA, and can (if you wish) automatically issue certificates to all vCenter 6.0 components and ESXi 6.0 hosts in your environment. The VMCA is mostly command line driven, and does not have a fancy GUI like your Microsoft CA has. But once configured, it’s pretty much a hands off operation. Do take note that VMCA in vSphere 6.0 does NOT support the use of CRLs nor does it have the concept of certificate revocation. If you suspect one certificate was compromised, first remove it then replace all certificates.

VMCA Intermediate Certificate Requirements

If you wish to use the VMCA as a subordinate CA to your existing enterprise CA, take note of the certificate requirements. The requirements are:

  • Private Key Algorithm: RSA with 2048 bits. No fancy elliptical curve support.
  • Recommended Signature Algorithms: SHA256, SHA384, or SHA512
  • NOT Recommended algorithms: MD2, MD5, SHA1
  • Key Usage: Root certificate extension set to true and cert sign must be in the list of requirements
  • Use PEM certificate format, with a header of —–BEGIN CERTIFICATE——
  • Does NOT support Wildcard certificates or more than one DNS name
  • Certificate must be X.509 v3

More about being a subordinate CA later in this article.

Certificate Deployment Options

VMware has come up with four primary certificate deployment options in vSphere 6.0. This is more than any previous release, where you basically had two (use VMware certs, or replace with trusted certs). You need to fully understand all four options, then pick for your environment which one best meets your business and security requirements. Depending on your industry, you may be severely limited in your choices. For example, if you are a U.S. Government agency you are stuck with option 3, using an external CA for all certificates and you won’t care about the new VMCA.

2015-02-07_9-42-07

#1 VMCA Root CA

Option #1 is the simplest option, and probably the one a lot of organizations will go with. This is relying on the new VMCA to provision and manage certificates for vCenter and ESXi hosts. The VMCA is automatically created upon PSC installation, and requires no further configuration. However, for services accessed by a web browser (such as the web client) you will get an SSL warning unless you explicitly trust the VMCA root in your browser of choice. This is akin to the VMware signed certificate method in years past. Except in 6.0 there’s now a central CA managing the certificates and their lifecycle. If you do nothing, this is what you will get. Better than in prior vSphere releases, but still not fully trusted certificates. For better security see option #2.

#2 Subordinate VMCA

This is an entirely new option in vSphere 6.0, and wasn’t remotely possible in prior releases. Basically what happens in this mode is that the VMCA imports a root signing certificate from you trusted enterprise root CA. The VMCA then becomes an official subordinate CA to your enterprise root(s). All the certificates issued by the VMCA are trusted by your organization, even the web services exposed in browsers. As you deploy new vSphere components that are VMCA aware, they will get issued trusted SSL certificates. Since the VMCA now manages ESXi 6.0 host certificates, your ESXi hosts will also be issued trusted certificates without any manual intervention.

The BIG downside to this, is also the big upside. The VMware CA is now issuing fully trusted certificates, which may go against company policy. Or, if you are in a regulated environment like the US DoD, there’s no way in hell they will allow you to stand up a trusted subordinate CA. So I would say this option is good for environments that want more than VMware issued certificates, but aren’t so regulated that a VMware subordinate CA would be strictly prohibited. Call this a good compromise between security and simplicity. Thank you VMware! For even more security see option #3.

2015-02-07_12-07-12

#3 External CA

Using an External CA is not new, and has always been in option in vSphere dating back many versions. It replaces all certificates in the environment by ones issued from the corporate trusted CAs. VMCA is bypassed, so this is a much more labor intensive process and much higher management complexity. This will be the only option in highly regulated environments, and will cause the most customer pain. All of the benefits of the new VMCA will be ignored, in favor of a higher security posture. This process is also totally different from that in vSphere 5.5, so get ready to learn yet another tedious procedure.

2015-02-07_12-08-23

#4 Hybrid

A hybrid scenario features the usage of the VMCA in combination with an external CA. For example, you could use the automated VMCA certificates for all “internal” certificates and ESXi hosts and only replace externally facing certificates (such as web client) with those from an external CA. This adds complexity to the VMCA subordinate option (#2), but is less work than using an external CA for all certificates.  Personally, I don’t see this solution being used too much. I think the other three will be more popular, and the level of regulation and security consciousness will ultimately determine which route to take.

Certificate Types

In the vSphere 5.x era each service was issued its own unique SSL certificate. As you may recall, each certificate had to contain a unique “OU” field otherwise SSO would barf. This does not scale well, as VMware is constantly adding new services to vCenter. Even in vSphere 5.5, my toolkit had to generate 11 certificates for all the services. Whew!

In vSphere 6.0 we now have several types of certificates. As shown in the VMware graphic below, a lot of services are use these ‘common’ certificates. This reduces the total number of certificates needed in the environment.

2015-02-07_11-38-36

The following table lists each of the certificate types used in vSphere 6.0, how they are provisioned, and where they are stored.

2014-11-22_13-36-40

ESXi Certificate: As has been the case for many years, this certificate is used by the ESXi host to encrypt nearly all communications. Nothing new here.

Machine SSL Certificate: Each node (embedded installation, management node, or PSC) has its own machine SSL certificate. All services running on this node use this certificate for end point encryption. The vCenter service (vpxd), VMware directory service (vmdir) also use these certificates.

Solution User Certificates: These certificates are used for authentication to the vCenter SSO service. Once the certificate is presented to SSO, SSO will issue a SAML token. The service, such as vpxd, can then use this token to authenticate to other services. Baseline solution user certificates include vpxd, vpxd-extensions, and vSphere-webclient.

VMware End Point Certificate Store (VECS)

The VMware End Point Certificate store (VECS) is a local repository for certificates and private keys. VECS is a mandatory component, and will be used even if you don’t sign your certificates with the VMCA. Remember that ESXi certificates are stored on the ESXi host and not in the VECS. The VECS includes a number of keystores including machine SSL certificates, trusted roots, CRLs, solution users (machine, vpxd, vpx-extension, vSphere-webclient) and other keystores such as those for vVols.

2015-02-07_12-03-29

Summary

Securing your virtual infrastructure is important. There are many attack vectors, and attacking SSL may not be the highest risk. But with vSphere 6.0 and my Toolkit script, replacing SSL certificates is easier than it used to be. So strongly consider taking the time to understand the new deployment methods, assess your business requirements, then take steps to secure your environment. It’s of little use to secure your SSL connections if you give the ESXi root and vCenter admin passwords to everyone.

Next up in the series will be vCenter upgrade and deployment best practices, in Part 4. You can check that out here.

© 2017 - Sitemap