vCenter 4.1 U1 and FIPS encryption: Verify your IE Settings

During my regression testing of my vCenter 4.1 U1 installation instructions on Windows Server 2008 R2, I came across a problem that made me scratch my head. I was updating the vCenter SSL certificates, per my blog here. However, when I opened IE and tried to connect to the vCenter default home page would not come up. I got Internet Explorer cannot display the webpage.

OK I thought, maybe I goofed up the SSL certificates. I regenerated them, and nope, no good! The Windows Server 2008 R2 template that I’m using is locked down and has many security features enabled, including FIPS compliant encryption.

You can connect to vCenter with the vSphere client, but it appears the web services on port 443 are broken. For example, as I mentioned, the vCenter home page would not come up, the vCenter Service Status screen would not open, and performance graphs were also broken.

After additional research since my original post, the root cause appears to be the combination of two security settings: FIPS compliance, AND restricting what encryption algorithms IE is allowed to use.

The IE settings that cause the problem is the unchecking of TLS 1.0, as shown below.

This in combination with enabling FIPS on the server, as shown below, create a situation that doesn’t allow the TLS handshake to complete, so web based services that rely on IE settings break.

The lesson here is that if you have FIPS encryption enabled on the computer that you are accessing vCenter from, ensure your IE settings allow TLS 1.0. Normally TLS 1.0 is checked, so this won’t be a problem for most people. But if you are trying to enhance security by only allowing TLS 1.1 or higher, then you will run into issues.