vSphere 5.5 Toolkit Updated

This weekend I did a minor update to my VMware vSphere 5.5 SSL Toolkit script. It’s now at v1.59. I updated the OpenSSL download to use 0.9.8.zb, and also added a primitive PowerShell 3.0 check. PowerShell 3.0 and higher has always been required, but now I try and check for it. If you are running PS 3.0 and still get an error, then please leave a comment in this post. The logic isn’t all that intelligent, so may need tweaking.

If you aren’t familiar with my vSphere 5.5 toolkit script, then you can check out Part 8 of my 19 part vSphere 5.5 installation series. As always, you can download the latest version from vExpert.me/toolkit55.

Join the over 10,000 downloads of my Toolkit script and make your SSL life a lot easier.

Microsoft EMET 5.0 Released

Securing your Windows computer (both client and server) should be a top priority in your organization. To that end Microsoft just released EMET 5.0. It’s not a matter of IF, but WHEN you will have a security breach. But the trick is to raise the security bar to make it less likely that you will get hacked. One great tool for raising the security bar on Windows computers is Microsoft’s FREE EMET tool. The Enhanced Mitigation Experience Toolkit (EMET) 5.0 has a number of knobs to tweak the security posture of your system, and help protect both against known exploits and zero day exploits.

Back when I was responsible for building golden images, EMET was *always* in my images. I also install EMET on all of my home computers. In an enterprise environment you will need to test, test, test before you decide to roll it out. Some programs will have problems with EMET and may not launch, crash, or act weird. There are various knobs in EMET that can mitigate much of the side effects, and yet allow you to run in a more secure posture.

For the full EMET 5.0 announcement, see this MS blog post announcement. It covers the new features in 5.0. You can download EMET 5.0 here.

For my home system I use the maximum security settings profile, and also import all three protection profiles. Again, in the enterprise you will want to thoroughly test settings and will likely not be able to use the maximum security settings policy.



If you are responsible for making golden images, or work in your organization’s IT security department, do yourself a favor and seriously check out EMET and consider deploying it on all Windows computers.

VSS Labs vCert Manager Part 2

This is part 2 of the VSS Labs vCert Manager installation and configuration series. In Part 1 we got vCert Manager installed, and secured with a trusted SSL certificate. In this section we will get into the nuts and bolts configuration and start replacing certificates.

vCert Manager Configuration

1. First we will setup a SMTP server, which is used to send email notifications of various events such as expiring certificates. Login to vCert Manager and from the main menu select Settings. The in the left under Company Settings select Portal Settings.

2. Enter the SMTP server details for your organization. Notice that the tool supports SSL encrypted SMTP, and SMTP authentication. You can even test out the SMTP authentication from right within the tool. Here you can also setup different notification settings. I’ll just leave the defaults here.


3. You can also configure SYSLOG settings. You can easily change the port number, and protocol (TCP/UDP). This is great for services such as Splunk, where you can customize different SYSLOG listeners on different ports. Click the Save icon on the left to save all of your settings.


4. In the left pane click on Company Profile and fill out the details. These will be used for certificate generation.


5. In the left pane click on My Account. Here you can change the password for the default ‘admin’ account. Change it to a nice complex password.


6. In the left pane click on Sites change the site name to something relevant to you. This should reflect where the vCenter components reside. Mine are in San Diego.


7. The tool also supports role based access controls (RBAC), and you can add additional accounts that have different levels of permissions. Roles include Home, Cert Manager, Administration, Settings, Reports, Logs, About.

8. Now we need to establish a connection to our Microsoft CA. On the main page click on Administration in the top banner. In the left pane click on Certificate Authorities. Click on the green Add button. Fill in the details as needed. I would suggest setting up a service account that has proper permissions in your CA, vice your normal admin account like I show below. Better security, and better traceability. Shame on me. Click on Get Templates and select your VMware SSL template that you’ve already created.


After you add the CA, it will now be shown in the middle status pane.


9. In the left pane click on Infrastructures. Click on the green Add button. Enter your vCenter details, and service account information. Again, use a service account here and not your administrator account like I did. Test the connection to validate the information.


10. Now you will probably get a large certificate warning screen, since your vCenter certificate is probably not trusted at this point. Click on the I trust this certificate button.
















11. Next up is a credentials page where you need to enter passwords several times for the various components that it detects. After all of the passwords are entered, click on the Trust buttons for SSO and Inventory service. Note, that if you are using Windows authentication or SQL express for vCenter, just enter a dummy password in the DB Password field.



12. On the main menu bar click Cert Manager. You should now get a nice little graphic with the quantity of discovered components.


13. Click on the vCenter FQDN and you will see a table format of the same information. Click on the graphic to enlarge it.


We are now ready to actually replace the certificates. That will be coming up in Part 3. Stay tuned!


VSS Labs vCert Manager Part 1

Last August I wrote a blog post about this great new VMware SSL tool by VSS Labs called vCert Manager, which replaces many of your VMware SSL certificates all from the comfort of a nice GUI. It’s a full certificate lifecycle management tool for VMware vSphere and related components. For the full feature list and comparison with the free VMware tools, check out my post here. I’ll wait for as you read through that long article.

Ok with that out of the way, this time up I’ll actually walk you through the installation process, and see how easy it is to replace your vCenter 5.5 and ESXi host SSL certificates, all from a nice GUI. vCert manager is a licensed tool, but as you will see, has a lot of great features that no other tool that I’m aware of has.

If you are cash strapped and can only afford a free tool, then please remember my vSphere Toolkit script + the VMware certificate replacement tool combination. While the combination is less painful than the manual procedures, they fall very much short of the vCert Manager tool in terms of both functionality and ease of use.

VSS Labs has a special vExpert program, where you can get a NFR license that supports two vCenter servers and up to 10 ESXi hosts, for non-production usage. Their standard eval license supports 1 vCenter and 5 ESXi hosts.

What does it Update?

The list below is all of the VMware components that vCert Manager v1.2 can update:

  • vCenter 5.0: vCenter, Update Manager, Inventory Service, web client
  • vCenter 5.1: SSO, Inventory Service, vCenter, Web Client, Update Manager, Orchestrator
  • vCenter 5.5 (single location): SSO, Inventory Service, vCenter, Web Client, Update Manager, orchestrator
  • ESX/ESXi: 4.0 through 5.5

My Environment

In my environment I have Microsoft vCenter 5.5 server, and three ESXi 5.5 hosts. All of the vCenter components are installed on one server. I also have a two tiered Microsoft CA hierarchy. The root is offline, and I have an online issuing CA. Both are Windows Server 2012 R2. I have also setup a certificate template for VMware products, which you can read about here. You must have a certificate template, so if you don’t, go configure one following my guide.


1. I provisioned a Windows Server 2012 R2 VM from my standard template, which has no additional roles or features beyond .NET Framework 3.5 and .NET Framework 4.5. The tool is IIS and ASP.NET based, so we need to install IIS and ASP.NET to get started. Launch Server Manager and select the Web Server (IIS) role.


2. On the Features page expand .NET Framework 4.5 and select ASP.NET 4.5. Click through the rest of the wizard and wait for the components to install.


3. Go back to server manager and under Web Server enable ASP.NET 4.5, ISAPI Extensions and ISAPI Filters. Wait for the installation to complete.



4. Download the VSS Labs vCert Manager binary and start the installer. Click through the wizard until you get to the following screen. I chose the defaults, which are shown below. If you wanted, you could create a custom service account which the application pool would run under.

VSS Labs

5. Next up you can specify an account which the service uses to logon. Again, I left the default here. By using this default the computer machine account will need SQL database permissions, which I cover later.


6. Now we need to specify the SQL database connectivity, so that it can create and use a SQL database. I simply entered the database server, database name, and port. A great feature is supporting SQL encryption. According to the install guide a future version will support a system ODBC connection. The account I was logged in as has SQL sysadmin rights, so the Test Connection passed.


7. Finally you can change the installation directory. I choose the default here. The installation took less than a minute after I completed the wizard.


SQL Permissions

1. Go to your SQL server and add a new security login. Use the computer name (followed by a $) that vCert manager is installed on and change the database to the vCert manager database that was created during the installation process.






The installer will create a non-trusted SSL certificate, which I recommend we replace with one from your trusted CA. In my case I have an autoenrollment policy so that my computer already has a machine certificate that I can bind with in IIS. If your machine does not already have a machine certificate, then issue one and install it in the machine’s certificate store.

1. Open IIS and navigate to the vCert Manager site in the left pane. In the right pane click on Bindings then modify the bindings for the 8056 port entry. Select the thumbprint for your trusted machine certificate.



2. Open a web browser and navigate to https://FQDN:8056 and you should get a nice looking login page. They suggest using Chrome v26, Firefox 20 or IE 10. I used IE 11 and didn’t notice any issues. Verify there are no SSL warnings.

vCert Manager

And there you go..vCert manager is now fully installed and secured via a trusted SSL certificate. In the next installment we will dive into the management interface and start replacing certificates.

vCenter 5.5 Update 1b Now Out

2014-06-13_8-54-42Earlier this year the Heartbleed OpenSSL vulnerability came to light, and many products from many different vendors were affected. Heartbleed is a very serious vulnerability, and you should conduct a thorough audit to validate all of your software and hardware components are not affected. My bet is that many of your product are affected, and VMware was far from alone in having to issue software updates. Fresh off the presses is vCenter 5.5 Update 1b, which addresses CVE-2014-0224, Heartbleed. They have updated their OpenSSL libraries to 0.9.8za, 1.0.0m and 1.0.1h. I find it interesting that all three versions are used within the product.

As usual, this minor update includes some other bug fixes a well and not major new features. Other resolved issues include:

Host profile answer file might not be applied when you reboot the ESXi host
After you restart the vCenter services, if you use Auto Deploy to reboot a stateless ESXi host, the host profile answer file might not be applied to the ESXi host and the host configuration might fail. This issue occurs if the reference host is not available.

Under certain conditions, Virtual SAN storage providers might not be created automatically after you enable Virtual SAN on a cluster
When you enable Virtual SAN on a cluster, Virtual SAN might fail to automatically configure and register storage providers for the hosts in the cluster, even after you perform a resynchronization operation.

As is typical with VMware release notes there is a handy dandy list of known issues, which you should thoroughly read through before deploying this update. A few known issues that jumped out at me include:

  • Upgrade from vCenter Single Sign-On 5.1 Update x to 5.5 Update 1a fails if the password set for administrator@vsphere.local contains double quotation mark (“)
  • After you upgrade vCenter Server 4.x to 5.5 Update 1a, vCenter Server is inaccessible through vSphere Web Client (Due to SSL issues)
  • Attempts to upgrade from earlier versions of vCenter Single Sign-On 5.5 to 5.5 Update 1a might fail if the Windows Error Reporting Service is disabled
  • Installation using Simple Install fails on Windows Server 2008 R2 and Windows 2012 hosts


Given the extreme severity of the Heartbleed issue, this is one security update that I recommend you immediately start testing in your lab environment and roll out into product once you are satisfied that it hasn’t broken anything in your environment. You can find the full release notes here. You can download the updated ISO here.

vSphere 5.5 Toolkit v1.57 Released

Now that vSphere 5.5 has been out for a few months, if you haven’t already started working with it in your test environment, you should! A lot of great new features, and major improvements to the SSO experience. However, you still may find it a bit challenging to secure vCenter and ESXi hosts with trusted SSL certificates. So hot off the press is a minor bug fix version of my vSphere 5.5 Toolkit script. You can download the new version here. In case you aren’t familiar with it, here are some of the features:

  • Downloads and installs the proper version of OpenSSL (0.9.8.Y) if it’s not already installed
  • Creates 2048 bit RSA private keys in the proper format
  • Creates a directory for each service bundle of SSL certificates
  • Generates seven OpenSSL configuration files, one for each certificate, in the appropriate directory
  • Downloads both root and subordinate root public certificates
  • Submits the CSRs to the online CA and downloads the certificates
  • Creates the needed service PEM files for the vCenter certificate automation tool
  • Creates the required root/subordinate PEM files
  • Handles the special SSO 5.5 certificate requirements
  • Automatically uses the hostname of the server you run the script on for all certificates
  • Creates a pre-filled vCenter Certificate Automation environment script – Just run!
  • Works with offline CAs
  • Creates SSO 5.5 certificate replacement files – Only used if manual replacing certs
  • Creates customized SQL vCenter and VUM database creation script
  • Creates SQL ODBC DSNs for vCenter and VUM
  • Automatically downloads and installs SQL 2008 R2 or SQL 2012 client package
  • Linux vCenter Server Appliance support for online minting and offline CSR creation
  • Creates certificates for Auto Deploy, Dump Collector, Syslog collector, Authentication Proxy
  • Support Microsoft CAs that require manual certificate approval

Version 1.57, which you can download here, has a couple of minor fixes:

  • More robust handling of non-internet connected servers
  • Removed line continuation characters that caused issues for some people
  • Fixed bug when no subordinate CA was present (v1.56)
  • Changed Microsoft “renewal” default to 0 for root/subordinate (v1.56)

Although the script was developed for all-in-one vCenter servers, it will work for instances where services are distributed across several servers. You will just have to be intelligent about using the correct hostnames and merging together directories with the proper certificates. Not rocket science, but does take a little manual work.

In case you missed it, I also have a 19 part vCenter 5.5 installation and configuration series that covers how to use the Toolkit script in gory detail. You can check out that series here. The ESXi SSL portion of the tool also works with vSphere 5.0 and 5.1, so you aren’t just limited to 5.5 hosts.

1-11-2014 2-27-28 PM

Windows Server 2012 R2 Two-Tier PKI CA Pt. 3

1-10-2014 6-57-56 AMNow that we have our Windows Server 2012 R2 certificate authority configured in Part 1, and our subordinate setup in Part 2, now we should setup autoenrollment and secure the subordinate’s web certificate services with SSL. Autoenrollment is where domain joined Windows computers are automatically issued a computer certificate. Services such as IIS and Microsoft SCCM can take advantage of these certificates. Finally, I’ll show you how to configure certificate delegation so authorized administrators in your organization can submit certificate requests for certain templates. This is a short series, at just three installments. But this should point you in the right direction for thinking about how to deploy your two-tier Certificate Authority on Windows Server 2012 r2.

Autoenrollment Configuration

1. Open your domain level GPO (Default Domain Policy in my case) and navigate to Public Key Policies as shown in the figure below. Double click on the highlighted policy.

1-4-2014 8-51-24 PM

2. Enable the policy and check the two options below.

1-4-2014 8-51-07 PM3. On your subordinate CA, open the CA snap-in and manage the Certificate Templates as shown below.

1-4-2014 8-54-37 PM4. Scroll down and locate Workstation Authentication. Right click and Duplicate the template.

5. Click on the General tab and enter a template name (any name). I’ll use Client-Server Authentication. I also changed the validity period to 2 years.

1-4-2014 8-58-07 PM

6. Click on the Extensions tab. Highlight Application Policies and click Edit. Add Server Authentication.

1-4-2014 9-00-46 PM

7. Click on the Security tab and modify the Domain Computers group to enable Autoenroll. Close out the template and template window.

1-4-2014 9-01-46 PM

8. Back in the issuing CA console right click on Certificate Templates, select New, then Certificate Template to Issue. Select the template name you just created. Wait a few minutes for the settings to simmer a bit. If you want you could also publish the Domain Controller template. This will enable the DCs to offer LDAPS services. If the template you just created is not listed, you can simply wait a bit or restart the CA services and that should kick it in the pants.

windows server 2012 r2 certificate authority

Autoenrollment Validation

1. Open an elevated command prompt or Powershell and type gpupdate /force. Wait a couple of minutes, as certificate enrollment is not always instant.

2. Open a blank MMC console and add the Certificates snap-in. Manage the Computer account.

1-4-2014 9-14-11 PM

3. On your subordinate CA you should now see two certificates. In my case the top certificate was the one issued by the autoenrollment policy.

1-4-2014 9-16-20 PM

4. You can verify the certificate was issued from the proper template by opening the properties then on the Details tab look for the Certificate Template Information property. It will clearly state the template name used to create the certificate.

1-4-2014 9-17-29 PM

5. As the GPO refreshes on other computers in the domain, they should also be issued a certificate as well. Autoenrollment can run into snags, so I have seen cases where everything has been configured properly but for some reason a certificate is not issued.

Configure CA Web Services for SSL

1. After the autoenrollment certificate has been validated on the subordinate CA, open the IIS Manager on your subordinate CA.

2. In the left pane select Default Web Site. In the right pane select Bindings.

3. Click on https then click Edit.

4. Select the SSL certificate that was created from the client-server template. You can view the certificate in the GUI if you aren’t sure which one to pick.

1-4-2014 9-35-37 PM

5. Open IE and navigate to the FQDN of your subordinate CA and to the certsrv site (e.g. https://D002Misc01.contoso.local/certsrv). You will likely be prompted for credentials, then presented with the standard ADCS home page. You should not have any SSL errors or warnings.

1-4-2014 9-39-32 PM

Template Delegation

1. On your subordinate CA and open the Certificate Template manager as shown below.

1-10-2014 7-26-06 AM

2. Locate the certificate template which you want to delegate. In my case I have a VMware-SSL template that I want to delegate to the group we created earlier in this series. Open the properties for the certificate template and select the Security tab. Add the Role_Issue Certificates group (or whatever your group is called) and give it the Enroll permission.

1-10-2014 7-28-14 AM

3. Optionally you configure the CA to allow requests to be submitted, but require a CA administrator to approve the certificates before they can be issued. If you want to do this, open the Issuance Requirements tab and check to the CA certificate manager approval box. This would defeat the purpose of autoenrollment certificates, such as those for computers, so generally this would be for certificates that users are requesting.

1-10-2014 7-32-51 AM

What’s Next?

If you want to issue SSL certificates for your VMware infrastructure, then you can check out my post here for the template requirements. Although that article is for vSphere 5.5, the template will also work for vSphere 4.x and 5.x. Now you have a fully functional, for lab/home usage, offline root and online subordinate CA. As I stated in Part 1, this guide just shows you the general technical steps for a two-tier Certificate Authority. There’s a lot of processes and procedures that an organization needs to flesh out and document before deploying PKI in the environment. There could be legal or other consequences if you just throw this on a production network and then down the road experience security issues which can be traced back to a poorly implemented CA.

Windows Server 2012 R2 Two-Tier PKI CA Pt. 2

1-5-2014 2-43-05 PMNow that our root Windows Server 2012 R2 certificate authority is installed and published to Active Directory from Part 1, it is time to bring online our subordinate CA. The subordinate CA will be our online issuing CA, since it will be the CA which issues all certificates, be they for users, computers, ESXi hosts, etc. The VM will be joined to the domain, and be online 100% of the time.

As with the offline root, you should perform hardening of this VM as well. Enabling the Windows firewall (or a third party one), anti-virus software, Microsoft EMET, and following Microsoft security baseline settings are all strongly recommended. If you have security software that can monitor file changes or system integrity, that too would be a great idea. Auditing tools such as Splunk, for real time alerting, would be ideal for defense in depth.

Install Windows Server 2012 R2 Subordinate CA

1. Use Notepad and create a file called CAPolicy.inf in C:\Windows on your subordinate VM. Use the code snippet below, but change the URL to match that previously used in configuring your offline root.

Signature="$Windows NT$"
Notice="Legal Policy Statement"

4. Run the following PowerShell command. Change the CACommonName as needed. The command will completely instantly.

Add-WindowsFeature Adcs-Cert-Authority -IncludeManagementTools
Add-WindowsFeature Adcs-web-enrollment
Install-AdcsCertificationAuthority -CAType EnterpriseSubordinateCA -CACommonName "IssuingCA-D002MISC01" -KeyLength 2048 -HashAlgorithm SHA256 -CryptoProviderName "RSA#Microsoft Software Key Storage Provider"

5. Copy the resulting request (see the yellow information text from the last command for the path and file name) to the offline CA.

6. On the offline CA type the following command, using your filename:

certreq -submit D002MISC01.contoso.local_IssuingCA-D002MISC01.req

7. You will now see that the request is pending. Take note of the RequestId, as it will be unique to you.

1-4-2014 7-47-29 PM

8. Open the CA Manager snap-in on your offline root and issue the pending certificate.

1-4-2014 7-48-25 PM9. While still on the offline CA, enter the following command to download the new certificate. Replace “2” with your request ID, and change the filename as you see fit.

certreq -retrieve 2 c:\D002MISC01.contoso.local_IssuingCA-D002MISC01.crt

10. Copy the certificate file to the online subordinate CA. Note: Do NOT place it in the pki directory. Run the commands below to install the new certificate. Once the certificate is installed, delete the file and empty the trashcan.

Certutil –installcert a:\ D002MISC01.contoso.local_IssuingCA-D002MISC01.crt
start-service certsvc
copy c:\Windows\system32\certsrv\certenroll\*.cr* d:\pki\

Configure Subordinate CDPs

1. Next up we need to configure the proper CRLs for our subordinate CA. Enter the following commands in an elevated Powershell on your subordinate CA.

$crllist = Get-CACrlDistributionPoint; foreach ($crl in $crllist) {Remove-CACrlDistributionPoint $crl.uri -Force};
Add-CACRLDistributionPoint -Uri C:\Windows\System32\CertSrv\CertEnroll\%3%8%9.crl -PublishToServer -PublishDeltaToServer -Force
Add-CACRLDistributionPoint -Uri http://www.contoso.local/pki/%3%8%9.crl">http://www.contoso.local/pki/%3%8%9.crl -AddToCertificateCDP -Force
Add-CACRLDistributionPoint -Uri file://\\D002Misc01.contoso.local\pki\%3%8%9.crl" file://\\D002Misc01.contoso.local\pki\%3%8%9.crl -PublishToServer -PublishDeltaToServer -Force
$aialist = Get-CAAuthorityInformationAccess; foreach ($aia in $aialist) {Remove-CAAuthorityInformationAccess $aia.uri -Force};
Add-CAAuthorityInformationAccess -AddToCertificateAia http://www.contoso.local/pki/%1_%3%4.crt" http://www.contoso.local/pki/%1_%3%4.crt -Force
Certutil -setreg CA\CRLPeriodUnits 2
Certutil -setreg CA\CRLPeriod "Weeks"
Certutil -setreg CA\CRLDeltaPeriodUnits 1
Certutil -setreg CA\CRLDeltaPeriod "Days"
Certutil -setreg CA\CRLOverlapPeriodUnits 12
Certutil -setreg CA\CRLOverlapPeriod "Hours"
Certutil -setreg CA\ValidityPeriodUnits 5
Certutil -setreg CA\ValidityPeriod "Years"
certutil -setreg CA\AuditFilter 127
restart-service certsvc
certutil -crl

CA Delegation

1. Now that our online subordinate CA is up and running, for the most part, it is a good idea to delegate who has rights to manage the CA and issue certificates. I’m going to create two roles: One that can manage all aspects of the CA, and another that can just mint specific certificates. In AD create two groups: Role_CA Manager and Role_Issue Certificates. Or use whatever names you like.

2. On your subordinate CA, launch the CA MMC Snap-in. Right click on the CA name, open the properties, and select the Security tab, and add the Role_CA Manager group. Give it Manage CA permissions. If you want, you can remove rights from Domain Admins or Enterprise Admins, should you want to more tightly control CA access (which you should).

windows server 2012 r2 certificate authority


At this point in the configuration there are no published templates. So in the following post we will configure a couple of templates, and I’ll show you how to delegate permissions so that other administrators can mint their own certificates. In this installment we’ve done the bulk of the subordinate CA configuration. At this point the CA is now functional, although no templates have been configured. So coming up in the next installment is, among other things, the process to configure templates and computer autoenrollment. Check out Part 3 here.

Windows Server 2012 R2 Two-Tier PKI CA Pt. 1

windows server 2012 R2 certificate authorityWhile I have written a number of articles focused on SSL certificates and templates, I have not done a mini-series on how to actually install a Windows Certificate Authority. For this series I’m using Windows Server 2012 R2, but the steps are pretty much identical for Windows Server 2012. Microsoft blogs have several PKI configuration series, which directly guided the content of this series. But I always have my own spin, so I think its worthwhile to do yet anther blog post on configuring a MS CA…the “Mr. SSL” way.

Windows Server 2012 R2 Certificate Authority

The process is fairly simple: Build an offline root, create an online issuing CA, setup a couple of templates, setup auto-enrollment, then do a little post setup configuration. This requires two VMs, each running Windows Server 2012 R2 (or plain 2012 if you wish).

Building an enterprise CA is non-trivial, and should be highly process oriented. While this short series will provide the steps how to configure a two tiered hierarchy, it alone is not enterprise grade and ready for a fortune 500 company. Many operational procedures, access controls, etc. need to be defined by the organization. For example, who can issue certificates? Who can revoke them? Do users need PKI certificates or just computers? How about key recovery? Disaster recovery? Do you need a hardware security module (HSM)? Do you require FIPS compliance? What ciphers and hashing algorithms will you allow? Where do you store the offline CA?

As you can see, there are many questions and processes that need to be well documented for a solid PKI solution. However, for a lab environment where you want to test out a two-tiered model, then this short series is for you. Please don’t take this solution as-is and throw it into production. You will have a false sense of security and possibly do more harm than good.

The Microsoft CA issues industry standard certificates (x.509), and thus will work with third party hardware and software. For instance, they will work perfectly fine on the Linux vCenter appliance, or your hardware load balancers. You just need to use the proper certificate template, and verify compatible algorithms.

Offline Root CA Hardening

1. Provision a standalone Windows Server 2012 R2 server. I used vCenter 5.5 with customization specifications to create the VM. You can use the ‘standard’ edition of the OS since all SKUs in 2012 have the exact same feature set, unlike 2008 R2 and earlier. For security purposes I would not provision a NIC, or remove the NIC after you’ve built the CA to prevent future network attacks.

2. Configure a virtual floppy for the offline CA VM. This is a good way to transfer data between the offline CA and the subordinate, which is required during the configuration process. Yes you could connect a NIC, but then your offline CA is no longer offline and exposed to network attacks. Media needs to be read/write, so an ISO image will not suffice. You can use a tool like WinImage to create a floppy image.

3. Open the local security policy and modify the Audit Object Access to record Success and Failures. This is needed to audit certain CA actions, in conjunction with a CA flag we will set later on.

1-5-2014 1-18-44 PM

4. Depending on your VM template hardening, you may or may not need to modify the password policy. Again in the Local Security editor. Modify to meet your organization’s security requirements.

1-5-2014 1-23-08 PM

5. You should also rename the Administrator account, if that’s not already built into your templates. Make sure to record the new name, or you could be in a pickle. For good measure I’d rename the guest account, although it should be disabled.

1-5-2014 1-25-28 PM

6. Obviously you should change the administrator password and not use your template default. Be sure to record the password in a secure location.

7. You should also think about where you will store the offline CA VM once it is build and this project is complete. If you leave it sitting on a production ESXi host, then it would be fairly trivial to power on the VM and compromise it. I would not call storing your “offline” CA in a powered off state on a production ESXi host “offline”. I would look at exporting the VM to an OVF file, then storing that file on removable media in a very secure location. You could use a DVD, Blu-Ray, or USB stick.

Install Offline Root CA

1. After your VM is provisioned and hardened, make sure the computer name is configured. In my case the offline CA is name D002CA01. Reboot if you changed the name.

2. Use Notepad and create a file called CAPolicy.inf in C:\Windows. Use the code snippet below, but change the URL. This URL is where your Certification Practice Statement (CPS) is located. It will also be where the CRL (certificate revocation list) will be published. For a production deployment you’d want to create a CPS, but for this exercise we will skip it, however the URL will be configured for future usage. For additional details see this TechNet link. You probably want to use a different URL like CA.yourdomain or PKI.yourdomain since we will be publishing other data to this address such as the CRL. For simplicity I stuck with www.contoso.local. Make sure the filename does not have any extra extensions like .txt. Verify from the command line.

Signature="$Windows NT$"
Notice="Legal Policy Statement"

3. Run the following PowerShell command. Change the CACommonName as needed. The command will complete instantly. I would make it clear in the name that this is the Root CA. This name will be present in all issued certificates, so make it obvious what it is and not just some generic hostname that is not meaningful. Notice that we are using SHA256 here, since SHA1 is no longer considered secure. You could also use SHA512.

Add-WindowsFeature Adcs-Cert-Authority -IncludeManagementTools

Install-AdcsCertificationAuthority –CAType StandaloneRootCA –CACommonName "ContosoRootCA" –KeyLength 2048 –HashAlgorithm SHA256 –CryptoProviderName "RSA#Microsoft Software Key Storage Provider"

1-4-2014 2-05-36 PM

4. Run the following commands, using the appropriate URL for your organization. We aren’t using HTTPS here, because that requires SSL and certificate validation. This is just used to download the CPS and CRLs, so don’t get clever and use HTTPS here. We will configure SSL for the web enrollment module, though.

$crllist = Get-CACrlDistributionPoint; foreach ($crl in $crllist) {Remove-CACrlDistributionPoint $crl.uri -Force};
Add-CACRLDistributionPoint -Uri C:\Windows\System32\CertSrv\CertEnroll\%3%8.crl -PublishToServer -Force
Add-CACRLDistributionPoint -Uri http://www.contoso.local/pki/%3%8.crl -AddToCertificateCDP -Force
$aialist = Get-CAAuthorityInformationAccess; foreach ($aia in $aialist) {Remove-CAAuthorityInformationAccess $aia.uri -Force};
Certutil -setreg CA\CRLOverlapPeriodUnits 12
Certutil -setreg CA\CRLOverlapPeriod "Hours"
Certutil -setreg CA\ValidityPeriodUnits 10
Certutil -setreg CA\ValidityPeriod "Years"
Certutil -setreg CA\AuditFilter 127
restart-service certsvc
certutil -crl

5. Verify that two and only two CRL distribution points are configured.

Get-CACRLDistributionPoint | format-list

1-4-2014 3-12-39 PM6. Navigate to C:\Windows\System32\CertSrv\CertEnroll. You should see two files, one ending in CRL and another ending in .CRT. These two files need to be copied to what will be the online subordinate CA.

1-4-2014 4-17-37 PM

Publish Root CA to the Forest

1. Provision a Windows Server 2012 R2 VM which will be your online CA. Join it to the domain. In my case the VM is named D002MISC01. Do not try and be clever and use a Domain Controller. The server will later need IIS installed and access to local accounts, which is not possible on a DC. So use a member server for your online CA, even in a home lab.

2. Login to what will be your online subordinate CA with an account that is a member of both Domain Admins and Enterprise Admins. Mount the media which has the two files copied from your offline CA. Open an elevated Powershell and enter the following commands, using the file names for your instance. This will publish the offline root CA information to AD, just as if it were an online CA. By doing this all domain joined clients will automatically trust your root CA. If you have standalone computers, then you can import the .crt file into their trusted certificate store.

certutil –dspublish –f D002CA01_ContosoRootCA.crt RootCA
certutil –addstore –f root D002CA01_ContosoRootCA.crt
certutil –addstore –f root ContosoRootCA.crl

1-4-2014 4-19-11 PM

CPS and CRL Distribution

1. Now you need create a DNS record for the host that will be publishing your online CA information. In this case it’s D002MISC01, and per my previous steps I stuck with ‘www’ as the site name. I’m assuming the proper DNS zone already exists, since you have a domain with Active Directory up and running. This must be configured prior to continuing, as the subordinate will fail to properly configure if the CRL file is not available.

1-4-2014 4-25-31 PM

2. We need to install IIS, since we will be distributing the CPS and CRL via the HTTP. On the VM which will be your online CA, run the following command:

Install-WindowsFeature Web-WebServer -IncludeManagementTools

3. Open an elevated PowerShell and enter the following commands. If you have an official CPS, then you can skip the second command and just copy your cps.txt file to the directory. For security purposes I’d recommend putting the files on the D: drive, so you aren’t serving content from the OS drive.

new-item -path D:\pki -type directory
write-output "This is a sample CPS. Modify as needed." | out-file D:\pki\cps.txt
new-smbshare -name pki D:\pki -FullAccess SYSTEM,"Contoso\Domain Admins" -ChangeAccess "Contoso\Cert Publishers"

4. Open the IIS Manager and add a Virtual Directory as shown below.

1-4-2014 7-19-27 PM

1-4-2014 7-20-39 PM

5. Verify pki is selected in the left pane, then single click Authentication in the middle pane, and in the right Actions pane click on Edit Permissions.

6. Select the Security tab and select Edit. Add the Cert Publishers group with Modify permissions (which will add several others under it).

1-4-2014 7-10-14 PM

7. In the same dialog box, click add but change the from this location to the local computer. Manually enter IIS AppPool\DefaultAppPool. Leave the default permissions. If you use the user/group browser this will not be listed, so please manually enter it.

8. At this point any anonymous browser can now read your CPS statement and see the public root certificate. You can test this by going to http://www.yourdomain/pki/cps.txt and verify the sample file opens.

9. In the middle pane, with pki still selected, click once on Request Filtering. In the right pane click on Edit Feature Settings and check the box next to Allow double escaping.

1-4-2014 7-12-45 PM

10. Run iisreset from an elevated Powershell command.


In this installment we’ve configured our offline root CA, performed some hardening, and published the root CA information to the domain. All computers in the domain will now trust your root CA. We also configured IIS to serve up your CPS and CRLs to anonymous users. Next up is configuring the online subordinate CA. Check out the next installment in Part 2.

VMware SSL pain? vCert Manager to the Rescue

VSS LabsOne of the most critical aspects to securing your VMware vSphere environment is replacing the self-signed certificates with trusted certificates. VSS Labs is releasing a new product, called vCert Manager, which will vastly improve the SSL certificate management experience for VMware customers.

Over the years I’ve written several articles detailing the process, which got exponentially harder with vSphere 5.1. I often hear users complain about the SSL certificate replacement process. Feedback on my blog has been quite verbose and colorful…some posts had to be censored. If you are struggling with the vSphere 5.1 installation, you can check out my vSphere 5.1 resources here.

While VMware did release the vCenter Certificate Automation tool earlier this year for usage with vCenter 5.1, it is command line/menu driven and does not fully automate the entire process. Given this tool was the only game in town I recommended my readers use the tool. You can find my multi-part usage series for the tool here.

The real game changer is the enterprise-scale VSS Labs vCert Manager. This goes way beyond the basic vCenter Certificate Automation tool, and provides full certificate lifecycle support for securing your vSphere environment with trusted SSL certificates. By full lifecycle I mean from automated vSphere infrastructure discovery, CSR generation, certificate minting, certificate installation, email expiration notices, automatic expired certificate updating, auditing, reporting, plus a lot more.

The entire process is managed via a web interface (securable with SSL), and requires no knowledge of OpenSSL, CSRs, PEM files, or leaf certificates. Plus it supports vSphere 4.x and 5.x environments, including ESXi hosts. The full support matrix is below.

If you are attending VMworld 2013 in San Francisco, be sure to stop by the New Innovator Pavilion and find the VSS Labs booth. Once you see the tool in action, you will be chomping at the bit to get a copy.

vCert Manager Features

Since the product has a very long feature list I’ve put together three tables that compare and contrast the features of the VSS Labs vCert Manager product and the VMware Certificate Automation Tool v1.01. The first table covers the VMware product/feature support of each tool. The second table covers certificate support, and the third table compares the tool core feature set and licensing.

VMware Product Support

VSS Labs VMware Support

As you can see from the table above, vCert Manager supports a broad range of VMware infrastructure. The VMware vCenter Certificate Automation tool is limited to just replacing vCenter 5.1 certificates. Version 1.01 has no support for prior versions of vCenter or ESXi hosts. Good tool for a specific use case, don’t get me wrong.

In contrast the vCert Manager product supports vSphere 4.0 through 5.1, including their respective ESX/ESXi host versions. I expect both companies to release updated certificate management products as new vSphere releases hit the streets. Neither tool has yet expanded to vCOPS, vCloud Director, or Horizon View. VSS Labs is targeting vCloud Director and Horizon View support in the near term, likely around the end of September.

Another great feature is that vCert Manager can manage dozens of vCenter instances and thousands of ESXi hosts from a single server and interface. No need to install the tool on each vCenter server. Credentials are server and host specific, so you can manage them all even if they are in different business units with different credentials.

Certificate Support

VSS Labs Certificate Support

Next to comprehensive VMware product support, broad certificate authority support and a high degree of automation is extremely important. This is where the vCert Manager product really shines. The VMware vCenter Certificate Automation tool requires a bit of manual pre-requisite work and using intermediate CAs can be tricky. Version 1.0.1 added CSR generation capability, so I certainly appreciate the added ease of use.

In contrast, vCert Manager automates the ENTIRE process, and you don’t have to know anything about certificate properties, SANs, how many certificates you need to mint, or how to submit and download certs from your CA. It can be configured to interface with an online Microsoft CA, so everything happens behind the scenes. Offline CAs are supported to, and as much of the process as possible is automated given the human intervention needed to mint the certificates.

In short, if you have an online CA the entire certificate request creation, minting, and deployment is executed through a GUI interface in an automated and orchestrated fashion. This makes certificate replacement almost foolproof.

Tool Features

VSS Labs vCert Manager Features

The most striking difference to me between the two tools is the user interface. vCert Manager sports a fully webserver driven GUI interface for all operations. If you haven’t used the VMware vCenter Certificate Automation tool you may not know that it’s completely command window based with text menus that instruct you in what order to perform the various steps. The VMware tool also needs to be installed on each server that has a vCenter component, unlike the vCert Manager which is installed on a single server and can manage thousands of vSphere components.

When updating ESXi hosts it requires DRS be enabled on the cluster so that it can put the host in maintenance mode, update the certificate, and return the host into production. This is all automated and when certificate is about to expire, it will automatically replace the certificates in the same manner.

Other great enterprise features of vCert Manager include syslog, full audit trail, basic reporting, and role-based access controls. Unlike the VMware tool, vCert Manager DOES require a Microsoft SQL database. So that’s a little extra configuration, but the installer does most of that work too. It supports Windows Authentication to the SQL database. SQL 2008, 2008 R2, and 2012 are supported.

Licensing and Support

There of course has to be a discussion about licensing and support. The VMware tool is free and if you have active SnS with VMware, they will support the tool for production usage. Clearly that’s a great value, and does ease much of the pain with vSphere 5.1 certificates. I certainly appreciate VMware releasing the tool, and the improvements in v1.01. I expect more improvements in the future.

vCert Manager is a licensed product, and support is provided by VSS Labs. The licensing model is per vCenter server, and per actively managed component (such as an ESXi host). For pricing details you will need to contact VSS Labs.

How to get a Free License

VSS Labs has a free NFR license available for home lab usage, with a special bonus for current fellow VMware vExperts. The tool will discover all of your components, even if they exceed your license, but you can only actively manage the number up to your licensed threshold.

8-19-2013 8-50-46 PM


vCert Manager should go GA by the end of August, if not sooner. When it does go live I’ll post a link to the download. I’ve been using the beta versions, and once the code goes GA I will post a multi-part blog post series on using the tool. So stay tuned!

For a version 1.0 product, I’m very excited to see the depth of support vCert Manager has packed into their first release. Throughout the beta process the company took feedback very seriously. After spending countless hours writing blog articles on VMware SSL certificates over the years, you have no idea how glad I am to see such a comprehensive tool. Maybe less blog articles for me to publish, but better for the community. Less hair pulling, and frustration for all!

The small home lab NFR licenses will give anyone interested in the product a good feel for how it works. I really appreciate the vExpert “bonus”, which will help the community test it against various vCenter versions more easily.

Sample Screenshots

The screenshots below are from a beta version, but will give you a little taste of the user interface and features. My installation series will contain a boatload of screenshots, once I get the GA code.

vCert Manager

vCert Manager Summary

vCert Manager Configuration

vCert Manager certificate authority

vCert Manager managed components

© 2017 - Sitemap