vSphere 6.0 Toolkit Update

In my new role at Nutanix I’ve had the pleasure of working with end customers, and configuring their vSphere 6.0 environment. During this process, SSL certificates have come up. Surprisingly, thus far my clients have chosen the VMCA method of deploying certificates. This is great, as it automates certificate deployments in a vSphere 6.0 environment. Even with the VMware certificate tools, there are some manual steps for configuring the VMCA. My vSphere 6.0 toolkit automates most of those steps.

However, while going through the process we stumbled upon a slight bug in my Toolkit when using an intermediate certificate authority. I’ve since fixed that bug, and uploaded the latest vSphere 6.0 SSL Toolkit here.

I’ve been exceptionally busy the last few months, which is why blogging and updating the Toolkit script has taken a back seat. But I did want to get this script update pushed out so other customers don’t run into VMCA problems.

If you are unfamiliar with my vSphere 6.0 SSL Toolkit, then read up on my full vSphere 6.0 installation series here.

VMworld 2015: vSphere 6.0 in the Real World

Session INF4712

Compatibility Maximums – Review the document and stay within the guidelines.

vCenter 6 Platform choice: Windows and VCSA support same maximums and performance

  • Up to you, but look at things like Linux experience, licensing, existing skills, etc.

vCenter – New deployment architecture

  • PSC – SSO, License service, lookup service, vmdir, VMCA
  • vCenter – web client, inventory service, auto deploy, ESXi dump collector, syslog collector, etc.

PSC – Which architecture?

  • Embedded: Single site, no expansion past one vCenter
  • External: Supports up to 4 vCenters. HA mode is much more complex (3rd party load balancer)
  • Multiple sites – PSCs in each site, and replicate with each other.
  • Max size: 6 PSCs, 3 sites, 10 vCenters
  • Once a deployment model is chosen, you can’t change it in 6.0. U1 will allow changes.

VMware Certificate Authority – Favorite feature.

  • VMCA removes a lot of the certificate complexity
  • No longer uses self signed certificates
  • Built into the PSC
  • VMCA should you use it? Yes.
  • See KB 2111219 or my vSphere 6.0 install guide here

Standard vs. Distributed Switches

  • Always use VDS if you are licensed for it
  • Many of the past issues with VDS are now no longer an issue


  • Policy based storage management
  • Not all vSphere hardware is supported. Carefully check HCLs.
  • Learning curve for operational procedures and recovery
  • May require new hardware purchase

SMP Fault Tolerance

  • Long awaited SMP support (up to 4 vCPU)
  • Basically a continuous vMotion that only stops when there’s a hardware failure
  • 10Gb NIC requirement
  • Max 4 FT VMs per host

Content Library

  • New to vSphere 6.0
  • Storage for templates, appliances, ISOs, scripts, etc.
  • Should you use it? Definitely


VMworld 2015: Certificates for Mere Mortals

Session INF4529

Note: Although not mentioned in this session, I have a SSL toolkit for vSphere 6.0 which makes the replacement process easier. Check out my vSphere 6.0 install guide here for all the details.

Certificate Lifecycle Management

  • VMCA: VMware certificate authority
  • VECS: VMware Endpoint Certificate store


  • Dual Operational modes: Root CA and Issuer CA
  • Root CA: Automated, can issue other certs, all solutions and endpoint certificates are created and trusted to this root cert
  • Issuer CA: Can replace all default root CA certificate created during installation. Basically subordinate CA to your enterprise CA.


  • Repository for certificates and private keys
  • Mandatory component
  • Key stores: machine SSL certs, trusted roots, CRLs, solution users, others (e.g. VVOLS).
  • Managed through veccs-CLI
  • Does not manage SSO certificates

vSphere 6.0 Certificate Types

  • ESXi certificates – autogenerated post-install. New modes in 6.0, one of which can use VMCA certs. Can renew in webclient.
  • Machine SSL certificates – Creates server-side SSL (HTTPS, LDAP, etc.). Each node has its own machine SSL certificate.
  • Solution User certificates – Machine, vpxd, vpxd-extension, vsphere-webclient. Encapsulates one or more vCenter services.
  • Single-sign-on: Not stored in VECS. Stored in filesystem. STS certificate. Renew/update via GUI, not filesystem replacement.

Certificate Replacement Options

  • VMCA as root. Easiest deployment option.
  • VMCA as Enterprise CA subordinate – VMCA will issue certs on behalf of your enterprise CA
  • Custom CA – Only use custom certs all around. Not recommended except for Gov’t/Financial.
  • Hybrid – User facing certs replace, then let VMCA manage solution user and ESXi certs.

VMware vSphere 6.0 Certificate Manager

  • Available on both Windows and VCSA
  • Menu driven (GUI in 6.0 U1)

VMCA as Subordinate

  • RSA with 2048 bits
  • x.509v3
  • SHA256, 384 or 512
  • No wildcards in SubjectAltName
  • Cannot create subsidiary CAs of VMCA
  • Sync time for all nodes

Session videos, slides and scripts: http://vmware.com/go/inf4529


VMworld 2015: vCenter Server HA

Session INF4945

Why is vCenter HA important?

  • Primary administrative console
  • Critical component in end-to-end cloud provisioning
  • Foundation for VDI
  • Backup and DR solutions rely on vCenter
  • vCenter target availability is 99.99% from VMware’s design perspective (5 min a month)


Make every layer of the vCenter stack HA

  • vCenter DB
  • Host
  • SAN
  • Network
  • DC power and cooling

Reduce dependencies to improve nines

  • In moving from 5.1 and 5.5 to 6.0 you see a consolidation of vCenter services into VMs (e.g. just PSC and vCenter in 6.0)
  • vCenter 5.5 U3 supports SQL AAGs
  • vCenter 6.0 U1 supports SQL AAGs

Hardware/Host Failure protection: vSphere HA

  • Test tested solution
  • Protects against hardware failures
  • Some downtime for failover
  • Easy to setup and manage
  • DRS rules can be leveraged
  • High restart priority for vCenter components

Hardware/host failure protection: vSphere FT

  • Continuous availability with zero downtime and data loss
  • vCenter tested with FT for 4 vCPUs or less (only the ‘tiny’ and ‘small’ deployments fit)
  • About 20% overhead
  • Downtime during guest OS patching

Application failure protection: Watchdog

  • Watchdog monitors and protects vCenter applications
  • Automatically enabled on install on VCSA and Windows
  • On failure watchdog attempts to restart processes, if restart fails then VM is rebooted
  • Separate watchdog per vCenter server component

Application failure protection: Windows Server Failover clustering

  • Provides protection against OS level and application downtime
  • Provides protection for database
  • Some downtime during failure
  • Reduces downtime during OS patching
  • Tested with vCenter 5.5 and 6.0

Platform Services Controller HA

  • Two models: Embedded PSC or external PSC
  • PSC high availability in 6.0 requires a third party load balancer (removed in future vSphere versions)
  • Multiple PSC nodes in same site

vCenter Backup

  • Backup both embedded PSC and external PSC configurations
  • Recover from failures to vCenter node, PSC node or both
  • When vCenter node restored, it connects to PSC and reconciles the differences
  • When PSC node restored, it replicates from the other nodes
  • Uses VADP
  • Out-of-the box integration with VMware VDP

Tech Preview (vSphere 6.1?): Native HA

  • Native active-passive HA
  • Uses witness
  • No third party technology needed
  • Recover in minutes (target is 15 minutes), not hours
  • Protects against hardware, host and application failures
  • No shared storage required
  • 1-click automated HA setup
  • Fully integrated into the product
  • Out of box for the VCSA

VMworld: What’s new in vSphere 6.0?

Session INF5060

VMware’s architecture for IT: Any device, any application, one cloud

EVO SDDC is about deploying a new datacenter in less than two hours

Compute strategic imperatives: Cloud native infrastructure, hybrid cloud, virtualization leadership

vSphere 6.0 – Largest vSphere release ever

  • Shipped March 2015
  • 2x to 4x scale increase across the platform
  • Enhanced 2D/3D support with NVIDIA Grid
  • Rapid provisioning with 10x faster instance clone
  • Content library
  • More responsive web client
  • 64 hosts in a cluster
  • Long distance vMotion and cross-vCenter vMotion, SMP-FT
  • VMware integrated OpenStack
  • Extended containers support – CoreOS

Key stats:

  • 30% of customers are running 6.0
  • 100K downloads since GA

vSphere 6.0 U1

  • vCSA – easier to install and upgrade
  • Web client – VUM support
  • Faster maintenance mode -4x to 7x improvement
  • Certificate authority – CLI to UI
  • Live refresh in web client
  • VCSA performance increased by 20%
  • vSphere APIs for IO Filtering – 3rd party plug-ins

How does VMware enable containers?

  • Photon OS
  • Instant Clone
  • APIs for orchestration

VMware photon platform – future

  • Photon machine
  • Photon OS
  • Support for 100K containers or more
  • Available in 2016

Unified hybrid cloud allows best of both worlds

Cross-cloud vMotion and content sync with vCloud Air


VMworld 2015: DRS Advancements in vSphere 6.0

Session INF5306

DRS is the #1 scheduler in the datacenter today

92% of clusters have DRS enabled. 79% are in fully automated mode. 87% have affinity and anti-affinity rules.

43% of clusters have resource pools enabled and use them

99.8% of cluster use maintenance mode

Bottom line: DRS is popular

DRS collects innumerable stats every 20 seconds for its calculations

  • CPU Reserved
  • Memory reserved
  • CPU active, run and peak
  • memory overhead, growth-rate
  • Active, consumed and idle memory
  • Shared memory pages, balloon, swapped, etc.
  • VM happiness is the most important metric (if demands/entitlementws are always met, then VM is ‘happy’)

Constraints for initial placement and load balancing

  • Constraints are a big part of decision making
  • HA admission control policies
  • Affinity and anti-affinity rules
  • # concurrent vMotions
  • Time to complete vMotion
  • Datastore connectivity
  • vCPU to pCPU ratio
  • Reservations, limits and share settings
  • Agent VMs
  • Special VMs (SMP-FT, vFlash, etc.)

Cost Benefit and minGoodness

  • Cost-benefit analysis – VM happiness is evaluated against the cost of a migration
  • Cost considerations: per vMotion of 30% CPU core for 1Gb and 100% of a core for 10Gb; Memory consumption of ‘shadow VM’ at the destination host
  • Benefit considerations: Positive performance benefit to VMs at the source host, overall workload distribution has to be much better
  • Each analysis results in a rating from -2 to +2
  • MinGoodness (migration threshold slider) is -2 to +2. User can set this.


  • VM happiness is the #1 influence
  • Influenced by real time stats, constraints and cost/benefit analysis
  • A small imbalance should not be a concern
  • Default setting of DRS aggressiveness is best

New Features in vSphere 6.0

  • Network-aware DRS – ability to specify bandwidth reservation for important VMs
  • Initial placement based on VM bandwidth reservation
  • Automatic remediation in response to reservation violations due to pNIC saturation, pNIC failure
  • Tight integration with the vMotion team and will do a unified recommendation for cross-vCenter vMotion
  • Runs a combined DRS and SDRS algorithm to generate a tuple (host, DS)
  • CPU, memory, and network reservations are considered as part of admission control
  • All the constraints are respected as part of the placement
  • VM-to-VM affinity and anti-affinity rules are carried over during cross-cluster and cross-vCenter migration
  • Initial placement enforces the affinity and anti-affinity constraints
  • Improved overhead computation – greatly improves the consolidation during power-on

Cluster Scale and Performance Improvements

  • Increased cluster capacity to 64 hosts and 8K VMs
  • DRS and HA extensively tested at maximum scale for VCSA and Windows
  • Up to 66% performance increase in vCenter (power on, DRS calcs, etc.)
  • VM power-on latency has reduced by 25%
  • vMotion operation is 60% faster
  • Faster host maintenance mode

Extensive Algorithm Usage

  • DRS is the lynchpin of the SDDC vision
  • vSphere HA
  • VUM
  • vCloud Director
  • vCloud Air
  • Fault Tolerance
  • ESX Agent Manager

Best Practices

  • Tip #1: Full storage connectivity
  • Tip #2: Power management settings – Set BIOS to OS control and vSphere to balanced.
  • Tip #3: Threshold setting – Default of 3 works great.
  • Tip #4: Automation level – Fully automated is best choice
  • Tip #5: Beware of resource pool priority inversion. Make sure that cramming more VMs won’t dilute the shares.
  • Tip #6: Avoid setting CPU-affinity

Future Directions

Proactive HA

  • Proactive evacuation of VMs based on hardware health metrics
  • Partnering with hardware vendors to integrate and certify
  • Moderately degraded mode and severely degraded modes
  • VI admin can configure the DRS action for each health state event
  • Host maintenance mode and host quarantine mode
  • VI admin can filter events

Network DRS v2

  • Take pNIC saturation into account
  • Tighter integration with NSX
  • Ensure mice and elephant flow doesn’t share same network path
  • Network layout topology – leverage topology for availability and performance optimizations

Proactive DRS

  • Tighter integration with VRops analytics engine
  • Periodic and seasonality demands incorporated into decision making

What-if Analysis

  • A sandbox tab in UI to run ‘what if’ analysis
  • VM availability assessment by simulating host failures
  • Cluster over commitment during maintenance window

Auto-scale of VMs

  • Horizontal and vertical scaling to maintain end-to-end SLA guarantees
  • Spin-up and spin-down VMs based on workload
  • Will first be offered as a service in vCloud air
  • Increase CPU and memory resources to meet performance goals
  • CPU/memory hot add is an additional option for DB tier

Hybrid DRS

  • Make vCloud-air a seamless extension of enterprise datacenter capacity through policy based scheduling



VMworld 2015: vSphere 6.1 Upgrade & Deployment Pt. 1

Session INF4944

Goal: Deliver and enhanced customer experience for deploying and upgrading vCenter environments.

vCenter server 6.0 platforms: Windows and VCSA support the same scale and performance

Enhanced Linked mode is brand new in 6.0 and supported on Windows or VCSA. Policies and tags are now supported in Linked Mode.

Deployment Models

  • PSC is no longer just SSO, but adds certificates and licensing
  • PSC supports data replication
  • Embedded deployment: PSC and vCenter running on single VM
  • External PSC: vCenter and PSCs on separate VMs
  • vCSA is the recommended deployment package

vCenter Server Install

  • Both Windows and VCSA have similar simplified installs.
  • Supports GUI or scripted installs
  • Simple

vCenter Best Practices

  • Sizing
  • Windows OS and DB compatibility
  • Use FQDN
  • vCSA install target will support vCenter and ESXi in 6.0 U1
  • Time sync is important
  • DNS forward and reverse lookups
  • If using VDS use ephemeral port group
  • Ensure routing works

vCenter Server Upgrade

  • Multi-stage process: SSO/PSC, vCenter, ESXi, VMs, VMFS/VDS
  • Order is important KB2109760
  • Don’t forget about plug-ins, add-ons, VMFS, VDS, etc.
  • Approach upgrades with a holistic view of your infrastructure
  • vCSA upgrade is migration based and required temporary IP
  • Windows vCenter upgrade is in-place

Upgrade Paths

  • Windows Server – From 5.0 on up is supported. Prior to 4.0 you need to upgrade to 5.x.
  • vCSA upgrade from 5.1 later only

Upgrade best Practices

  • Sizing – 6.0 is larger.
  • Windows OS and DB compatibility
  • VCSA Oracle DB deprecation (use embedded DB)
  • Backup DB and VM prior to upgrade
  • Stick to recommended topologies
  • Time sync is very important
  • DB password issues: don’t use dash, question mark, underscore, left paren, equal, exclamation

Repointing from embedded deployment to external PSC – In 6.0 U1

  • First upgrade to 6.0 U1
  • Then deploy external PSC and replicate with embedded PSC
  • Repoint VC to the external PSC

vCSA Management UI (U1)

  • https://vcsa IP/:5480

PSC Management UI (U1)

  • https://PSC IP/psc


vSphere 6 Hardening Guide GA

During much of my career, I’ve been in the Government space and had to implement DISA STIGs for a variety of products including hypervisors. If you are a VMware customer and plan on using vSphere 6.0, you will be pleased to know that the vSphere 6.0 hardening guide is now GA. Some big changes were made in this version versus previous versions, so it should be more usable. You can find the full VMware blog post here.

I never saw this before, but VMware has a great landing page for security guides. From this page you can download a variety of guides and spreadsheets, very easily. That landing page is here.

What I’d really like to see from VMware is the majority of the security settings baked into the hypervisor with automated reporting. It can take weeks or months of STIG testing to get all of the settings right, run reports, etc. I hope that VMware will make hardening the hypervisor even easier, and take away much of the pain.

vSphere Install Pt. 16: User Solution Certificates

Now that we have vCenter installed, it’s time to update our User Solution certificates for the vCenter services. This is a fairly straight forward process, using the combination of the VMware Certificate Manager tool and my vCenter 6.0 Toolkit. The VMware Certificate manager tool will automatically create the private keys and CSRs for each user solution certificate. My toolkit will then take the CSRs and submit them to your enterprise CA and also create the chained PEM files the VMware toolkit needs to install the certificates. Then we flip back to the VMware tool to let it actually install the certificates. I decided against duplicating functionality between my Toolkit and the VMware tool, so there’s  little flipping back and forth.

If you are using the VMCA, then that’s even easier, as we can fully rely on the VMware tool to update the required certificates. I’ll go over all of the scenarios here.

Also take note that you need at least version 0.85 of my vCenter toolkit for this article to work properly. So download it, or a newer version, from the permalink below.

Blog Series

vSphere 6.0 Install Pt. 1: Introduction
vSphere 6.0 Install Pt. 2: Platform Services Controller
vSphere 6.0 Install Pt. 3: Certificate Management
vSphere 6.0 Install Pt. 4: vCenter Upgrade Best Practices
vSphere 6.0 Install Pt. 5: ESXi Upgrade Best Practices
vSphere 6.0 Install Pt. 6: Install Windows PSC
vSphere 6.0 Install Pt. 7: Config SQL DBs
vSphere 6.0 Install Pt. 8: Toolkit Configuration
vSphere 6.0 Install Pt. 9: SSL Templates
vSphere 6.0 Install Pt. 10: Install VCSA PSC
vSphere 6.0 Install Pt. 11: VMCA as Subordinate
vSphere 6.0 Install Pt. 12: PSC Machine Certificate
vSphere 6.0 Install Pt. 13: Directory Services Certificate
vSphere 6.0 Install Pt. 14: Windows vCenter Install
vSphere 6.0 Install Pt. 15: VCSA vCenter Install
vSphere 6.0 Install Pt. 16: User Solution Certificates

Permalink to this series: vexpert.me/Derek60
Permalink to my Toolkit script: vexpert.me/toolkit60

User Solution Certificates with VMCA

1. Open a command prompt and run the ‘certificate-manager’ tool from C:\Program Files\VMware\vCenter Server\vmcad. If you are using the VCSA, open a bash shell and go to the /usr/lib/vmware-vmca/bin directory.

2. Select Option 6, Replace Solution user certificates with VMCA certificates.


3. Enter your SSO password.

4. Enter the IP address of your external PSC. Confirm you want to replace the certificates using the VMCA. Wait a couple of minutes for the procedure to complete.


User Solution Certificates with Custom Certificates

1. Open a command prompt and run the ‘certificate-manager’ tool from C:\Program Files\VMware\vCenter Server\vmcad. If you are using the VCSA, open a bash shell and go to the /usr/lib/vmware-vmca/bin directory.

2. From the main menu select Option 5. Enter your SSO password and PSC IP address.

3. Select Option 1 from the sub menu, to generate CSRs and keys. Enter a directory path of C:\Certs. If you are using the VCSA, enter an appropriate local directory.


4. If you look in the C:\Certs directory you will see a bunch of files created. If you are using the VCSA, copy all of the created files down into C:\Certs.


5. Open a new PowerShell window and launch my vCenter 6.0 Toolkit. Select Option 5 from the main menu, “User Solution Certificate Menu”.

6. If you are using an Online Microsoft CA then select Option 1, Mint User Solution certificates with an online Microsoft CA. Wait a few seconds, and all of the CSRs will be submitted to your online CA and the certificates downloaded. If your CA requires certificate approval, go to your CA approve the certificates, then select Option 2 to resume the download.

7. If you look at the C:\Certs directory you will now see several subdirectories, one for each corresponding CSR. Skip to Step 10 if you are an online Microsoft CA user. 2015-04-25_13-05-578. If you need to manually submit all of the CSRs to your CA (offline Microsoft CA, or third-party CA), then save each minted certificate as a base-64 encoded non-chained file with the following names in the C:\Certs directory:


9. From my User Solution Certificate menu select option 3, which will create your PEM files and move your certificate files into their own directory. Only use this option if you manually downloaded your CRT files from your CA.

10. If you are using the VCSA, copy the new folders in C:\Certs up to the appliance. Also, upload the chain.cer file as well.

10. Back in the VMware Certificate Manager tool select Option 2, Import Custom certificates… Input all of the requested file names, using the “.cer” and “.key” filenames for the corresponding option. Note: Due to a bug, if you try and use the “chain.cer” file for the signing certificate, the operation may fail at 0% and rollback. So until they fix the bug, use the “root64.cer” file for the last response. 2015-05-02_17-22-32

11. Type Y to continue with the replacement. Wait until the process is completed.


Replacing the user solution certificates is not a difficult process, if you combine my Toolkit script with the VMware certificate manager. Even with the multiple CA VMware bug, there’s an easy workaround .

vSphere 6.0 Install Pt. 15: VCSA vCenter Install

If you are a VCSA (vCenter server appliance) convert, and wish to use the vCenter server appliance, this post is for you! It assumes you already have an external VCSA PSC setup, per Part 10. So this post will walk you through a similar deployment process, but this time install vCenter instead of a PSC. If you’ve deployed a Windows vCenter, then skip this post.

Blog Series

vSphere 6.0 Install Pt. 1: Introduction
vSphere 6.0 Install Pt. 2: Platform Services Controller
vSphere 6.0 Install Pt. 3: Certificate Management
vSphere 6.0 Install Pt. 4: vCenter Upgrade Best Practices
vSphere 6.0 Install Pt. 5: ESXi Upgrade Best Practices
vSphere 6.0 Install Pt. 6: Install Windows PSC
vSphere 6.0 Install Pt. 7: Config SQL DBs
vSphere 6.0 Install Pt. 8: Toolkit Configuration
vSphere 6.0 Install Pt. 9: SSL Templates
vSphere 6.0 Install Pt. 10: Install VCSA PSC
vSphere 6.0 Install Pt. 11: VMCA as Subordinate
vSphere 6.0 Install Pt. 12: PSC Machine Certificate
vSphere 6.0 Install Pt. 13: Directory Services Certificate
vSphere 6.0 Install Pt. 14: Windows vCenter Install
vSphere 6.0 Install Pt. 15: VCSA vCenter Install
vSphere 6.0 Install Pt. 16: User Solution Certificates

Permalink to this series: vexpert.me/Derek60
Permalink to my Toolkit script: vexpert.me/toolkit60

Deploy VCSA vCenter

1. Download the VCSA ISO (yes ISO, not OVA) and mount it on a Windows VM.

2. Open the root of the ISO and click on the vcsa-setup.html file.

3. Since I’m assuming a fresh install, click on Install.2015-03-29_19-42-354. Accept the license agreement and click Next.

5. Enter the FQDN or IP address of the ESXi server which you want the PSC deployed on. Enter the associated credentials. Click next and wait for the verification to complete. You may get a warning about an untrusted SSL certificate. Accept it.


6. On your DNS server configure A and PTR records for the vCenter’s address. This is critical!

7. Enter the FQDN of your appliance, and a complex password. If your password is not complex enough it will warn you and provide the complexity requirements.


8. Select “Install vCenter Server” on the next screen, since we will be using our external PSC.2015-04-24_19-39-009. Now enter the FQDN of your PSC, and the SSO password.

2015-04-24_19-40-4910. Up next is VCSA sizing. In my case I selected Tiny, since this is a small home lab.


11. Here you get to select your database. SQL is NOT an option, due to the lack of a supported ODBC binary for Linux. I’ll use the built-in Database.

2015-04-24_19-44-4112. Up next is configuring all the network settings. This is pretty self explanatory. Do take note of the time sync options. In a production environment I would suggest syncing to a trusted NTP source and not the ESXi host. Although you should have ESXi configured to sync with a trusted source as well. Not shown in the screenshot is the ability to enable SSH. Since I’m in a lab environment, I enabled SSH.


13. Review all of the settings to ensure they are 100% correct. Click Next, and sit back for the installation to complete.



Deploying vCenter in the form of a VCSA is easy peasy! I really like the VCSA for its ease of deployment, and self-contained nature. Clearly VMware has put a lot of development time into the VCSA, and it shows. Now that vCenter is installed, it’s time to replace more SSL certificates. That’s coming up next in Part 16.