vSphere 6.0 Install Pt. 3: Certificate Management


As long as I can recall certificate management in vSphere has been difficult, and for many customers, something they completely ignore. I’m surprised how many customer designs (even those done by VCDXs) I’ve seen where they feel it’s too difficult to deploy vSphere certificates so they accept the risk of using the non-trusted VMware provided certificates. While I don’t think untrusted SSL certificates are the biggest security threat to an enterprise, I do feel that using trusted certificates is the right thing to do and an add extra layer of security. If you work in a highly regulated industry like finance, healthcare or Government you may be mandated to use fully trusted certificate. Most of my career has been in the Government sector, so using trusted certs was not even a question and just a basic security requirement.

Starting in vSphere 5.1, SSL complexity really shot up and was pretty ‘cocked up’ to put it politely. In vSphere 5.5 VMware did address some of the complexity with a command line tool to help replace certificates. That was still complex, so I wrote the widely used vCenter 5.5 toolkit, which made the whole process super easy. Feedback on that effort has been super positive, and kept me motivated to do the same for 6.0. Now with vSphere 6.0 my toolkit script has to do less because VMware has made it easier, but I still want to make it even easier for customers. Fortunately or unfortunately, depending on how you look at it, vSphere 6.0 has new certificate management options which at first look make SSL more complex than in the past. We’ll dig into each option in this article.

Blog Series

vSphere 6.0 Install Pt. 1: Introduction
vSphere 6.0 Install Pt. 2: Platform Services Controller
vSphere 6.0 Install Pt. 3: Certificate Management
vSphere 6.0 Install Pt. 4: vCenter Upgrade Best Practices (coming soon)
vSphere 6.0 Install Pt. 5: ESXi Upgrade Best Practices (coming soon)

Permalink to this series: vexpert.me/Derek60
Permalink to the Toolkit script: vexpert.me/toolkit60 (coming soon)

Who cares about SSL?

Why should you go through the headaches of replacing all the VMware self-signed certificates? What’s the risk of using untrusted certificates? What can happen if the SSL connection is compromised?

Hypervisors are likely the underpinnings of your business critical apps and intellectual property. If your hypervisor is compromised then it’s just a few short commands to access your critical business data. Unless you like your infrastructure being p0wn3d, then you don’t want your VMware infrastructure compromised. If you don’t use trusted certificates, and just click through all the VI client SSL warnings (you have clicked Ignore and trust this certificate many times…haven’t you?) then you won’t know that a man-in-the-middle attack has taken place.

A man-in-the-middle attack is where a third party intercepts your “secure” communications and relays data between you, the attacker’s device, and the end host (like an ESXi server). This can be accomplished by ARP spoofing, or other means. If it’s second nature to ignore and click through all VI certificate warnings, you will have no idea your credentials have been intercepted….in clear text. No fancy brute force decryption required. Just sit back, grab a coke, and enjoy cleartext flowing across your screen. An interesting article on attacking VMware is here.

There are certainly many other ways to compromise your virtual infrastructure, like stealing the credentials of an administrator and gaining direct access to vCenter. Or using pass the hash, and gaining vCenter access that way. So ‘hacking’ SSL may not be the first choice for an attacker, but it is an attack vector you should consider and secure.

VMware Certificate Authority (VMCA)

This is a new an exciting component in vSphere 6.0 that will radically change how many will issue and deploy SSL certificates in their vSphere environment. SSL certificates are used extensively to secure communications in a vSphere environment. This ensures data confidentiality and integrity. Any attempt to modify data in transit is detected, such as man-in-the-middle attacks.

The VMCA is a built-in certificate authority, which is included in the Platform Services Controller (PSC) service. This is a full blown CA, and can (if you wish) automatically issue certificates to all vCenter 6.0 components and ESXi 6.0 hosts in your environment. The VMCA is mostly command line driven, and does not have a fancy GUI like your Microsoft CA has. But once configured, it’s pretty much a hands off operation. Do take note that VMCA in vSphere 6.0 does NOT support the use of CRLs nor does it have the concept of certificate revocation. If you suspect one certificate was compromised, first remove it then replace all certificates.

VMCA Intermediate Certificate Requirements

If you wish to use the VMCA as a subordinate CA to your existing enterprise CA, take note of the certificate requirements. The requirements are:

  • Private Key Algorithm: RSA with 2048 bits. No fancy elliptical curve support.
  • Recommended Signature Algorithms: SHA256, SHA384, or SHA512
  • NOT Recommended algorithms: MD2, MD5, SHA1
  • Key Usage: Root certificate extension set to true and cert sign must be in the list of requirements
  • Use PEM certificate format, with a header of —–BEGIN CERTIFICATE——
  • Does NOT support Wildcard certificates or more than one DNS name
  • Certificate must be X.509 v3

More about being a subordinate CA later in this article.

Certificate Deployment Options

VMware has come up with four primary certificate deployment options in vSphere 6.0. This is more than any previous release, where you basically had two (use VMware certs, or replace with trusted certs). You need to fully understand all four options, then pick for your environment which one best meets your business and security requirements. Depending on your industry, you may be severely limited in your choices. For example, if you are a U.S. Government agency you are stuck with option 3, using an external CA for all certificates and you won’t care about the new VMCA.


#1 VMCA Root CA

Option #1 is the simplest option, and probably the one a lot of organizations will go with. This is relying on the new VMCA to provision and manage certificates for vCenter and ESXi hosts. The VMCA is automatically created upon PSC installation, and requires no further configuration. However, for services accessed by a web browser (such as the web client) you will get an SSL warning unless you explicitly trust the VMCA root in your browser of choice. This is akin to the VMware signed certificate method in years past. Except in 6.0 there’s now a central CA managing the certificates and their lifecycle. If you do nothing, this is what you will get. Better than in prior vSphere releases, but still not fully trusted certificates. For better security see option #2.

#2 Subordinate VMCA

This is an entirely new option in vSphere 6.0, and wasn’t remotely possible in prior releases. Basically what happens in this mode is that the VMCA imports a root signing certificate from you trusted enterprise root CA. The VMCA then becomes an official subordinate CA to your enterprise root(s). All the certificates issued by the VMCA are trusted by your organization, even the web services exposed in browsers. As you deploy new vSphere components that are VMCA aware, they will get issued trusted SSL certificates. Since the VMCA now manages ESXi 6.0 host certificates, your ESXi hosts will also be issued trusted certificates without any manual intervention.

The BIG downside to this, is also the big upside. The VMware CA is now issuing fully trusted certificates, which may go against company policy. Or, if you are in a regulated environment like the US DoD, there’s no way in hell they will allow you to stand up a trusted subordinate CA. So I would say this option is good for environments that want more than VMware issued certificates, but aren’t so regulated that a VMware subordinate CA would be strictly prohibited. Call this a good compromise between security and simplicity. Thank you VMware! For even more security see option #3.


#3 External CA

Using an External CA is not new, and has always been in option in vSphere dating back many versions. It replaces all certificates in the environment by ones issued from the corporate trusted CAs. VMCA is bypassed, so this is a much more labor intensive process and much higher management complexity. This will be the only option in highly regulated environments, and will cause the most customer pain. All of the benefits of the new VMCA will be ignored, in favor of a higher security posture. This process is also totally different from that in vSphere 5.5, so get ready to learn yet another tedious procedure.


#4 Hybrid

A hybrid scenario features the usage of the VMCA in combination with an external CA. For example, you could use the automated VMCA certificates for all “internal” certificates and ESXi hosts and only replace externally facing certificates (such as web client) with those from an external CA. This adds complexity to the VMCA subordinate option (#2), but is less work than using an external CA for all certificates.  Personally, I don’t see this solution being used too much. I think the other three will be more popular, and the level of regulation and security consciousness will ultimately determine which route to take.

Certificate Types

In the vSphere 5.x era each service was issued its own unique SSL certificate. As you may recall, each certificate had to contain a unique “OU” field otherwise SSO would barf. This does not scale well, as VMware is constantly adding new services to vCenter. Even in vSphere 5.5, my toolkit had to generate 11 certificates for all the services. Whew!

In vSphere 6.0 we now have several types of certificates. As shown in the VMware graphic below, a lot of services are use these ‘common’ certificates. This reduces the total number of certificates needed in the environment.


The following table lists each of the certificate types used in vSphere 6.0, how they are provisioned, and where they are stored.


ESXi Certificate: As has been the case for many years, this certificate is used by the ESXi host to encrypt nearly all communications. Nothing new here.

Machine SSL Certificate: Each node (embedded installation, management node, or PSC) has its own machine SSL certificate. All services running on this node use this certificate for end point encryption. The vCenter service (vpxd), VMware directory service (vmdir) also use these certificates.

Solution User Certificates: These certificates are used for authentication to the vCenter SSO service. Once the certificate is presented to SSO, SSO will issue a SAML token. The service, such as vpxd, can then use this token to authenticate to other services. Baseline solution user certificates include vpxd, vpxd-extensions, and vSphere-webclient.

VMware End Point Certificate Store (VECS)

The VMware End Point Certificate store (VECS) is a local repository for certificates and private keys. VECS is a mandatory component, and will be used even if you don’t sign your certificates with the VMCA. Remember that ESXi certificates are stored on the ESXi host and not in the VECS. The VECS includes a number of keystores including machine SSL certificates, trusted roots, CRLs, solution users (machine, vpxd, vpx-extension, vSphere-webclient) and other keystores such as those for vVols.



Securing your virtual infrastructure is important. There are many attack vectors, and attacking SSL may not be the highest risk. But with vSphere 6.0 and my Toolkit script, replacing SSL certificates is easier than it used to be. So strongly consider taking the time to understand the new deployment methods, assess your business requirements, then take steps to secure your environment. It’s of little use to secure your SSL connections if you give the ESXi root and vCenter admin passwords to everyone.

Next up in the series will be vCenter upgrade and deployment best practices, in Part 4. Stay tuned for that installment.

VMTurbo Win a Home Lab

Disclaimer: VMTurbo is a sponsor of this blog.

Do you need a free home lab? Then register for the upcoming VMTurbo 5.1 webinar and get a chance to win a free home lab. Three lucky winners will be chosen at random so carve out an hour of your day and see what’s new with VMTurbo 5.1. Register for free here.


Nutanix NOS 1-Click Upgrade

One of the never ending tasks in IT is keeping up with software builds and firmware updates. These are usually a somewhat painful process, may require downtime, and can often get pushed to the back burner of IT life. At Nutanix we recognized that pain point, and starting in NOS 4.0 we’ve introduced one-click upgrades.

What does one click upgrade really mean? It means that NOS can automatically check for newer software versions of NOS, then all from our HTML5 Prism GUI perform a non-disruptive and automated rolling upgrade of our controller VMs. During this time no vMotions are required, no I/O interruption, no host maintenance modes, and no data relocation is required. It’s so easy, even a junior administrator can perform the upgrade during production hours. Of course you probably will do this during a scheduled maintenance window “just in case”, but theoretically you can do it anytime.

So in this blog post I’ll walk you through the process of a “dark site” one click upgrade. What is a dark site? Well it’s one without internet connectivity, such as a classified environment. Nutanix has a lot of dark site customers, so enabling easy upgrades for this user base was a priority. This process is easy, and if your Nutanix cluster has internet connectivity then it’s even simpler as NOS can automatically download new software updates in the background.

NOS Cluster Upgrade

1. First, login to the Nutanix support portal here. As you can see, the support portal has a similar look at feel to our popular Prism interface. Click on Downloads.


2. Since NOS 4.1.1 is brand new, it’s on the splash page. You will need to download two files, the NOS 4.1.1 tarball, plus the upgrade JSON metadata file. You should also review the release notes prior to doing the upgrade. Blind upgrades are not recommended.


3. After both files have downloaded, login to the Prism interface for the Nutanix cluster that you want to upgrade. Click on the gear icon in the upper right corner and select Upgrade Software. As a side note, from the Prism interface you can see all kinds of cluster stats such as hypervisor version, number of hosts and blocks, performance data, storage summary, and health status. All in glorious HTML5.

2015-02-09_10-24-00a4. On the upgrade software window you can now upload the tarball and the metadata file. It also shows what version the cluster is currently at, which is 4.0.1 in my case.


5. During the upload process it provides a progress indicator, and shows which version you are uploading. In my case I’m updating to 4.1.1.


6. Now that the software and been uploaded, you simply click on Upgrade, acknowledge you want to upgrade, then sit back and wait.


7. During the upgrade a preupgrade check is done, which finishes in a couple of minutes.


8. Once the precheck completes it will start doing the rolling software upgrade. You can expand the progress indicator and see the status of each node. You can even click on Nothing to do, and play Tetris while the upgrade is going on. Yes, Nutanix can also keep you entertained.


9. Here you can see the cluster upgrade is now completed. You are also able to drill into upgrade subtasks, so you know exactly where the upgrade process is at.


10. After the upgrade completes you will now see the new NOS version. And if you look carefully, you will see even MORE 1-click upgrades added. This covers hypervisor upgrades, firmware upgrades, and NCC (a health utility).



As you can see, doing software upgrades for the Nutanix NOS just takes a few clicks. There is ZERO down time, no vMotions needed, no I/O interruption, no host maintenance mode, and no fuss. Starting with NOS 4.1.1 there is now also 1-click upgrades for hypervisors, firmware and NCC. Uncompromisingly simple.

vSphere 6.0 Install Pt. 2: Platform Services Controller


In this second installment of the vSphere 6.0 installation series we cover the new service called the Platform Services Controller. This new service has consumed the SSO service, and added a number of new features. vCenter functionality is now divided into two primary roles, the management node and the PSC. The remainder of this article will cover this split, what the PSC is, and what it means to your vCenter topology.

Blog Series

vSphere 6.0 Install Pt. 1: Introduction
vSphere 6.0 Install Pt. 2: Platform Services Controller
vSphere 6.0 Install Pt. 3: Certificate Management
vSphere 6.0 Install Pt. 4: vCenter Upgrade Best Practices (coming soon)
vSphere 6.0 Install Pt. 5: ESXi Upgrade Best Practices (coming soon)

Permalink to this series: vexpert.me/Derek60
Permalink to the Toolkit script: vexpert.me/toolkit60 (coming soon)

vCenter Separation

Prior to vSphere 6.0 most people thought of vCenter as a single component. Some larger complex deployments may have had multiple SSO servers, but you still think of SSO as fundamental to vCenter. Starting in vSphere 6.0 vCenter is being decoupled into two distinct parts. The first is known as the Platform Services Controller (PSC), and the remainder are all the vCenter services. You can see this in the VMware graphic below, showing the management node (vCenter services) and the PSC. Other VMware products like the vRealize suite can utilize the PSC services, such as authentication.


When you install vCenter now, it no longer prompts you for various components. It installs the full suite every time…whether you want everything or not. This full suite includes the following components:

  • vCenter Server
  • vSphere Web Client
  • Inventory Service
  • Profile-driven storage
  • Auto Deploy
  • Syslog Collector
  • ESXi Dump Collector
  • and more…

Separately installable IS vSphere Update Manager. You can combine the VCSA (vCenter appliance) and a Windows VUM server, so it makes sense they give you the option to separately install VUM. The only other separately installable component is the vSphere authentication proxy. Keep in mind you still need the Windows thick C# client to manage all the VUM features. VMware is working on a VUM replacement, but it’s TBD when that will be released. Personally I wouldn’t look for that anytime soon.

Platform Services Controller (PSC)

As we all remember with vSphere 5.1, VMware introduced a major new component called SSO, or single sign on. In the 5.1 days this caused a lot of headaches and pain, particularly around SSL certificates. The situation was improved in vSphere 5.5 with SSO 2.0, and VMware provided more guidance to customers for SSO topologies. Fast forward to vSphere 6.0, and now there’s a whole new component called Platform Services Controller (PSC). PSC is a shared service that supports vCenter server and vCenter server components. Various VMware products can use the PSC such as vCenter and vRealize Operations. Think of PSC as a foundational service which provides the following services:

  • License service (formerly held by vCenter)
  • Single Sign-on Service (Secure Token Service, identity management service, directory service)
  • Lookup Service
  • VMware Certificate Authority (VMCA)
  • VMware Endpoint Certificate Store (VECS)
  • Authentication framework daemon
  • Component Manager Service (CM)
  • HTTP reverse proxy

The big news here is the brand new VMware certificate authority (VMCA) and VMware Endpoint Certificate store (VECS). This will have a major impact on how you deploy and manage certificates in a vCenter environment. More to come on that in my next post.

Also keep in mind that you can mix and match PSC embedded installs along with external instances. All will share the same SSO domain, and will replicate other information like licensing. This lets you start out small, say with an embedded install, and expand in the future with a replicated external instance. Or perhaps you add a second datacenter down the road, with a local vCenter instance. There’s a limit of 8 PSCs per site.

Do keep in mind that if you start out with an embedded install (vCenter + PSC), there’s no supported method to “move” the PSC to a dedicated server or add more PSCs. So my advice is to start out by dedicating a VM to the PSC, and anther VM to all the other vCenter services. This provides maximum scalability, and future proofs your infrastructure for the foreseeable future.

High availability (HA) for the PSC service is via an external load balancer, such as a Citrix Netscaler or F5. Unfortunately built-in HA didn’t make it into this release, so we must rely on a third party solution. In the VMware graphic below you can see three vCenter instances, all pointing to a redundant and load balanced PSC deployment.



vCenter single sign-on allows vSphere components to communicate with each other using secure tokens rather than authenticating separately to each component. The SSO service is comprised of the security token service, administration server, VMware directory service (vmdir) and identity management service. The STS service uses SAML tokens for authentication. The administration server allows administrators with the proper privileges to manage the SSO service. VMdir is a multi-master, multi-tenant directory service that uses LDAP, much like Active Directory. However it does not use AD nor ADAM/ADLS. It uses port 389 and 11711. If there is more than one instance of PSC in your environment then one update in vmdir will be replicated to all other instances. Vmdir also stores some certificate information. The identity management service processes identity sources and STS authentication requests.

Starting with vSphere 6.0 SSO is either deployed via the embedded PSC (co-resident with your vCenter server) or as part of an external PSC deployment. Of great importance is the order of installation. If you are using an external PSC, then it must be installed prior to vCenter server. This make sense, as vCenter depends on a number of the services in the PSC. If you choose the embedded install then everything is installed in the right order.

A single external PSC deployment can support up to eight vCenter instances. Each new vCenter instance connects to the same PSC server. If you are a monster enterprise environment and have more than 8 vCenters then you can deploy multiple PSC instances.

When you deploy the SSO component you will need to configure a password for the administrator@vsphere.local account. This password needs to be at least 8 characters, one lower case character, one numeric character and one special character. And the length must NOT exceed 20 characters. Only use visible ASCII characters, meaning you can’t use a space. You are also not allowed to use the single quote (‘) either. Other special characters may cause problems (remembering the SSO 5.1 and 5.5 days) so be sure to test out your “complex” passwords.

By default users are locked out of the SSO service after five consecutive failed attempts in three minutes. Accounts are unlocked after five minutes. These policies can be changed, if desired. See the vSphere Security Guide for additional details.

Below is a graphic from VMware showing the progression of SSO to the PSC, from vSphere 5.1 to 6.0.



As you can see, the Platform Services controller is new to vSphere 6.0 and incorporates a number of new services. Medium to large deployments can get complex, with a mix of multiple vCenter servers and highly available PSCs. Carefully evaluate your business requirements for scalability and high availability, then chose a the simplest deployment model that best meets those requirements.

VVol Technical Overview

Another great session at PEX 2015 by Rawlinson Rivera.

Traditional storage architectures can’t keep up:

  • Specialized and costly HW – Not commodity, low utilization, overprovisioning
  • Device-centric silos – Static classes of service, rigid provisioning, lack of granular control
  • Complex processes – Lack of automation, time consuming processes, slow reaction to request

Hypervisor Enables App-Centric Storage Automation

  • Knows the needs of all apps in real time
  • Sits directly in the I/O path
  • Has global view of all underlying storage systems
  • Can configure storage dynamically
  • Hardware agnostic

Traditional Model – long provisioning cycles, managing LUNs, complex, frequent data migrations

App-Centric Automation – Dynamic delivery of storage services when needed. Fine control of data services at the VM level. Common management across heterogeneous devices.

vSphere Virtual Volumes

  • Virtualizes SAN and NAS devices
  • Virtual disks are natively represented on arrays
  • Enables VM granular storage operations using array-based data services
  • Storage policy-based management enables automated consumption at scale
  • Industry-wide initiative supported by major storage vendors
  • Included with vSphere

What is a VVol?

  • Five types of VVols (objects): Config, Data, MEM, SWAP, Other
  • NO filesystem needed (VMFS is history)
  • Virtual machine objects are stored natively on the array

Storage Container

  • Logical storage constructs for grouping of virtual volumes
  • Typically defined by storage administrators on the array in order to define storage capacity allocation and restrictions
  • Capacity is based on physical storage capacity

Differences between Storage Container and LUNs

  • SC Size based on array capacity
  • Max number of SCs depends on the array
  • Size of SC can be extended
  • LUNs need a filesystem, fixed size mandates more number of LUNs

Protocol End Points

  • Access points that enables communications between ESXi hosts and storage array systems
  • Scope of Protocol End Points: iSCSI, NFS v3, FC, FCoE
  • A protocol endpoint can support any one of the protocols at a given time
  • End points receive SCSI or NFS reads, write commands
  • Storage container – For large number of VMs metadata and data files

Management Plane

  • VASA Provider (VP) – Developed by storage array vendors
  • Single VASA provider can manage multiple arrays
  • VASA provider can be implemented within the array or as a virtual appliance
  • Out of band communications
  • ESX and vCenter server connect to VASA provider

Storage Capabilities and VM Storage Policies

  • Are array based features and data services specifications that capture storage requirements that can be satisified by a storage array’s advertised capabilities.
  • Storage array capabilities define what the array can offer to storage containers as opposed to what the VM requires
  • VM storage policies is a component of the storage policy based management framework (SPBM)
  • Published capability – Array based features and data services. Advertised to ESX through VASA APIs
  • Managed on a per vDisk basis
  • Has a concept of compliance to ensure service level is being met

Operations Scenarios

  • Can offload: VM provisioning, machine deletes, full clones, linked clones, snapshots
  • Snapshots: Managed by ESX OR managed by the array

Note: SRM will NOT support vVols in this release. You will need to wait for the next release for this support.

Q&A: Future VVols will allow storage pool to span physical arrays.

What’s new in vSphere 6.0

Straight from PEX 2015 in San Francisco, here’s a recap of the February 2 announcement for what’s new in vSphere 6.0

Platform Enhancements:

  • 128 vCPUs per VM
  • 4TB of RAM per VM
  • 64 hosts per cluster
  • 12TB of system RAM
  • 480 vCPUs per host
  • Hot-add RAM is now vNUMA aware
  • WDDM 1.1 GDI acceleration features
  • xHCI controller for USB 3 and OSX
  • Serial and parallel port enhancements (they can now be removed)
  • Account lockout (after 10 attempts, for 2 minutes) applies to SSH and Web SDK. DCUI is still available.
  • Can change default password complexity via API call or advanced setting
  • Improved auditing: All actions now list the actual username doing the action vs. just vpxuser
  • Support for SQL 2012 AlwaysOn Availability groups within MSCS
  • IPV6 Support with MSCS
  • vMotion support for MSCS nodes (with pRDMs)
  • MSCS supports PVSCSI controllers
  • New Support for Intel GPUs – vmklinux driver
  • Expanded NVidia Support

vCenter 6.0 Features

  • Parity with Windows and Appliance scalability
  • New Platform Services Controller – Embedded or Centralized model
  • Linked Mode – Feature Parity with Windows and Appliance. Supports policies and tags
  • New Certificate Lifecycle Management for vCenter and ESXi
  • VMCA -VMware Certificate Authority – Provisions certificates
  • VECS – Stores certificates
  • Certificate options – VMCA default, VMCA Enterprise, Custom
  • Cross  vSwitch vMotion – vSS to vSS, vSS to vDS, vDS to vDS. Requires L2 network connectivity. Still transparent to Guest.
  • Cross vCenter vMotion – Simultaneously change Compute, storage, network and vCenter. Targeted for local, metro, intra-continental. Tested u pto 150ms latency.
  • Long Distance vMotion – Tested up to 150ms. Supports vVols but not required. Needs 250 Mbps per vMotion. VM UUID not changed. MAC address is also preserved. Shares, limits, and reservations are also maintained.
  • Content library – Simple content management for VM templates, vApps, ISO images and scripts
  • vSphere C# client still here – Added support for HW v10 and v11 read-only
  • vSphere web client – Improved login time (13x), right click 4x faster, charts appear faster
  • Get anywhere in the web client in one click
  • Brought back recent tasks in the web client
  • NIOC – Reserve bandwidth to guarantee service levels. Applied at vNIC level.
  • Multiple TCP/IP stacks – vMotion network will cross L3 boundaries
  • vMotion can now use its own TCP/IP stack


  • vSphere Virtual Volumes – Virtualizes SAN and NAS devices. No more LUNs on block devices.
  • VVols enables finer control with VM level storage operations using array-based operations
  • VVols Supports block and NFS protocols
  • VVols is included with vSphere at all licensing levels
  • NFS 4.1 with Kerberos

vSphere 6.0 Fault Tolerance

  • Multi-vCPU support – 4 vCPUs
  • No longer require EZT disks – Can use any disk format
  • No support for vVOls
  • Up to 8 vCPUs protected per host (mix and match VMs)
  • Greatly increased FT host compatibility
  • Requires a 10Gb network – Segmented is strongly recommended
  • Heavily modified version of xvMotion
  • Each VM has its own vmx config file, vmdsk files. Can store second VM on another array
  • Supports backup snapshosts (only), no user snapshots

vSphere Replication

  • End-to-end network compression
  • Network traffic isolation
  • Linux file system quiescing
  • Faster full sync
  • Same 15 minute RPO

vSphere 6.0 Data Protection

  • No more advanced edition – All features available in base version
  • Included with vSphere essentials and higher
  • Supports up to 800 VMs per vCenter
  • For ROBOs up to 20 VDP appliances per vCenter
  • Replicate backup data between VDP appliances and EMC Avamar
  • EMC Data domain support with DD boost
  • Automated backup verification

vSphere 6.0 Install Pt. 1: Introduction

At VMworld 2014 VMware revealed bits and pieces of what’s new in vSphere 6.0. A lot of information was still under NDA and not disclosed, so attendees didn’t get the full picture of the new virtualization platform. But now that it’s announced, all can now be revealed. Unlike the last several years where the release has been in the fall of each year, this release cycle has been extended. Hopefully some extra QA was involved, so there aren’t so many issues with the GA release.

I’ve created a shortened permalink that you can use for quick reference: vexpert.me/Derek60 for this series. Feel free to use however you like…PowerPoint slides, email, etc. If you find this series helpful, please spread the word. This will be very similar to my vSphere 5.5 series, where we walk through some of what’s new, installation process, SSL certificate replacement, and other processes. The articles will be released slowly over the next couple of months.

Series Agenda

Like my vSphere 5.5 series it will cover at least the following topics:

  • Upgrade or fresh install?
  • Deep dive on the Platform Services Controller (PSC)
  • vCenter upgrade best practices and tips
  • ESXi upgrade best practices and tips
  • Right sizing your Windows vCenter VM
  • VMware Certificate Authority (VMCA)
  • Creating vCenter SSL certificates
  • Using a SQL 2012 AlwaysOn Failover Cluster for the vCenter database
  • Installing the full vCenter stack of software on Windows
  • Configuring VUM
  • ESXi host SSL certificate replacement
  • Deploying the vCenter Server Appliance (VCSA)

While I have two entire blog posts dedicated to upgrade best practices and tips, the step-by-step instructions will assume a fresh install. This is the VMware recommended approach, but doesn’t work for everyone. Upgrade how-to’s are complex, IMHO, since customer configurations will wildly vary. This is particularly true with SSO and the many deployment options, coupled with little VMware best practices around SSO.

Blog Series

vSphere 6.0 Install Pt. 1: Introduction
vSphere 6.0 Install Pt. 2: Platform Services Controller
vSphere 6.0 Install Pt. 3: Certificate Management
vSphere 6.0 Install Pt. 4: vCenter Upgrade Best Practices (coming soon)
vSphere 6.0 Install Pt. 5: ESXi Upgrade Best Practices (coming soon)
vSphere 6.0 Install Pt. 6: Install PSC (coming soon)

Permalink to this series: vexpert.me/Derek60
Permalink to the Toolkit script: vexpert.me/toolkit60 (coming soon)

Database Support

VMware now officially supports SQL 2012 AlwaysOn failover clusters (using shared storage) for the vCenter database. It does NOT support AlwaysOn Availability groups or database mirroring. To that end I recently wrote a soup to nuts guide (12 parts) on how to install a SQL 2012 Failover Cluster on Windows Server 2012. If that’s something you want to do, you can dive head first into that while waiting on me to post the next vCenter installation installments. Many of you may not be clustering experts, so it should be enough to get you all the way up, with a ton of best practices incorporated. Here’s a quick reference chart for all of the SQL 2012/2014 HA/DR options.

9-29-2013 5-44-04 PM

Derek’s Toolkit Script

This year I’ve updated my PowerShell Toolkit script that I cover in-depth in a future post, which takes most of the pain away in creating your certificate requests and making the files the VMware certificate automation tool needs. As I go through the series it will also do tasks like creating your ODBC connectors. The script will be updated on a regular basis. A screenshot from my 6.0 tool will be available in the coming weeks, as it gets developed when VMware GAs their code.



You can also download the latest version at: vexpert.me/toolkit60 (coming soon)


As I add new installments to the series this landing page will be updated with links to each part. Feedback is always welcome, so leave comments about your experiences. This can help other people that may have the same problem. One last comment…and I can’t stress this enough. You must, must, must read the vSphere 6.0 release notes.

You can find the next part in this series, the Platform Services Controller, here.

PEX 2015 VCDX Mock Defense Room

For the past several VMware VCDX defenses Nutanix has been excited to sponsor a mock defense hotel room for the candidates to practice their art. I know when I was going through the defense program having the Nutanix sponsored room in San Francisco at PEX 2014 was a truly invaluable experience and greatly contributed to my first time success through the program. This year Thomas Brown from Varrow has been the study group ring leader, and has done a great job of organizing people.

Since PEX 2014 we’ve also sponsored rooms at VMworld 2014, and for the 2014 Palo Alto defenses in October. This just really shows how much Nutanix stands behind the VCDX program, and believes in the quality of people in turns out. In fact, Nutanix now has 11 VCDXs on staff and looks forward to adding even more world-class talent. Outside of VMware we have more VCDXs than several of our biggest competitors combined. Wow!

This coming weekend is crunch time for candidates across the globe. Candidates from several countries will be defending at PEX 2015 in San Francisco. Nutanix has committed to do something special for the candidates coming to PEX 2015!

Nutanix is sponsoring a room this weekend (Saturday and Sunday) at the Hotel Nikko, 222 Mason Street, for mock defenses, troubleshooting, design scenarios, and general preparation. I am working with Mark Brunstad, of VCDX program fame, to get the message out to the local VCDXs, so candidates can benefit from their experience as they prepare for the big day. I hope to have some surprises guests on Sunday for our candidates. I’ll be there the whole weekend helping the candidates prepare.

If you are a VCDX candidate who is flying in this weekend OR an existing VCDX in the Bay Area and you’d like to contribute, you should plan to spend some time at the VCDX mock defense room this weekend! Please Tweet me (@vDereks) to RSVP, and get the details about the schedule. I’m sure you will benefit greatly from the VCDXs’ guidance and experience, as they help prepare you for your defense next week.

Very best of luck to all candidates next week, and I hope to see your names in the VCDX Directory shortly. I’ll also be at PEX 2015 all week, so if you spot me please don’t be shy and come up and say HI! Always fun to meet my readers and get feedback about the blog.

Sizing Microsoft Lync Server 2013 on Nutanix

In this article I’ll be covering how to size and deploy Microsoft Lync Server 2013 on Nutanix. This article is a subset of a newly released whitepaper which covers SMB (1,000 users), medium (10,000 users), and large (26,664 users) size Lync deployment scenarios. You can download the full whitepaper for free here. In this blog post I’ll just cover the medium sizing scenario and you can check out the whitepaper for SMB and enterprise configurations.

The whitepaper also covers a lot of Hyper-V best practices for virtualizing Lync, so be sure to check it out. I would also like to add that Jason Sloan from Perficient (@Jason_D_Sloan), a Lync Certified Master, played a key role in the content. He has extensive experience with global Lync deployments, so tapping his expertise for this whitepaper was a big win.

When deploying Lync Server 2013, you want a resilient and scalable platform so that as your Lync environment grows, so can your infrastructure. Nutanix delivers a highly resilient converged compute and storage platform that brings the benefits of web-scale infrastructures to all organizations. Designed for supporting virtualized environments including VMware vSphere, Microsoft Hyper-V, and KVM, Nutanix is the ideal infrastructure for running all types of virtual workloads including real time technologies such as Microsoft Lync Server 2013.

Lync 2013 Overview

The core deployment for Lync Server 2013 consists of the “Front End” role which is required for a deployment, commonly referred to as the “Pool”.  The Front End service can run on either a Standard Edition Lync Server or the Enterprise Edition Lync Server.  The total amount of servers required starts with identifying which edition an organization requires based on business requirements.

In addition to the Lync Server 2013 Front End role, the complete Lync infrastructure is comprised of additional roles.  Not all roles however, require their own server as some can be collocated together.  Although Lync Server 2013 can be deployed a la carte the following roles are included in the Lync Server 2013 workloads:

  • Lync Front End Server – Standard or Enterprise edition
  • Lync Director Server – Client authentication
  • Lync Edge Server – Remote access
  • Microsoft Office Web Apps Server – PowerPoint Sharing and rendering
  • Lync Mediation Server – Audio codec transcoding
  • Lync Persistent Chat Server – Chat rooms
  • Microsoft SQL Server – Database servers

The total number of each of these servers can be extensive depending on the design requirements.  There are two key requirements that must be identified that drive the server design and ultimately determine the Lync Server 2013 Edition and total amount servers: user count and high availability.  It is important for an organization to understand each server and role they provide.  As the environments grow from SMB to large enterprise, the design requirements for the Lync Server architecture begins to scale up. This mirrors the scale up story of the Nutanix platform.

Standard Edition Server – Does not offer high availability, and hosts many services including instant messaging and presence, enterprise voice, archiving, persistent chat, and monitoring.

Enterprise Edition Pool – Comprised of three or more front end servers, they use an external SQL server and it is recommended the SQL server be highly available. Also, persistent chat servers must be deployed separately.

Lync on Nutanix

The Nutanix Virtual Compute Platform is a hyperconverged solution that combines storage and compute resources into a single, integrated appliance. It provides the manageability benefits of centralized storage while delivering high performance by keeping data local to server resources. IT managers can now deliver Unified Communications services with enterprise storage features without the cost and complexity of SAN and NAS systems. The scalability of the Nutanix cluster can accommodate Lync clusters of any size. A Lync deployment on Nutanix is flexible and scalable.

Nutanix delivers a highly available, easy to manage, fault tolerant platform for Lync Server 2013. The fault tolerance and self-healing of the underlying NDFS infrastructure allows critical UC services to be provided to end users without interruption. Nutanix’s ease of installation and setup cuts down UC deployment times by slashing the time needed to configure storage infrastructure. No more Fibre Channel zoning, LUN creation, LUN masking, etc.

Lync Full Capacity Sizing

With Lync Server 2013, server platform system requirements for physical hardware and virtualization are identical which simplifies planning.  These Microsoft requirements are based solely on full capacity physical deployments (200K users), which may not apply to your environment.  To view workload capacities, see the Microsoft TechNet article for more details. For each Front End, Back End and persistent chat servers Microsoft recommends 12 vCPUs (dual hex-core) at 2.26 GHz or higher with 32GB of RAM. For full capacity Edge, directors and mediation servers they recommend 8 vCPUs and 16GB of RAM. Finally, for Office web apps servers for 200K users, a 16 vCPU VM is recommended. Clearly, for smaller environments the requirements can be scaled back.

Microsoft has developed the Lync Capacity Calculator, which you can download from here. Translating calculator results into hypervisor vCPUs is not exactly straight forward. Unlike the Exchange calculator which is amazing, some estimation for Lync sizing is required. So next up is going through the sizing exercise for a “medium” deployment of 10,000 users.

Microsoft does have some virtualization guidelines for Lync, which you can find here. In there Microsoft clearly states that CPU oversubscription is strictly prohibited for Lync server media workloads. This limits the VM density, as we will see later on in this blog post. Lync is not the only UC application that mandates no CPU oversubscription, Cisco and Avaya do it as well.

10,000 Seat Sizing

In this example we are sizing the core Lync components for 10,000 users, N+1 availability, no persistent chat, and will enable archiving/monitoring. In the graphic below, you can see the assumptions we made using the Lync calculator spreadsheet. Your environment will be different, so make sure you adjust accordingly.

According to the calculator two FE servers are required, not taking into account N+1 availability. Taking that into account, and allowing for planned business growth over the next three years, I decided on a total of four FE servers.


Because this deployment is far under the full deployment model, 12 vCPUs for each FE server is likely overkill. So for this model we will use the Lync calculator estimate of 45% CPU load, and translate that into 6 vCPUs (12 vCPUs * .45). Clearly, depending on the usage of Lync in your environment this may be overkill, or inadequate. Proper POC testing is important, to help gauge sizing and ensure optimum Lync performance.

Now that we have calculated how many FE servers we need, the table below lists the remaining servers that we need to account for. Here we can see 2 edge servers, 3 SQL servers (one is SQL express for witness), and two Office app servers for N+1 availability. This deployment requires a total of 11 VMs.


Nutanix has a variety of platforms, each with a different number of CPU cores. Starting with our Haswell nodes, you can customize the CPUs to tailor the hardware to your needs. So the sizing example below is just one possible VM layout, and it would change depending on the model and processor that you select. Nutanix also has free sizing web app that partners can use to map out VMs to servers and get a full bill of materials.

In this example I selected a NX-3000 series node, with dual 10 CPUs. I also fully reserved 8 vCPUs for the Nutanix CVM, to ensure maximum storage performance. The remaining cores were distributed among the 11 VMs, taking into account the CPU oversubscription prohibition for certain VMs like the FE servers. We also included N+1 availability at the Nutanix block level, so any single node can fail and the hypervisor can restart the VMs on another node with adequate capacity. Click on the graphic below to enlarge it and see the proposed VM layout.



If you used the NX-3000 series it would require five nodes (four with VMs, one free for HA capacity), which can be deployed as two block using a total of 4 RU. Three additional slots are open for future expansion of the Lync environment, or other business critical apps. A minimum of 128GB of RAM is needed in each server. To round out the solution two 10GB top-of-rack switches are needed, for a grand total of 6RU. This is a very dense solution, which is also very power efficient. And yes, no additional rack space is needed for a shared storage solution due to the magic of our distributed file system, NDFS.


Microsoft Lync Server 2013 can be successfully deployed on the Nutanix platform, delivering high availability and true linear scalability. Traditional SAN/NAS complexity is eliminated while providing a highly resilient storage and compute infrastructure with a small datacenter footprint and full Microsoft support. If you need to dramatically scale your Lync infrastructure, you can easily just add more Nutanix nodes to linearly scale the performance.

Nutanix is the optimal compute and storage platform for critical real-time UC applications, allowing Unified Communications pool scaling as needed during a deployment, instead of building the entire infrastructure up front for only a small number of users. This flexibility makes Nutanix a perfect fit for on premise, private-cloud, or “UC as a Service” deployment models where scale over time is crucial.

By following the guidelines and best practices presented in the Nutanix whitepaper available here, you can successfully design and deploy a Microsoft Lync Server 2013 solution on Nutanix at any scale in a highly reliable manner using the minimum amount of rack space and power.

You can download the full whitepaper here.

Nutanix Releases NOS 4.1.1

I am very pleased to announce the latest release of the Nutanix Operating System, NOS 4.1.1, is generally available today. Download your copy today! This release offers rich enterprise capabilities to meet the needs of the most demanding applications and critical infrastructure requirements in the datacenter.

Here’s a summary of what’s new in this release.

  • Metro Availability: Nutanix Metro Availability provides continuous data availability for business-critical applications during planned maintenance or unplanned disasters that affect entire sites. Nutanix is the only hyper-converged infrastructure vendor to deliver continuous data protection across multiple datacenters. You can read more about Metro Availability here and here.
  • Cloud Connect: Nutanix Cloud Connect seamlessly integrates public cloud services, such as Amazon Web Services (AWS), with Nutanix-powered private cloud environments. It allows Nutanix customers to combine private and public cloud technology into their long-term infrastructure strategy without requiring third-party software or hard to maintain plug-ins. Read more about Cloud Connect here and here.
  • Encryption Support: Nutanix now provides strong data protection by encrypting user and application data to a level of FIPS 140-2 Level 2 compliance. Data at rest encryption is delivered through self-encrypting drives (SED) that are factory-installed in the Nutanix hyper-converged appliance. Find out more here.
  • Simplified Hypervisor Upgrade: Nutanix radically simplifies the process of upgrading the hypervisor in a Nutanix cluster with this new feature. This release will support upgrade of VMware ESXi and Microsoft Hyper-V, with KVM support coming soon. You can read about it here.
  • Security/STIG Enhancements: Nutanix has developed its own comprehensive security technical implementation guides (STIGs) to speed up the accreditation process for DIACAP/DIARMF, which is typically slow and manual. A process that typically takes nine to twelve months can now be accomplished in minutes. Over 500 embedded configuration changes were made to the system for compliance purposes. Major kudos to the security and development teams. Learn more here.
  • Prism Central Scalability: You can now manage up to 100 Nutanix clusters and 10,000 VMs. Prism Central now also supports Hyper-V.
  • Microsoft SCOM Pack: While not part of NOS, coinciding with the 4.1.1 release Nutanix has a Microsoft SCOM pack that monitors both Nutanix hardware and software. Great for Microsoft-centric shops.
  • XenDesktop Plug-In:  Again, while separate from NOS, Nutanix is unique in the industry by releasing a XenDesktop plug-in which enables you to configure per-VM SLAs. This is all configured with a few simple clicks from the XenDesktop Studio.

As part of the ongoing security hardening of NOS, 4.1.1 also addresses several attack vectors for NTP. Nutanix recommends upgrading to 4.1.1 to address these security issues. Certificate management in PRISM has also had a few improvements as well. For a deep dive on additional security information, check out this blog post by Nutanix’s Eric Hammersley. After seeing all the focus around securing our product automating the STIG process, you will agree with me that this is really industry leading.

For a “point” release of NOS, they’ve packed in A LOT of new features plus broader ecosystem support highlighted by the SCOM and XenDesktop add-ons. If you think these features are great, just wait to see what’s in our next point release.

© 2015 - Sitemap