Archives for February 2015

vSphere 6.0 Install Pt. 3: Certificate Management


As long as I can recall certificate management in vSphere has been difficult, and for many customers, something they completely ignore. I’m surprised how many customer designs (even those done by VCDXs) I’ve seen where they feel it’s too difficult to deploy vSphere certificates so they accept the risk of using the non-trusted VMware provided certificates. While I don’t think untrusted SSL certificates are the biggest security threat to an enterprise, I do feel that using trusted certificates is the right thing to do and an add extra layer of security. If you work in a highly regulated industry like finance, healthcare or Government you may be mandated to use fully trusted certificate. Most of my career has been in the Government sector, so using trusted certs was not even a question and just a basic security requirement.

Starting in vSphere 5.1, SSL complexity really shot up and was pretty ‘cocked up’ to put it politely. In vSphere 5.5 VMware did address some of the complexity with a command line tool to help replace certificates. That was still complex, so I wrote the widely used vCenter 5.5 toolkit, which made the whole process super easy. Feedback on that effort has been super positive, and kept me motivated to do the same for 6.0. Now with vSphere 6.0 my toolkit script has to do less because VMware has made it easier, but I still want to make it even easier for customers. Fortunately or unfortunately, depending on how you look at it, vSphere 6.0 has new certificate management options which at first look make SSL more complex than in the past. We’ll dig into each option in this article.

Blog Series

vSphere 6.0 Install Pt. 1: Introduction
vSphere 6.0 Install Pt. 2: Platform Services Controller
vSphere 6.0 Install Pt. 3: Certificate Management
vSphere 6.0 Install Pt. 4: vCenter Upgrade Best Practices
vSphere 6.0 Install Pt. 5: ESXi Upgrade Best Practices
vSphere 6.0 Install Pt. 6: Install Windows PSC
vSphere 6.0 Install Pt. 7: Config SQL DBs
vSphere 6.0 Install Pt. 8: Toolkit Configuration
vSphere 6.0 Install Pt. 9: SSL Templates
vSphere 6.0 Install Pt. 10: Install VCSA PSC
vSphere 6.0 Install Pt. 11: VMCA as Subordinate
vSphere 6.0 Install Pt. 12: PSC Machine Certificate
vSphere 6.0 Install Pt. 13: Directory Services Certificate
vSphere 6.0 Install Pt. 14: Windows vCenter Install

Permalink to this series:
Permalink to my Toolkit script:

Who cares about SSL?

Why should you go through the headaches of replacing all the VMware self-signed certificates? What’s the risk of using untrusted certificates? What can happen if the SSL connection is compromised?

Hypervisors are likely the underpinnings of your business critical apps and intellectual property. If your hypervisor is compromised then it’s just a few short commands to access your critical business data. Unless you like your infrastructure being p0wn3d, then you don’t want your VMware infrastructure compromised. If you don’t use trusted certificates, and just click through all the VI client SSL warnings (you have clicked Ignore and trust this certificate many times…haven’t you?) then you won’t know that a man-in-the-middle attack has taken place.

A man-in-the-middle attack is where a third party intercepts your “secure” communications and relays data between you, the attacker’s device, and the end host (like an ESXi server). This can be accomplished by ARP spoofing, or other means. If it’s second nature to ignore and click through all VI certificate warnings, you will have no idea your credentials have been intercepted….in clear text. No fancy brute force decryption required. Just sit back, grab a coke, and enjoy cleartext flowing across your screen. An interesting article on attacking VMware is here.

There are certainly many other ways to compromise your virtual infrastructure, like stealing the credentials of an administrator and gaining direct access to vCenter. Or using pass the hash, and gaining vCenter access that way. So ‘hacking’ SSL may not be the first choice for an attacker, but it is an attack vector you should consider and secure.

VMware Certificate Authority (VMCA)

This is a new an exciting component in vSphere 6.0 that will radically change how many will issue and deploy SSL certificates in their vSphere environment. SSL certificates are used extensively to secure communications in a vSphere environment. This ensures data confidentiality and integrity. Any attempt to modify data in transit is detected, such as man-in-the-middle attacks.

The VMCA is a built-in certificate authority, which is included in the Platform Services Controller (PSC) service. This is a full blown CA, and can (if you wish) automatically issue certificates to all vCenter 6.0 components and ESXi 6.0 hosts in your environment. The VMCA is mostly command line driven, and does not have a fancy GUI like your Microsoft CA has. But once configured, it’s pretty much a hands off operation. Do take note that VMCA in vSphere 6.0 does NOT support the use of CRLs nor does it have the concept of certificate revocation. If you suspect one certificate was compromised, first remove it then replace all certificates.

VMCA Intermediate Certificate Requirements

If you wish to use the VMCA as a subordinate CA to your existing enterprise CA, take note of the certificate requirements. The requirements are:

  • Private Key Algorithm: RSA with 2048 bits. No fancy elliptical curve support.
  • Recommended Signature Algorithms: SHA256, SHA384, or SHA512
  • NOT Recommended algorithms: MD2, MD5, SHA1
  • Key Usage: Root certificate extension set to true and cert sign must be in the list of requirements
  • Use PEM certificate format, with a header of —–BEGIN CERTIFICATE——
  • Does NOT support Wildcard certificates or more than one DNS name
  • Certificate must be X.509 v3

More about being a subordinate CA later in this article.

Certificate Deployment Options

VMware has come up with four primary certificate deployment options in vSphere 6.0. This is more than any previous release, where you basically had two (use VMware certs, or replace with trusted certs). You need to fully understand all four options, then pick for your environment which one best meets your business and security requirements. Depending on your industry, you may be severely limited in your choices. For example, if you are a U.S. Government agency you are stuck with option 3, using an external CA for all certificates and you won’t care about the new VMCA.


#1 VMCA Root CA

Option #1 is the simplest option, and probably the one a lot of organizations will go with. This is relying on the new VMCA to provision and manage certificates for vCenter and ESXi hosts. The VMCA is automatically created upon PSC installation, and requires no further configuration. However, for services accessed by a web browser (such as the web client) you will get an SSL warning unless you explicitly trust the VMCA root in your browser of choice. This is akin to the VMware signed certificate method in years past. Except in 6.0 there’s now a central CA managing the certificates and their lifecycle. If you do nothing, this is what you will get. Better than in prior vSphere releases, but still not fully trusted certificates. For better security see option #2.

#2 Subordinate VMCA

This is an entirely new option in vSphere 6.0, and wasn’t remotely possible in prior releases. Basically what happens in this mode is that the VMCA imports a root signing certificate from you trusted enterprise root CA. The VMCA then becomes an official subordinate CA to your enterprise root(s). All the certificates issued by the VMCA are trusted by your organization, even the web services exposed in browsers. As you deploy new vSphere components that are VMCA aware, they will get issued trusted SSL certificates. Since the VMCA now manages ESXi 6.0 host certificates, your ESXi hosts will also be issued trusted certificates without any manual intervention.

The BIG downside to this, is also the big upside. The VMware CA is now issuing fully trusted certificates, which may go against company policy. Or, if you are in a regulated environment like the US DoD, there’s no way in hell they will allow you to stand up a trusted subordinate CA. So I would say this option is good for environments that want more than VMware issued certificates, but aren’t so regulated that a VMware subordinate CA would be strictly prohibited. Call this a good compromise between security and simplicity. Thank you VMware! For even more security see option #3.


#3 External CA

Using an External CA is not new, and has always been in option in vSphere dating back many versions. It replaces all certificates in the environment by ones issued from the corporate trusted CAs. VMCA is bypassed, so this is a much more labor intensive process and much higher management complexity. This will be the only option in highly regulated environments, and will cause the most customer pain. All of the benefits of the new VMCA will be ignored, in favor of a higher security posture. This process is also totally different from that in vSphere 5.5, so get ready to learn yet another tedious procedure.


#4 Hybrid

A hybrid scenario features the usage of the VMCA in combination with an external CA. For example, you could use the automated VMCA certificates for all “internal” certificates and ESXi hosts and only replace externally facing certificates (such as web client) with those from an external CA. This adds complexity to the VMCA subordinate option (#2), but is less work than using an external CA for all certificates.  Personally, I don’t see this solution being used too much. I think the other three will be more popular, and the level of regulation and security consciousness will ultimately determine which route to take.

Certificate Types

In the vSphere 5.x era each service was issued its own unique SSL certificate. As you may recall, each certificate had to contain a unique “OU” field otherwise SSO would barf. This does not scale well, as VMware is constantly adding new services to vCenter. Even in vSphere 5.5, my toolkit had to generate 11 certificates for all the services. Whew!

In vSphere 6.0 we now have several types of certificates. As shown in the VMware graphic below, a lot of services are use these ‘common’ certificates. This reduces the total number of certificates needed in the environment.


The following table lists each of the certificate types used in vSphere 6.0, how they are provisioned, and where they are stored.


ESXi Certificate: As has been the case for many years, this certificate is used by the ESXi host to encrypt nearly all communications. Nothing new here.

Machine SSL Certificate: Each node (embedded installation, management node, or PSC) has its own machine SSL certificate. All services running on this node use this certificate for end point encryption. The vCenter service (vpxd), VMware directory service (vmdir) also use these certificates.

Solution User Certificates: These certificates are used for authentication to the vCenter SSO service. Once the certificate is presented to SSO, SSO will issue a SAML token. The service, such as vpxd, can then use this token to authenticate to other services. Baseline solution user certificates include vpxd, vpxd-extensions, and vSphere-webclient.

VMware End Point Certificate Store (VECS)

The VMware End Point Certificate store (VECS) is a local repository for certificates and private keys. VECS is a mandatory component, and will be used even if you don’t sign your certificates with the VMCA. Remember that ESXi certificates are stored on the ESXi host and not in the VECS. The VECS includes a number of keystores including machine SSL certificates, trusted roots, CRLs, solution users (machine, vpxd, vpx-extension, vSphere-webclient) and other keystores such as those for vVols.



Securing your virtual infrastructure is important. There are many attack vectors, and attacking SSL may not be the highest risk. But with vSphere 6.0 and my Toolkit script, replacing SSL certificates is easier than it used to be. So strongly consider taking the time to understand the new deployment methods, assess your business requirements, then take steps to secure your environment. It’s of little use to secure your SSL connections if you give the ESXi root and vCenter admin passwords to everyone.

Next up in the series will be vCenter upgrade and deployment best practices, in Part 4. You can check that out here.

VMTurbo Win a Home Lab

Disclaimer: VMTurbo is a sponsor of this blog.

Do you need a free home lab? Then register for the upcoming VMTurbo 5.1 webinar and get a chance to win a free home lab. Three lucky winners will be chosen at random so carve out an hour of your day and see what’s new with VMTurbo 5.1. Register for free here.


Nutanix NOS 1-Click Upgrade

One of the never ending tasks in IT is keeping up with software builds and firmware updates. These are usually a somewhat painful process, may require downtime, and can often get pushed to the back burner of IT life. At Nutanix we recognized that pain point, and starting in NOS 4.0 we’ve introduced one-click upgrades.

What does one click upgrade really mean? It means that NOS can automatically check for newer software versions of NOS, then all from our HTML5 Prism GUI perform a non-disruptive and automated rolling upgrade of our controller VMs. During this time no vMotions are required, no I/O interruption, no host maintenance modes, and no data relocation is required. It’s so easy, even a junior administrator can perform the upgrade during production hours. Of course you probably will do this during a scheduled maintenance window “just in case”, but theoretically you can do it anytime.

So in this blog post I’ll walk you through the process of a “dark site” one click upgrade. What is a dark site? Well it’s one without internet connectivity, such as a classified environment. Nutanix has a lot of dark site customers, so enabling easy upgrades for this user base was a priority. This process is easy, and if your Nutanix cluster has internet connectivity then it’s even simpler as NOS can automatically download new software updates in the background.

NOS Cluster Upgrade

1. First, login to the Nutanix support portal here. As you can see, the support portal has a similar look at feel to our popular Prism interface. Click on Downloads.


2. Since NOS 4.1.1 is brand new, it’s on the splash page. You will need to download two files, the NOS 4.1.1 tarball, plus the upgrade JSON metadata file. You should also review the release notes prior to doing the upgrade. Blind upgrades are not recommended.


3. After both files have downloaded, login to the Prism interface for the Nutanix cluster that you want to upgrade. Click on the gear icon in the upper right corner and select Upgrade Software. As a side note, from the Prism interface you can see all kinds of cluster stats such as hypervisor version, number of hosts and blocks, performance data, storage summary, and health status. All in glorious HTML5.

2015-02-09_10-24-00a4. On the upgrade software window you can now upload the tarball and the metadata file. It also shows what version the cluster is currently at, which is 4.0.1 in my case.


5. During the upload process it provides a progress indicator, and shows which version you are uploading. In my case I’m updating to 4.1.1.


6. Now that the software and been uploaded, you simply click on Upgrade, acknowledge you want to upgrade, then sit back and wait.


7. During the upgrade a preupgrade check is done, which finishes in a couple of minutes.


8. Once the precheck completes it will start doing the rolling software upgrade. You can expand the progress indicator and see the status of each node. You can even click on Nothing to do, and play Tetris while the upgrade is going on. Yes, Nutanix can also keep you entertained.


9. Here you can see the cluster upgrade is now completed. You are also able to drill into upgrade subtasks, so you know exactly where the upgrade process is at.


10. After the upgrade completes you will now see the new NOS version. And if you look carefully, you will see even MORE 1-click upgrades added. This covers hypervisor upgrades, firmware upgrades, and NCC (a health utility).



As you can see, doing software upgrades for the Nutanix NOS just takes a few clicks. There is ZERO down time, no vMotions needed, no I/O interruption, no host maintenance mode, and no fuss. Starting with NOS 4.1.1 there is now also 1-click upgrades for hypervisors, firmware and NCC. Uncompromisingly simple.

vSphere 6.0 Install Pt. 6: Install Windows PSC

Now that we’ve gotten some background and best practices behind us, now it’s time to start the actual software installation. As previously mentioned, in all but the smallest environments it’s recommended to have a dedicated Platform Services Controller (PSC), rather than an embedded one. So first up here will be provisioning a new Windows Server 2012 R2 VM which will be dedicated to the PSC role. No other vCenter roles or services will be installed on here. Then further along in this series we will provision a second VM, which will run all of the vCenter services. So let’s roll up our sleeves and get started with installing the PSC.

Blog Series

vSphere 6.0 Install Pt. 1: Introduction
vSphere 6.0 Install Pt. 2: Platform Services Controller
vSphere 6.0 Install Pt. 3: Certificate Management
vSphere 6.0 Install Pt. 4: vCenter Upgrade Best Practices
vSphere 6.0 Install Pt. 5: ESXi Upgrade Best Practices
vSphere 6.0 Install Pt. 6: Install Windows PSC
vSphere 6.0 Install Pt. 7: Config SQL DBs
vSphere 6.0 Install Pt. 8: Toolkit Configuration
vSphere 6.0 Install Pt. 9: SSL Templates
vSphere 6.0 Install Pt. 10: Install VCSA PSC
vSphere 6.0 Install Pt. 11: VMCA as Subordinate
vSphere 6.0 Install Pt. 12: PSC Machine Certificate
vSphere 6.0 Install Pt. 13: Directory Services Certificate
vSphere 6.0 Install Pt. 14: Windows vCenter Install

Permalink to this series:
Permalink to my Toolkit script:

Platform Services Controller (PSC) Installation

1. From your favorite template provision a new Windows VM. It needs at least 2 vCPUs and I’d give it 4-6GB of RAM. I would recommend Windows Server 2012 R2. Join it to the domain, and patch it. No need to install Flash, as the PSC has no web or Flash based interface. The PSC install is also very small, so I wouldn’t even attach a second drive. Just install it to the C drive.

2. Mount the vCenter 6.0 ISO and start the Autorun. Click on vCenter Server for Windows then click on Install.


2. Click Next on the build screen.


3. Accept the license agreement.


4. On the Select Deployment Type screen, choose Platform Services Controller. Click Next.


5. At this point the installer should auto-populate the system name with the FQDN of the VM you are installing the PSC on. Accept this value and click Next. As soon as you click next you may get a IPv6 warning. Ignore this warning if you aren’t using IPv6 (and who is?).


6. This is the most important configuration screen. First up, now in vSphere 6.0 you can change the SSO domain. Per VMware do *NOT* make this the same domain as your Active Directory. This will create a show-stopper problem. You CAN, however, use a subdomain of your corporate domain. For example, sso.contoso.local. Personally, I would keep it simple and stick with the default.

Now you will need to configure a password for the administrator@vsphere.local account. This password needs to be at least 8 characters, one lower case character, one numeric character and one special character. And the length must NOT exceed 20 characters. Only use visible ASCII characters, meaning you can’t use a space. You are also not allowed to use the single quote (‘) either. Other special characters may cause problems (remembering the SSO 5.1 and 5.5 days) so be sure to test out your “complex” passwords.

Finally, you can change the site name. I would recommend you do change the name, and have it reference your physical location. While it appears that information is not currently used by vSphere, VMware hinted down the road the ‘site’ name may play a more important role.

If you have a current vSphere deployment that is using SSO, you can join that domain here. In my case I’m doing a fresh install, so I created a new domain.


7. On the Configure Ports screen I would leave these all at their default values.


8. Here you can change the installation directory if you wish. The install is so small that I don’t see the need to install on a D drive, for example.


9. On the final screen it shows a summary of the options you’ve selected. Verify everything is OK and click Install. Wait a few minutes for the installation to complete.



As you can see, the PSC installer is very straight forward. Unlike SSO in vSphere 5.1 that needed an external database, the PSC is fully self-contained. Nice! Now that we have the PSC up and running, it is time to install the Windows vCenter server.

vSphere 6.0 Install Pt. 2: Platform Services Controller


In this second installment of the vSphere 6.0 installation series we cover the new service called the Platform Services Controller. This new service has consumed the SSO service, and added a number of new features. vCenter functionality is now divided into two primary roles, the management node and the PSC. The remainder of this article will cover this split, what the PSC is, and what it means to your vCenter topology.

Blog Series

vSphere 6.0 Install Pt. 1: Introduction
vSphere 6.0 Install Pt. 2: Platform Services Controller
vSphere 6.0 Install Pt. 3: Certificate Management
vSphere 6.0 Install Pt. 4: vCenter Upgrade Best Practices
vSphere 6.0 Install Pt. 5: ESXi Upgrade Best Practices
vSphere 6.0 Install Pt. 6: Install Windows PSC
vSphere 6.0 Install Pt. 7: Config SQL DBs
vSphere 6.0 Install Pt. 8: Toolkit Configuration
vSphere 6.0 Install Pt. 9: SSL Templates
vSphere 6.0 Install Pt. 10: Install VCSA PSC
vSphere 6.0 Install Pt. 11: VMCA as Subordinate
vSphere 6.0 Install Pt. 12: PSC Machine Certificate
vSphere 6.0 Install Pt. 13: Directory Services Certificate
vSphere 6.0 Install Pt. 14: Windows vCenter Install

Permalink to this series:
Permalink to my Toolkit script:

Prior to vSphere 6.0 most people thought of vCenter as a single component. Some larger complex deployments may have had multiple SSO servers, but you still think of SSO as fundamental to vCenter. Starting in vSphere 6.0 vCenter is being decoupled into two distinct parts. The first is known as the Platform Services Controller (PSC), and the remainder are all the vCenter services. You can see this in the VMware graphic below, showing the management node (vCenter services) and the PSC. Other VMware products like the vRealize suite can utilize the PSC services, such as authentication.


When you install vCenter now, it no longer prompts you for various components. It installs the full suite every time…whether you want everything or not. This full suite includes the following components:

  • vCenter Server
  • vSphere Web Client
  • Inventory Service
  • Profile-driven storage
  • Auto Deploy
  • Syslog Collector
  • ESXi Dump Collector
  • and more…

Separately installable IS vSphere Update Manager. You can combine the VCSA (vCenter appliance) and a Windows VUM server, so it makes sense they give you the option to separately install VUM. The only other separately installable component is the vSphere authentication proxy. Keep in mind you still need the Windows thick C# client to manage all the VUM features. VMware is working on a VUM replacement, but it’s TBD when that will be released. Personally I wouldn’t look for that anytime soon.

Platform Services Controller (PSC)

As we all remember with vSphere 5.1, VMware introduced a major new component called SSO, or single sign on. In the 5.1 days this caused a lot of headaches and pain, particularly around SSL certificates. The situation was improved in vSphere 5.5 with SSO 2.0, and VMware provided more guidance to customers for SSO topologies. Fast forward to vSphere 6.0, and now there’s a whole new component called Platform Services Controller (PSC). PSC is a shared service that supports vCenter server and vCenter server components. Various VMware products can use the PSC such as vCenter and vRealize Operations. Think of PSC as a foundational service which provides the following services:

  • License service (formerly held by vCenter)
  • Single Sign-on Service (Secure Token Service, identity management service, directory service)
  • Lookup Service
  • VMware Certificate Authority (VMCA)
  • VMware Endpoint Certificate Store (VECS)
  • Authentication framework daemon
  • Component Manager Service (CM)
  • HTTP reverse proxy

The big news here is the brand new VMware certificate authority (VMCA) and VMware Endpoint Certificate store (VECS). This will have a major impact on how you deploy and manage certificates in a vCenter environment. More to come on that in my next post.

Also keep in mind that you can mix and match PSC embedded installs along with external instances. All will share the same SSO domain, and will replicate other information like licensing. This lets you start out small, say with an embedded install, and expand in the future with a replicated external instance. Or perhaps you add a second datacenter down the road, with a local vCenter instance. There’s a limit of 8 PSCs per site.

Do keep in mind that if you start out with an embedded install (vCenter + PSC), there’s no supported method to “move” the PSC to a dedicated server or add more PSCs. So my advice is to start out by dedicating a VM to the PSC, and anther VM to all the other vCenter services. This provides maximum scalability, and future proofs your infrastructure for the foreseeable future.

High availability (HA) for the PSC service is via an external load balancer, such as a Citrix Netscaler or F5. Unfortunately built-in HA didn’t make it into this release, so we must rely on a third party solution. In the VMware graphic below you can see three vCenter instances, all pointing to a redundant and load balanced PSC deployment.



vCenter single sign-on allows vSphere components to communicate with each other using secure tokens rather than authenticating separately to each component. The SSO service is comprised of the security token service, administration server, VMware directory service (vmdir) and identity management service. The STS service uses SAML tokens for authentication. The administration server allows administrators with the proper privileges to manage the SSO service. VMdir is a multi-master, multi-tenant directory service that uses LDAP, much like Active Directory. However it does not use AD nor ADAM/ADLS. It uses port 389 and 11711. If there is more than one instance of PSC in your environment then one update in vmdir will be replicated to all other instances. Vmdir also stores some certificate information. The identity management service processes identity sources and STS authentication requests.

Starting with vSphere 6.0 SSO is either deployed via the embedded PSC (co-resident with your vCenter server) or as part of an external PSC deployment. Of great importance is the order of installation. If you are using an external PSC, then it must be installed prior to vCenter server. This make sense, as vCenter depends on a number of the services in the PSC. If you choose the embedded install then everything is installed in the right order.

A single external PSC deployment can support up to eight vCenter instances. Each new vCenter instance connects to the same PSC server. If you are a monster enterprise environment and have more than 8 vCenters then you can deploy multiple PSC instances.

When you deploy the SSO component you will need to configure a password for the administrator@vsphere.local account. This password needs to be at least 8 characters, one lower case character, one numeric character and one special character. And the length must NOT exceed 20 characters. Only use visible ASCII characters, meaning you can’t use a space. You are also not allowed to use the single quote (‘) either. Other special characters may cause problems (remembering the SSO 5.1 and 5.5 days) so be sure to test out your “complex” passwords.

By default users are locked out of the SSO service after five consecutive failed attempts in three minutes. Accounts are unlocked after five minutes. These policies can be changed, if desired. See the vSphere Security Guide for additional details.

Below is a graphic from VMware showing the progression of SSO to the PSC, from vSphere 5.1 to 6.0.



As you can see, the Platform Services controller is new to vSphere 6.0 and incorporates a number of new services. Medium to large deployments can get complex, with a mix of multiple vCenter servers and highly available PSCs. Carefully evaluate your business requirements for scalability and high availability, then chose a the simplest deployment model that best meets those requirements.

VVol Technical Overview

Another great session at PEX 2015 by Rawlinson Rivera.

Traditional storage architectures can’t keep up:

  • Specialized and costly HW – Not commodity, low utilization, overprovisioning
  • Device-centric silos – Static classes of service, rigid provisioning, lack of granular control
  • Complex processes – Lack of automation, time consuming processes, slow reaction to request

Hypervisor Enables App-Centric Storage Automation

  • Knows the needs of all apps in real time
  • Sits directly in the I/O path
  • Has global view of all underlying storage systems
  • Can configure storage dynamically
  • Hardware agnostic

Traditional Model – long provisioning cycles, managing LUNs, complex, frequent data migrations

App-Centric Automation – Dynamic delivery of storage services when needed. Fine control of data services at the VM level. Common management across heterogeneous devices.

vSphere Virtual Volumes

  • Virtualizes SAN and NAS devices
  • Virtual disks are natively represented on arrays
  • Enables VM granular storage operations using array-based data services
  • Storage policy-based management enables automated consumption at scale
  • Industry-wide initiative supported by major storage vendors
  • Included with vSphere

What is a VVol?

  • Five types of VVols (objects): Config, Data, MEM, SWAP, Other
  • NO filesystem needed (VMFS is history)
  • Virtual machine objects are stored natively on the array

Storage Container

  • Logical storage constructs for grouping of virtual volumes
  • Typically defined by storage administrators on the array in order to define storage capacity allocation and restrictions
  • Capacity is based on physical storage capacity

Differences between Storage Container and LUNs

  • SC Size based on array capacity
  • Max number of SCs depends on the array
  • Size of SC can be extended
  • LUNs need a filesystem, fixed size mandates more number of LUNs

Protocol End Points

  • Access points that enables communications between ESXi hosts and storage array systems
  • Scope of Protocol End Points: iSCSI, NFS v3, FC, FCoE
  • A protocol endpoint can support any one of the protocols at a given time
  • End points receive SCSI or NFS reads, write commands
  • Storage container – For large number of VMs metadata and data files

Management Plane

  • VASA Provider (VP) – Developed by storage array vendors
  • Single VASA provider can manage multiple arrays
  • VASA provider can be implemented within the array or as a virtual appliance
  • Out of band communications
  • ESX and vCenter server connect to VASA provider

Storage Capabilities and VM Storage Policies

  • Are array based features and data services specifications that capture storage requirements that can be satisified by a storage array’s advertised capabilities.
  • Storage array capabilities define what the array can offer to storage containers as opposed to what the VM requires
  • VM storage policies is a component of the storage policy based management framework (SPBM)
  • Published capability – Array based features and data services. Advertised to ESX through VASA APIs
  • Managed on a per vDisk basis
  • Has a concept of compliance to ensure service level is being met

Operations Scenarios

  • Can offload: VM provisioning, machine deletes, full clones, linked clones, snapshots
  • Snapshots: Managed by ESX OR managed by the array

Note: SRM will NOT support vVols in this release. You will need to wait for the next release for this support.

Q&A: Future VVols will allow storage pool to span physical arrays.

What’s new in vSphere 6.0

Straight from PEX 2015 in San Francisco, here’s a recap of the February 2 announcement for what’s new in vSphere 6.0

Platform Enhancements:

  • 128 vCPUs per VM
  • 4TB of RAM per VM
  • 64 hosts per cluster
  • 12TB of system RAM
  • 480 vCPUs per host
  • Hot-add RAM is now vNUMA aware
  • WDDM 1.1 GDI acceleration features
  • xHCI controller for USB 3 and OSX
  • Serial and parallel port enhancements (they can now be removed)
  • Account lockout (after 10 attempts, for 2 minutes) applies to SSH and Web SDK. DCUI is still available.
  • Can change default password complexity via API call or advanced setting
  • Improved auditing: All actions now list the actual username doing the action vs. just vpxuser
  • Support for SQL 2012 AlwaysOn Availability groups within MSCS
  • IPV6 Support with MSCS
  • vMotion support for MSCS nodes (with pRDMs)
  • MSCS supports PVSCSI controllers
  • New Support for Intel GPUs – vmklinux driver
  • Expanded NVidia Support

vCenter 6.0 Features

  • Parity with Windows and Appliance scalability
  • New Platform Services Controller – Embedded or Centralized model
  • Linked Mode – Feature Parity with Windows and Appliance. Supports policies and tags
  • New Certificate Lifecycle Management for vCenter and ESXi
  • VMCA -VMware Certificate Authority – Provisions certificates
  • VECS – Stores certificates
  • Certificate options – VMCA default, VMCA Enterprise, Custom
  • Cross  vSwitch vMotion – vSS to vSS, vSS to vDS, vDS to vDS. Requires L2 network connectivity. Still transparent to Guest.
  • Cross vCenter vMotion – Simultaneously change Compute, storage, network and vCenter. Targeted for local, metro, intra-continental. Tested u pto 150ms latency.
  • Long Distance vMotion – Tested up to 150ms. Supports vVols but not required. Needs 250 Mbps per vMotion. VM UUID not changed. MAC address is also preserved. Shares, limits, and reservations are also maintained.
  • Content library – Simple content management for VM templates, vApps, ISO images and scripts
  • vSphere C# client still here – Added support for HW v10 and v11 read-only
  • vSphere web client – Improved login time (13x), right click 4x faster, charts appear faster
  • Get anywhere in the web client in one click
  • Brought back recent tasks in the web client
  • NIOC – Reserve bandwidth to guarantee service levels. Applied at vNIC level.
  • Multiple TCP/IP stacks – vMotion network will cross L3 boundaries
  • vMotion can now use its own TCP/IP stack


  • vSphere Virtual Volumes – Virtualizes SAN and NAS devices. No more LUNs on block devices.
  • VVols enables finer control with VM level storage operations using array-based operations
  • VVols Supports block and NFS protocols
  • VVols is included with vSphere at all licensing levels
  • NFS 4.1 with Kerberos

vSphere 6.0 Fault Tolerance

  • Multi-vCPU support – 4 vCPUs
  • No longer require EZT disks – Can use any disk format
  • No support for vVOls
  • Up to 8 vCPUs protected per host (mix and match VMs)
  • Greatly increased FT host compatibility
  • Requires a 10Gb network – Segmented is strongly recommended
  • Heavily modified version of xvMotion
  • Each VM has its own vmx config file, vmdsk files. Can store second VM on another array
  • Supports backup snapshosts (only), no user snapshots

vSphere Replication

  • End-to-end network compression
  • Network traffic isolation
  • Linux file system quiescing
  • Faster full sync
  • Same 15 minute RPO

vSphere 6.0 Data Protection

  • No more advanced edition – All features available in base version
  • Included with vSphere essentials and higher
  • Supports up to 800 VMs per vCenter
  • For ROBOs up to 20 VDP appliances per vCenter
  • Replicate backup data between VDP appliances and EMC Avamar
  • EMC Data domain support with DD boost
  • Automated backup verification

vSphere 6.0 Install Pt. 1: Introduction

At VMworld 2014 VMware revealed bits and pieces of what’s new in vSphere 6.0. A lot of information was still under NDA and not disclosed, so attendees didn’t get the full picture of the new virtualization platform. But now that it’s announced, all can now be revealed. Unlike the last several years where the release has been in the fall of each year, this release cycle has been extended. Hopefully some extra QA was involved, so there aren’t so many issues with the GA release.

I’ve created a shortened permalink that you can use for quick reference: for this series. Feel free to use however you like…PowerPoint slides, email, etc. If you find this series helpful, please spread the word. This will be very similar to my vSphere 5.5 series, where we walk through some of what’s new, installation process, SSL certificate replacement, and other processes. The articles will be released slowly over the next couple of months.

Series Agenda

Like my vSphere 5.5 series it will cover at least the following topics:

  • Upgrade or fresh install?
  • Deep dive on the Platform Services Controller (PSC)
  • vCenter upgrade best practices and tips
  • ESXi upgrade best practices and tips
  • Right sizing your Windows vCenter VM
  • VMware Certificate Authority (VMCA)
  • Creating vCenter SSL certificates
  • Using a SQL 2012 AlwaysOn Failover Cluster for the vCenter database
  • Installing the full vCenter stack of software on Windows
  • Configuring VUM
  • ESXi host SSL certificate replacement
  • Deploying the vCenter Server Appliance (VCSA)

While I have two entire blog posts dedicated to upgrade best practices and tips, the step-by-step instructions will assume a fresh install. This is the VMware recommended approach, but doesn’t work for everyone. Upgrade how-to’s are complex, IMHO, since customer configurations will wildly vary. This is particularly true with SSO and the many deployment options, coupled with little VMware best practices around SSO.

Blog Series

vSphere 6.0 Install Pt. 1: Introduction
vSphere 6.0 Install Pt. 2: Platform Services Controller
vSphere 6.0 Install Pt. 3: Certificate Management
vSphere 6.0 Install Pt. 4: vCenter Upgrade Best Practices
vSphere 6.0 Install Pt. 5: ESXi Upgrade Best Practices
vSphere 6.0 Install Pt. 6: Install Windows PSC
vSphere 6.0 Install Pt. 7: Config SQL DBs
vSphere 6.0 Install Pt. 8: Toolkit Configuration
vSphere 6.0 Install Pt. 9: SSL Templates
vSphere 6.0 Install Pt. 10: Install VCSA PSC
vSphere 6.0 Install Pt. 11: VMCA as Subordinate
vSphere 6.0 Install Pt. 12: PSC Machine Certificate
vSphere 6.0 Install Pt. 13: Directory Services Certificate
vSphere 6.0 Install Pt. 14: Windows vCenter Install
vSphere 6.0 Install Pt. 15: VCSA vCenter Install
vSphere 6.0 Install Pt. 16: User Solution Certificates

Permalink to this series:
Permalink to my Toolkit script:

Database Support

VMware now officially supports SQL 2012 AlwaysOn failover clusters (using shared storage) for the vCenter database. It does NOT support AlwaysOn Availability groups or database mirroring. To that end I recently wrote a soup to nuts guide (12 parts) on how to install a SQL 2012 Failover Cluster on Windows Server 2012. If that’s something you want to do, you can dive head first into that while waiting on me to post the next vCenter installation installments. Many of you may not be clustering experts, so it should be enough to get you all the way up, with a ton of best practices incorporated. Here’s a quick reference chart for all of the SQL 2012/2014 HA/DR options.

9-29-2013 5-44-04 PM

Derek’s Toolkit Script

This year I’ve updated my PowerShell Toolkit script that I cover in-depth in a future post, which takes most of the pain away in creating your certificate requests and making the files the VMware certificate automation tool needs. As I go through the series it will also do tasks like creating your ODBC connectors. The script will be updated on a regular basis. A screenshot from my 6.0 tool will be available in the coming weeks, as it gets developed when VMware GAs their code.



You can also download the latest version at: (coming soon)


As I add new installments to the series this landing page will be updated with links to each part. Feedback is always welcome, so leave comments about your experiences. This can help other people that may have the same problem. One last comment…and I can’t stress this enough. You must, must, must read the vSphere 6.0 release notes.

You can find the next part in this series, the Platform Services Controller, here.