vSphere 6.0 Install Pt. 7: Config SQL DBs

Now that we have the Windows PSC installed, it is time to prepare for installing vCenter. vCenter can support three database types: embedded vPostGres (supports up to 20 hosts and 2000 VMs), Microsoft SQL, and Oracle. SQL seems to be the most popular choice, so that’s what I’ll help you configure here. Now to be frank, nothing has really changed here in vSphere 6.0 for the SQL setup. But it does fully support SQL 2014, which is great. To find out if your particular SQL version is supported, you can check out the VMware Product Interoperability Matrixes. Be sure to select “Solution/Database interoperability” so you can view the supported Oracle and SQL databases.

Do take note that VMware fully supports “legacy” SQL failover clusters for the vCenter database. This is distinctly different from AlwayOn Availability Groups, which are currently NOT supported. Nag your VMware TAM about AlwaysOn Availability Group support. I wrote an entire blog series about setting up a SQL 2012 failover cluster, which you can check out here. It’s nearly the same steps for SQL 2014.

Blog Series

vSphere 6.0 Install Pt. 1: Introduction
vSphere 6.0 Install Pt. 2: Platform Services Controller
vSphere 6.0 Install Pt. 3: Certificate Management
vSphere 6.0 Install Pt. 4: vCenter Upgrade Best Practices
vSphere 6.0 Install Pt. 5: ESXi Upgrade Best Practices
vSphere 6.0 Install Pt. 6: Install Windows PSC
vSphere 6.0 Install Pt. 7: Config SQL DBs
vSphere 6.0 Install Pt. 8: Install vCenter (coming soon)

Permalink to this series: vexpert.me/Derek60
Permalink to my Toolkit script: vexpert.me/toolkit60

Create DB Files

VMware unfortunately does not provide a tool to automatically create your SQL database for you. So it’s up to you to size and configure the SQL databases prior to installing vCenter. You must also configure the proper DSN, and install the appropriate SQL client. Since VMware left these tasks up to the customer to do, I’ve included them in my vCenter toolkit to help expedite your installation process.

My vCenter toolkit script was very popular for 5.5 users, so I’ve updated the script for 6.0. Some of the SSL work isn’t quite done, so I’ll be releasing future updates to complete the SSL setup. But the current version does support the SQL DB creation, so let’s get to work.

1. Go to this permalink (here) and download my PowerShell script. To create the SQL databases you can run the script from anywhere. But for simplicity I’d suggest running it on what will be your vCenter server. Run the script, and you should see a menu similar to the screenshot below. Menus may change a little between releases.


2. Select the option to create the vCenter and VUM SQL database file. In the version of the script above, this is option #5. You will then be prompted for a series of responses, to properly size the database and log files for both vCenter and VUM. The screenshot below shows all of the prompts, and example configuration.


3. After the configuration file is written, copy it over to your SQL server and open it in SQL Studio. Modify the paths to the files as needed, then run the script. You should not have an errors, and two databases should now appear on your SQL server.


4. During my vCenter testing I found that even though the service account was DBO on the two databases, the vCenter installer complained. So for installation purposes, I gave the service account temp ‘sysadmin’ permissions at the SQL level, as shown below.


4. Back on the vCenter server run the Toolkit script again but this time we need to create the vCenter DSN. Select that option from the menu, option 6 in the version shown. Enter the required information, then download and install the SQL client as indicated.


5.  Just to make sure the DSN will work, launch “odbcad32.exe”, click on System DSN, then find your vCenter DSN. Click on Configure, click Next through the whole wizard, then click on  Test Data Source. Verify success.


6. If you are going to use VUM, then we need to repeat a similar process to create the DSN and test the connector. Using my Toolkit script, select option 7 (may change in future versions). Follow the prompts to create the DSN, then from the Windows start screen search for “data” and select the ODBC Data Sources (32-bit) option. Perform a DSN test and verify success.


We’ve now created both the vCenter and VUM databases in SQL, configured the ODBC connectors, and verified they work. The final step in getting vCenter up and running is actually installing vCenter using the databases we just created. That will be done in Part 8.

vSphere 6.0 Install Pt. 5: ESXi Upgrade Best Practices

In this installment of the vSphere 6.0 installation how-to series we cover upgrading ESXi hosts, VMs, and VMFS. You do need to understand ESXi/VM/VMFS upgrade best practices, recommended order, and gotchas. That’s what this post is for.

Blog Series

vSphere 6.0 Install Pt. 1: Introduction
vSphere 6.0 Install Pt. 2: Platform Services Controller
vSphere 6.0 Install Pt. 3: Certificate Management
vSphere 6.0 Install Pt. 4: vCenter Upgrade Best Practices
vSphere 6.0 Install Pt. 5: ESXi Upgrade Best Practices
vSphere 6.0 Install Pt. 6: Install Windows PSC
vSphere 6.0 Install Pt. 7: Config SQL DBs
vSphere 6.0 Install Pt. 8: Install vCenter (coming soon)

Permalink to this series: vexpert.me/Derek60
Permalink to my Toolkit script: vexpert.me/toolkit60

Upgrade Overview

First of all, planning is key. Even in a lab environment you want to settle on an upgrade strategy and understand the order. Order is huge!  If you are running the basic vSphere stack and no other products like SRM, vCAC, etc, the order looks like this:

1) vCenter
2) VUM
3) ESXi hosts
4) VMs

But don’t just plow ahead full steam ahead and forget about things like vCenter plug-ins, VDI dependencies, backup software support, SRM, and the plethora of other VMware and third-party products. Once you get vCenter and VUM updated it is fully supported to do rolling ESXi host upgrades. Now you have to think about VM hardware versions, VM tools, and VDS configuration. For a great summary of the upgrade order if your use other VMware products check out this KB.

Bottom line: Think through and plan the ENTIRE upgrade before starting any part of it, including vCenter. Many times third party products like backup software can lag significantly in vSphere support. So you may be waiting a while before you can upgrade.

VIBs and Image Profiles

Understanding how VMware packages ESXi is important to better understand the upgrade path. Vendors like HP, Cisco, Dell, and others provide customized ESXi ISO media. VMware packages software (drivers, agents, etc.) as VIBs (vSphere Installation Bundle). It’s similar to a zip file or tarball. VIBs can be bundled into an ISO file (such as the ESXi installer), or as a zip depot file.

An image profile defines the VIBs which will be installed. A “standard” profile contains VMware tools and a “no-tools” profile has no VMware tools (mostly for autodeploy). You can use the image builder CLI to create a custom profile.

9-29-2013 2-45-06 PM

If you want to view the VIBs on your ESXi host use the following command:

esxcli software vib list

There are many third party custom ISOs, bundles, and online depots. VMware recommends that you use a vendor customized ISO for your hardware. Some vendors are extremely timely, while others lag or nearly non-existent. I know from personal experience the HP install ISOs are heavily customized, while the Cisco ones only have a handful of drivers. Nutanix, for example, goes through a thorough testing process and bakes the ESXi install into our Foundation product. So no need to deal with custom ISOs or VIBs, as Foundation will deploy everything needed in an automated fashion.

Upgrading vSphere Hosts

The big question is: Should I upgrade the host or do a fresh install? Unlike vCenter where VMware recommends to do a fresh install, if possible, they recommend upgrading ESXi hosts. You can leverage features like HA, DRS, storage vMotion, and host profiles to quickly roll through hosts. Fresh installs should be limited to a small number of hosts, maybe for test purposes.

Before you upgrade check the VMware Compatibility Guide. Just because your host works with 5.0 or 5.5, does NOT mean it will work with 6.0. For example, historically HP BladeSystem has needed newer firmware to address gotchas with new ESXi builds. Don’t just blow this step off and think you have a tier-1 vendor so all is good. Likely specific firmware versions will be required/approved. Also, with 6.0 VMware removed some drivers like RealTek NICs. So if you do a fresh install you may suddenly be missing your NICs on a whitebox server. Good news is that if you are using a Mac Mini, many models come with out of the box support in 6.0!

If you are Nutanix customer, you can do a one-click hypervisor upgrade once we have qualified vSphere 6.0. This means you don’t need to use VUM, as the Nutanix PRISM GUI fully automates the upgrade process for you. Keep an eye out for the Nutanix announcement of vSphere 6.0 support. Our stated SLA is 90 days from GA.

Release Notes

The vSphere 6.0 release notes are quite lengthy. A number of support calls can be avoided by getting a heads up of issues. That’s why planning is so important. Get a cup of coffee or Five Hour Energy and read every issue in the release notes. It can pay dividends! The vSphere 6.0 release notes are here.

ESXi Upgrade Methods

  • ESXi Installer – Boot from ISO, choose upgrade
  • vSphere Update Manager – Import ISO, create upgrade baseline, remediate
  • ESXCLI – Stage ZIP, execute ‘esxcli system profile update’
  • Scripted Upgrades – Update/customize upgrade script
  • Nutanix – One click upgrades

The most popular and automated method is using VUM. It will orchestrate host maintenance modes, respect DRS directives, and generally make it seamless. You can directly upgrade from ESXi 5.x to 6.0.

Upgrading Clusters

Rolling upgrades within clusters are supported and highly recommended. Do take note that vCenter 6.0 does not support ESX/ESXi 4.x hosts, so upgrade them to 5.x prior to upgrading vCenter. Be careful with VM hardware compatibility in such situations though. ESXi 6.0 has wide latitude in virtual hardware support, so there’s no critical rush to upgrade to v10 or later hardware. Be sure to leverage HA, DRS, vMotion and storage vMotion to enable minimal/zero downtime upgrade. If you are using Enterprise Plus, leverage host profiles. It minimizes configuration drift and enables stricter configuration control.

Upgrading ESXi Hosts

The boot disk is not re-partitioned during the upgrade process. However, the contents ARE overwritten. If there’s a VMFS datastore on the boot volume it will be preserved. Same for scratch. Absolute minimum is 1GB of space on your boot volume. Here’s a good KB on boot volume sizing. I personally use 5-6GB LUNs for boot-from-SAN configurations. The figure below shows the basic partition layout of an ESXi installation. This scheme has not changed for 6.0.

9-29-2013 3-42-30 PM

VM Upgrades

VMware has changed their nomenclature in how they refer to VM hardware compatibility. Previously they always called out the specific “hardware” version such as 4, 7, 9, etc. But that didn’t obviously relate to a specific release, and people got confused. Plus they thought on my gosh I’m on HW 4 and they are up 9, I’m way out of date…upgrade!

Now VMware calls out the “Compatibility” level and ties that to a release of ESXi. For example, if under the covers the VM is HW v7 it will show ESX 4.x and later in the web GUI. Do NOT feel pressure to always upgrade the compatibility level. Sometimes you need to, such as provisioning a monster VM that wasn’t supported on older versions of ESXi. And sometimes there are performance gains to be had when using new vHW versions. My advice is to upgrade the vHW as part of your overall upgrade plan. Do realize that some new VM features can’t be edited in the Windows C# client, but basic properties like RAM and vCPUs can be modified. Click on the graphic to expand and see the various upgrade paths.



Upgrading tools and VM hardware is OPTIONAL, and VMware officially supports N-4 versions. VM hardware versions are NOT backwards compatible, though. You won’t be running HW version 11 VMs on anything but vSphere 6.0.

VMware tools are backward and forward compatible to a very large degree. Don’t freak out if your VM isn’t running the latest tools. VMware recommends you DO keep up (performance, security, compliance checking, etc.), but you have wide latitude. Backup software, HA, heartbeats and other functions rely on VMware tools so if they have problems, verify the tools version matches your host. VUM is excellent for verifying compliance. My recommendation is to keep your VMware tools up to date, specially after a big upgrade such as going to 6.0.



For those of you that heard starting with vSphere 5.1 that upgrading VMware tools would no longer require a reboot, that’s not actually the case. The low-down is that VMware did make changes to VMware tools to leverage Windows hot-swap of some kernel modules. However, some modules like keyboard/mouse/USB still require reboots. VMware includes those non-hot-plug modules in each tools update. So the net result is still needing to reboot when doing VMtools updates. Perhaps in the future they will change that behavior, but that’s not in 5.1 through 6.0.

VMFS Upgrades

VMFS upgrades are simple, and completely non-disruptive. You can upgrade a VMFS datastore from VMFS-3 to VMFS-5 with running VMs. However, while this may sound perfect, keep reading as the reality is more complicated. The table below shows the differences between the two filesystem versions. Now that VMFS-5 has been around for a while, I hope you don’t have too many VMFS-3 datastores around.

9-29-2013 4-02-44 PM

Ok so you are thinking, why is an upgrade not ideal? The problem is that an upgraded volume does NOT look the same under the covers from a freshly formatted VMFS-5 volume. The table below shows the differences. The most impacting can be the block size. In vSphere 4.x and earlier you had a choice of block sizes that ranged from 1MB to 8MB. If your array supports VAAI extensions the VMFS volumes must have the same block size if you are doing operations such as copying VMs. Otherwise the disk operations revert back to legacy mode and will run slower.

9-29-2013 4-05-15 PM

The VMware recommendation is to create a fresh VMFS datastore then storage vMotion your VMs into the datastore. After the datastore is evacuated re-format or decommission it. If you aren’t licensed for storage vMotion, then during your vCenter upgrade don’t input a product key. This gives you 60 days of the ‘enhanced’ license features.


VMFS will play less of a role in vSphere 6.0 and beyond with the advent of VVols. VVols does not use a filesystem, so there’s no VMFS to deal with. Once your storage array supports VVols and you migrate VMs to vVols you can forget about VMFS. I have no insider knowledge here, but I’d be surprised if VMware released any major new VMFS versions given the VVols future.

SSL Certificates

New to vSphere 6.0 are different SSL certificate options. They are:

  • VMware Certificate Authority mode – VMCA automatically provisions host certificates
  • Custom Certificate mode – Enabled you to use your own certificates
  • Thumbprint mode – Can be used to retain vSphere 5.5 certificates during upgrade

Which mode you use depends on your business requirements. VMCA mode is the easiest, as it automates ESXi certificate deployment. I would recommend this mode. You could use custom certificate mode and then use my vCenter 6.0 toolkit to replace the certificates, but I’d only recommend that if you can’t use the VMCA and need to use trusted certificates.

Smart Card Authentication

Also new to vSphere 6.0 is the ability to use smart card authentication to your ESXi host. They support US DoD CAC cards as well as traditional industry standard smart cards. See the vSphere 6.0 Security guide for additional details on how to configure your ESXi hosts to use smart cards. I will not be covering that in this series.


  • Understand the vSphere Upgrade Process
  • Understand how ESXi is packaged and distributed
  • Understand patches vs. updates vs. upgrades
  • Know the different upgrade methods
  • Stay current on VMware tools
  • Freshly format VMFS5 volumes; don’t upgrade from VMFS3
  • Consciously pick which certificate deployment model you will use
  • Investigate smart card authentication, if you have a business requirement for it

Now that we’ve gotten the upgrade and best practices out of the way, in the next installment we will start installing the vSphere 6.0 PSC. You can check out that installment here.

The New High Bar: Nutanix NPX Certification

NPX logoToday Nutanix is proud to announce their Nutanix Platform Expert (NPX) certification. You can read the official press release here. The goal of this certification is to become the most rigorous technical computing qualification in the IT industry. That’s saying a lot, given other live performance based certifications that people are going through today, such as Cisco CCAr and VMware VCDX. They are very rigorous and anyone getting through those live defense processes should be VERY proud of their accomplishments.

Offered at *no charge* this live-defense based certification aims to set the bar even higher, by testing a wider variety of knowledge. For example, you must have “X”-level knowledge of at least two hypervisors of your choice (vSphere, Hyper-V or KVM), “X”-level knowledge of the Nutanix platform, familiar with web-scale concepts, plus the world-class architect and soft consulting skills required for successful global enterprise deployments.

I was lucky enough to be involved in the creation of the NPX program, along with more than a dozen other Nutanix consulting architects, solutions/performance engineers, SEs, and other staff. The bar we set for the minimally qualified candidate is high, comprehensive, and will be a challenge ready for conquering by the brightest minds in the IT industry.

The NPX process consists of two parts: Developing a Nutanix-based enterprise-ready design consisting of a number of documents (see the handbook for more details but this includes a CV, references, emerging technology essay, current state review, migration plan, architecture guide, etc.), submitting that design for review, and then if minimal scoring is met, being invited to defend in front of a live panel. The actual defense will consist of three parts: solution design presentation (90 minutes), hands-on troubleshooting exercise (40 minutes), and quizzing of a 3-tier-to-web-scale migration and second hypervisor solution stack (60 minutes).

During this defense the following skills will be assessed:

Consultation skills

  • Discovery of business requirements
  • Identification of risks and risk elimination or remediation
  • Identification of assumptions and constraints and removal or accommodation in the solution design
  • Incorporation of Web-scale technologies and operational models
  • Evaluation of organizational/operational readiness
  • Migration and transition planning

Conceptual/Logical Design Elements

  • Scalability
  • Resiliency
  • Performance
  • Manageability and Control Plane Architecture
  • Data Protection and Recoverability
  • Compliance and Security
  • Virtual Machine Logical Design
  • Virtual Networking Design
  • Third-party Solution Integration

Physical Design Elements

  • Resource Sizing
  • Storage Infrastructure
  • Platform Selection
  • Networking Infrastructure
  • Virtual Machine Physical Design
  • Management Component Design
  • Datacenter Infrastructure (Environmental and Power)

I was very impressed with the PhD from Alpine Testing that guided us through the rubric creation process, and feel that the result is very fair, relevant, yet obtainable by the right candidate. While there are a set of recommended third-party certifications that the NPX suggests you have passed, there is not a hard requirement to have passed any other third-party certification exam. You must have passed the Nutanix NPP, though.

Click on the graphic below to expand it, and take a look at the recommended primary and secondary certifications. For example, if you wanted to defend on vSphere and Hyper-V, then you should have the skills of a MCSE-Private cloud and VCDX (DCV, DT or Cloud). Again, this is a self-assessment and there is not a hard requirement to have passed these certifications to apply for NPX. But be assured the screening process will weed out those falling short, so don’t think you can fudge it and get NPX certified. Be brutally honest in your self-assessment. 2015-03-13_8-35-27 The screening process for the NPX applications will be comprehensive, and only those meeting a minimum score will be asked to defend. If you don’t meet the documentation bar, or fail the live defense, there are program guidelines for resubmission rules that you can read further about in the NPX documentation. Bottom line, is if you are a Nutanix customer, partner, or work for Nutanix and want to achieve a world class architecture-level certification then download the handbook and read up on exactly what is involved to see if you qualify. If you don’t yet qualify, then get cracking on the requirements, such as “X”-level knowledge of dual hypervisors of your choice.

Personally, I would recommend you actually take and pass the recommended third-party certifications. For example, I found going through the VCDX program to be invaluable on many levels. But Nutanix realizes for various reasons sometimes people can’t sit for those exams (or find little value in multiple choice tests), and we didn’t want that to be a barrier but that in no way lowers the bar since our screening process is very rigorous. Our minimally qualified candidate standard is very high so don’t just throw a 50 page design together and think it can pass.

Other performance based “X” level certification enterprise documentation packages can take months to prepare and run in excess of 200 pages and the NPX certification will be no different. This certification is NOT about showing off your technical prowess, and throwing every possible solution into your design. You shouldn’t include every Nutanix platform in your design, nor should you throw the entire ecosystem of hypervisor products into it either. It’s all about meeting business requirements in an efficient, simple, and easy to manage methodology using a web-scale approach.

To get started on your NPX certification just go to the registration page here. By registering you can download the free NPX Design Review Preparation Guide and the NPX Program Application. You can also contact Mark Brunstad, the NPX Program manager, at npx@Nutanix.com.

If you are aspiring to be an NPX, be sure to check out Rene Van Den Bedem’s NPX Link-o-Rama.

Good luck!

vSphere 6.0 Install Pt. 4: vCenter Upgrade Best Practices

Upgrades can be scary times with any enterprise product. The more your critical infrastructure relies on a particular solution, or set of solutions, the more imperative it is you fully understand and test the new product. Prior vSphere releases has taught us that thorough testing cannot be skipped and you should not rush a new product into production. No product is bug free, and each environment is different.

Normally for my vSphere installation series I do not cover upgrades, or go through an upgrade process in the series. Why? Customer environments wildly vary and a simple lab upgrade will likely not look like or behave like your environment. That’s why its so critical for you to test in your environment. My upgrade would not look like your upgrade. The more complex your topology, such as multiple SSO services, the more critical testing becomes.

But, what I am doing in this post and the next installment is covering upgrade best practices to help you understand your road ahead and things to keep in mind. This post covers vCenter only, and the next installment covers VMs, VMFS, and ESXi hosts.

Blog Series

vSphere 6.0 Install Pt. 1: Introduction
vSphere 6.0 Install Pt. 2: Platform Services Controller
vSphere 6.0 Install Pt. 3: Certificate Management
vSphere 6.0 Install Pt. 4: vCenter Upgrade Best Practices
vSphere 6.0 Install Pt. 5: ESXi Upgrade Best Practices
vSphere 6.0 Install Pt. 6: Install Windows PSC
vSphere 6.0 Install Pt. 7: Config SQL DBs
vSphere 6.0 Install Pt. 8: Install vCenter (coming soon)

Permalink to this series: vexpert.me/Derek60
Permalink to my Toolkit script: vexpert.me/toolkit60

vSphere 6.0 Upgrade Overview

  • Plan your upgrade – Extremely important. KB on update sequence is here.
  • Read the full vSphere 6.0 release notes here
  • Five major steps: vCenter, VUM, ESXi, VMs, VMFS
  • Key VMware Sites to bookmark: Documentation Center, Compatibility Guide, Interop matrix
  • If you upgrade Windows with a service pack or other system changes and get locked out of SSO, read this KB to regain access
  • Great KB on vCenter 6.0 topologies is here

Prior to 5.1 life was simple. You had vCenter Server, vCenter Database server, and vSphere web client. The vCenter server is NOT stateless, meaning the database is not all inclusive. The local vCenter server has SSL certificates and the ADAM database. ADAM is not just for linked mode but holds data such as licenses, roles, and permissions. If you are using vSphere 5.1, then ‘tags’ are also stored locally on the vCenter server and thus not in the database.

Starting with vSphere 5.1 and continuing with 5.5 you now have more roles, such as SSO, and you could even have a distributed topology. This makes upgrades more complex, and requires additional planning. vSphere 6.0 changes that up by adding the Platform Services Controller (PSC), which consumes the SSO service and adds new functionality. ADAM is now gone, replaced by an internal LDAP service.

Upgrade Matrix

  • In-place upgrade supports vCenter 4.x, 5.0.x, 5.1.x, and 5.5
  • VMware does NOT support directly migrating an existing 5.x or earlier vCenter Server to a new machine during the upgrade process
  • vCenter Server 6.0 can manage ESX/ESXi 5.x and higher hosts.
  • Check out the vSphere Upgrade Center here

System Requirements

  • Embedded install – 2 vCPUs, 8GB RAM (tiny environment), 100GB disk Recommended. For 400 hosts or 4000 VMs: 8 vCPU, 24GB RAM, 200GB disk. See this link for more Windows sizing details.
  • vCenter OS Support: Only supports Windows Server 2008 SP2 and later (including WS2012 R2). See this KB for the full support matrix.

New Install vs. In Place Upgrade

VMware recommends a fresh install, but sometimes its not just possible. However, do check out the “Inventory Snapshot” Fling, which is a great (unsupported) tool to migrate hosts, VM, and permissions from one Windows vCenter instance to another. It does NOT appear to support tags and currently has some vDS issues. Tags are not stored in the SQL database, so if you use tags then be sure to find a way to migrate them. If you are in a regulated industry and have strict audit requirements you may be legally required to maintain the historical data in your vCenter database and unable to start fresh.

Very recently released is the VCS to VCVA Converter. What is it? This is an unsupported (officially) method to migrate from a Windows vCenter to the Linux vCenter appliance. It’s released under the technical preview license. It looks very promising, and I’ve seen a lot of buzz on Twitter about it. So check it out, if you want to migrate to the vCenter appliance. I think the vCenter appliance is now production ready at-scale, so this is an excellent time to migrate off Windows.

If you are starting with a fresh install do take a close look at the VCSA. It now supports the same number of VMs and hosts as the Windows version, and is simple to deploy. New to vSphere 6.0 is the ability to do linked mode between VCSA instances. This is due to the removal of ADAM as a Linked Mode dependency. So if you’ve always been a Windows vCenter shop, now is a good time to evaluate going down the VCSA road. It has a new guided install, and pre-check installer too, so VMware is really trying to make it a full replacement. There’s still no external SQL server support, due to the lack of a GA Microsoft ODBC connector. But the embedded database is very scalable, so that shouldn’t be a big factor.

Installation – Then and Now

vSphere 6.0 features a new install sequence with a bit more guidance than previous versions. Gone is the “Simple Install” option and instead a scenario driven installer is used. For example, one of the first screens you will see presents several PSC deployment options. It also features a hard check for 2 vCPUs and at least 8GB of RAM. The following screen then presents you with SSO configuration options, such as creating a new SSO domain or joining an existing one. This is great for upgrades as you can connect to an existing SSO instance.

New to vSphere 6.0 is the embedded vPostgres database, which replaces the prior SQL express option. Don’t worry, you can still specify an external database, such as SQL or Oracle. I also like the new DSN refresh button, so you don’t have to remember to create your DSN before launching the installer. Unlike prior “simple” installer options, this new wizard prompts you for directory paths such as the base vCenter directory and a separate directory for the vCenter/PSC data. Nice!

Before you embark on your vCenter 6.0 install, a MUST read is the VMware vCenter Server 6.0 Deployment guide. It’s in excess of 100 pages, and goes through a lot of upgrade scenarios, deployment topologies, etc. I know it’s long, but after all this is an enterprise product with new topology options. Read thoroughly!

Linked Mode

Linked mode adds additional complications to the upgrade process. As you may recall you can’t link vCenters of different versions. So you first need to unjoin all vCenters from the linked mode group. Once you upgrade two vCenters to 6.0, you can then re-establish Linked Mode and add other 6.0 vCenters as they come online. The biggest problems with Linked Mode include DNS and NTP failures. It’s critical name resolution works (forward AND reverse) and that the server clocks are all synchronized. All vCenter servers that are linked must also be a part of the same SSO authentication domain. New to vSphere 6.0 is the ability to do linked mode between the VCSA and a Windows based vCenter. You can also do linked mode between VCSAs as well!

vCenter Appliance

The VCSA has undergone major scalability increases in 6.0. In 5.1 it was only rated for 5 hosts and 50 VMs when using the embedded database. With 6.0 that is increased to parity with the Windows scalability limits. So that makes it a much more viable solution for enterprise customers. You can NOT migrate from the Windows vCenter to the VCSA, officially. But as previously mentioned, you can try out the VCS to VCVA fling here.

Update Manager

Contrary to some rumors, VUM has not gone away in vSphere 6.0. Apparently the VUM replacement was not quite ready for prime time, so VUM still exists in 6.0. You can upgrade VUM from 4.x, 5.0 and 5.1 versions. VUM is still Windows only, so if you do deploy the VCSA you will still need a Windows server to host VUM. The web client in 6.0 also has limited VUM functionality, so the C# is still needed to do things like pushing patches and configuring baselines. During the upgrade you can’t change the installation or download paths. Scheduled tasks remain, but patch baselines are removed.


You need to carefully plan your upgrades, and understand all of the moving components. Generally you would start by upgrading vCenter, then your ESXi hosts. But you may have other products that depend on vCenter which need upgrading first. Thoroughly map out all of your dependencies, read the VMware documentation, then plan in an organized fashion how you are going to upgrade.

Ready, set, go! Download vSphere 6.0 NOW

After some teasing at VMworld 2014, and a few more sessions at PEX 2015, vSphere 6.0 is finally available for download! If you are in a big hurry to download, here are some useful links. vSphere 6.0 release notes can be found here. As always, TEST TEST TEST before putting this into production.

Also remember that I’m working on a long vSphere 6.0 install/configure series of blog posts, along the lines of what I did for vSphere 5.5. Now that vSphere 6.0 is GA, expect to see new posts on a more frequent basis. I’m also working on a new version of my vCenter SSL toolkit, which will debut sometime in the coming month.

Primary Download Links:

ESXi 6.0 and related ISOs
HP ESXi 6.0 Installer ISO
vCenter 6.0 for Windows and Appliance
PowerCLI 6.0
vSphere 6.0 Replication
Data Protection 6.0
VSAN 6.0


vSphere 6.0 Documentation (Full ZIP)
vSphere 6.0 PowerCLI Documentation

Related products also updated today:

VMware vRealize Automation 6.2.1
Site Recovery Manager 6.0
vRealize Infrastructure Nagivator 5.8.4
vCenter Operations Manager 5.8.5 in Virtual Appliance
vRealize Orchestration Appliance

VMware Horizon 6.1 (Release notes)
VMware Integrated OpenStack

Have fun!

Top Blogger Voting in Full Swing!

2015-03-03_7-55-50Each year Eric Siebert over at vSphere-land.com spends an enormous amount of time setting up the top blogger voting. This recognizes the very hard work that the top bloggers do, and the support they give to the community. Recognition is always fun, but shouldn’t be the primary purpose to blog.

So this year you, again, have your chance to vote for the top bloggers in various categories. When voting, think about what is important to you in a blogger. More ‘newsy’, or hard-core how-articles, or are they more opinion based? How frequently do they post? Do you keep referring back to them, or do they provide scripts/tools that make your job easier? Do they repeat content from other sites/sources or is it original?

Those are some of the criteria I think about when putting in my vote. As a quick year in review for my blog I’ve been consistently updating my vCenter 5.5 SSL toolkit, did a long SQL 2014 Always-On How-To series, live blogged from VMworld 2014 and PEX 2015, plus some Nutanix content.

Weigh in your mind which content has been most valuable to you, then vote based on that information. Last year I made it to #12, which I think had a lot to do with the popular vCenter SSL toolkit. This year I’ll be covering vSphere 6.0 and providing an updated SSL Toolkit.

Voting only runs for two weeks, so take 5 minutes out of your day right now and vote here. Don’t delay or you might forget. Also, remember you can only vote once. So don’t try and game the system, as Eric keeps a close watch out for duplicate or fake votes.

vSphere 6.0 Install Pt. 3: Certificate Management


As long as I can recall certificate management in vSphere has been difficult, and for many customers, something they completely ignore. I’m surprised how many customer designs (even those done by VCDXs) I’ve seen where they feel it’s too difficult to deploy vSphere certificates so they accept the risk of using the non-trusted VMware provided certificates. While I don’t think untrusted SSL certificates are the biggest security threat to an enterprise, I do feel that using trusted certificates is the right thing to do and an add extra layer of security. If you work in a highly regulated industry like finance, healthcare or Government you may be mandated to use fully trusted certificate. Most of my career has been in the Government sector, so using trusted certs was not even a question and just a basic security requirement.

Starting in vSphere 5.1, SSL complexity really shot up and was pretty ‘cocked up’ to put it politely. In vSphere 5.5 VMware did address some of the complexity with a command line tool to help replace certificates. That was still complex, so I wrote the widely used vCenter 5.5 toolkit, which made the whole process super easy. Feedback on that effort has been super positive, and kept me motivated to do the same for 6.0. Now with vSphere 6.0 my toolkit script has to do less because VMware has made it easier, but I still want to make it even easier for customers. Fortunately or unfortunately, depending on how you look at it, vSphere 6.0 has new certificate management options which at first look make SSL more complex than in the past. We’ll dig into each option in this article.

Blog Series

vSphere 6.0 Install Pt. 1: Introduction
vSphere 6.0 Install Pt. 2: Platform Services Controller
vSphere 6.0 Install Pt. 3: Certificate Management
vSphere 6.0 Install Pt. 4: vCenter Upgrade Best Practices
vSphere 6.0 Install Pt. 5: ESXi Upgrade Best Practices
vSphere 6.0 Install Pt. 6: Install Windows PSC
vSphere 6.0 Install Pt. 7: Config SQL DBs
vSphere 6.0 Install Pt. 8: Install vCenter (coming soon)

Permalink to this series: vexpert.me/Derek60
Permalink to my Toolkit script: vexpert.me/toolkit60

Who cares about SSL?

Why should you go through the headaches of replacing all the VMware self-signed certificates? What’s the risk of using untrusted certificates? What can happen if the SSL connection is compromised?

Hypervisors are likely the underpinnings of your business critical apps and intellectual property. If your hypervisor is compromised then it’s just a few short commands to access your critical business data. Unless you like your infrastructure being p0wn3d, then you don’t want your VMware infrastructure compromised. If you don’t use trusted certificates, and just click through all the VI client SSL warnings (you have clicked Ignore and trust this certificate many times…haven’t you?) then you won’t know that a man-in-the-middle attack has taken place.

A man-in-the-middle attack is where a third party intercepts your “secure” communications and relays data between you, the attacker’s device, and the end host (like an ESXi server). This can be accomplished by ARP spoofing, or other means. If it’s second nature to ignore and click through all VI certificate warnings, you will have no idea your credentials have been intercepted….in clear text. No fancy brute force decryption required. Just sit back, grab a coke, and enjoy cleartext flowing across your screen. An interesting article on attacking VMware is here.

There are certainly many other ways to compromise your virtual infrastructure, like stealing the credentials of an administrator and gaining direct access to vCenter. Or using pass the hash, and gaining vCenter access that way. So ‘hacking’ SSL may not be the first choice for an attacker, but it is an attack vector you should consider and secure.

VMware Certificate Authority (VMCA)

This is a new an exciting component in vSphere 6.0 that will radically change how many will issue and deploy SSL certificates in their vSphere environment. SSL certificates are used extensively to secure communications in a vSphere environment. This ensures data confidentiality and integrity. Any attempt to modify data in transit is detected, such as man-in-the-middle attacks.

The VMCA is a built-in certificate authority, which is included in the Platform Services Controller (PSC) service. This is a full blown CA, and can (if you wish) automatically issue certificates to all vCenter 6.0 components and ESXi 6.0 hosts in your environment. The VMCA is mostly command line driven, and does not have a fancy GUI like your Microsoft CA has. But once configured, it’s pretty much a hands off operation. Do take note that VMCA in vSphere 6.0 does NOT support the use of CRLs nor does it have the concept of certificate revocation. If you suspect one certificate was compromised, first remove it then replace all certificates.

VMCA Intermediate Certificate Requirements

If you wish to use the VMCA as a subordinate CA to your existing enterprise CA, take note of the certificate requirements. The requirements are:

  • Private Key Algorithm: RSA with 2048 bits. No fancy elliptical curve support.
  • Recommended Signature Algorithms: SHA256, SHA384, or SHA512
  • NOT Recommended algorithms: MD2, MD5, SHA1
  • Key Usage: Root certificate extension set to true and cert sign must be in the list of requirements
  • Use PEM certificate format, with a header of —–BEGIN CERTIFICATE——
  • Does NOT support Wildcard certificates or more than one DNS name
  • Certificate must be X.509 v3

More about being a subordinate CA later in this article.

Certificate Deployment Options

VMware has come up with four primary certificate deployment options in vSphere 6.0. This is more than any previous release, where you basically had two (use VMware certs, or replace with trusted certs). You need to fully understand all four options, then pick for your environment which one best meets your business and security requirements. Depending on your industry, you may be severely limited in your choices. For example, if you are a U.S. Government agency you are stuck with option 3, using an external CA for all certificates and you won’t care about the new VMCA.


#1 VMCA Root CA

Option #1 is the simplest option, and probably the one a lot of organizations will go with. This is relying on the new VMCA to provision and manage certificates for vCenter and ESXi hosts. The VMCA is automatically created upon PSC installation, and requires no further configuration. However, for services accessed by a web browser (such as the web client) you will get an SSL warning unless you explicitly trust the VMCA root in your browser of choice. This is akin to the VMware signed certificate method in years past. Except in 6.0 there’s now a central CA managing the certificates and their lifecycle. If you do nothing, this is what you will get. Better than in prior vSphere releases, but still not fully trusted certificates. For better security see option #2.

#2 Subordinate VMCA

This is an entirely new option in vSphere 6.0, and wasn’t remotely possible in prior releases. Basically what happens in this mode is that the VMCA imports a root signing certificate from you trusted enterprise root CA. The VMCA then becomes an official subordinate CA to your enterprise root(s). All the certificates issued by the VMCA are trusted by your organization, even the web services exposed in browsers. As you deploy new vSphere components that are VMCA aware, they will get issued trusted SSL certificates. Since the VMCA now manages ESXi 6.0 host certificates, your ESXi hosts will also be issued trusted certificates without any manual intervention.

The BIG downside to this, is also the big upside. The VMware CA is now issuing fully trusted certificates, which may go against company policy. Or, if you are in a regulated environment like the US DoD, there’s no way in hell they will allow you to stand up a trusted subordinate CA. So I would say this option is good for environments that want more than VMware issued certificates, but aren’t so regulated that a VMware subordinate CA would be strictly prohibited. Call this a good compromise between security and simplicity. Thank you VMware! For even more security see option #3.


#3 External CA

Using an External CA is not new, and has always been in option in vSphere dating back many versions. It replaces all certificates in the environment by ones issued from the corporate trusted CAs. VMCA is bypassed, so this is a much more labor intensive process and much higher management complexity. This will be the only option in highly regulated environments, and will cause the most customer pain. All of the benefits of the new VMCA will be ignored, in favor of a higher security posture. This process is also totally different from that in vSphere 5.5, so get ready to learn yet another tedious procedure.


#4 Hybrid

A hybrid scenario features the usage of the VMCA in combination with an external CA. For example, you could use the automated VMCA certificates for all “internal” certificates and ESXi hosts and only replace externally facing certificates (such as web client) with those from an external CA. This adds complexity to the VMCA subordinate option (#2), but is less work than using an external CA for all certificates.  Personally, I don’t see this solution being used too much. I think the other three will be more popular, and the level of regulation and security consciousness will ultimately determine which route to take.

Certificate Types

In the vSphere 5.x era each service was issued its own unique SSL certificate. As you may recall, each certificate had to contain a unique “OU” field otherwise SSO would barf. This does not scale well, as VMware is constantly adding new services to vCenter. Even in vSphere 5.5, my toolkit had to generate 11 certificates for all the services. Whew!

In vSphere 6.0 we now have several types of certificates. As shown in the VMware graphic below, a lot of services are use these ‘common’ certificates. This reduces the total number of certificates needed in the environment.


The following table lists each of the certificate types used in vSphere 6.0, how they are provisioned, and where they are stored.


ESXi Certificate: As has been the case for many years, this certificate is used by the ESXi host to encrypt nearly all communications. Nothing new here.

Machine SSL Certificate: Each node (embedded installation, management node, or PSC) has its own machine SSL certificate. All services running on this node use this certificate for end point encryption. The vCenter service (vpxd), VMware directory service (vmdir) also use these certificates.

Solution User Certificates: These certificates are used for authentication to the vCenter SSO service. Once the certificate is presented to SSO, SSO will issue a SAML token. The service, such as vpxd, can then use this token to authenticate to other services. Baseline solution user certificates include vpxd, vpxd-extensions, and vSphere-webclient.

VMware End Point Certificate Store (VECS)

The VMware End Point Certificate store (VECS) is a local repository for certificates and private keys. VECS is a mandatory component, and will be used even if you don’t sign your certificates with the VMCA. Remember that ESXi certificates are stored on the ESXi host and not in the VECS. The VECS includes a number of keystores including machine SSL certificates, trusted roots, CRLs, solution users (machine, vpxd, vpx-extension, vSphere-webclient) and other keystores such as those for vVols.



Securing your virtual infrastructure is important. There are many attack vectors, and attacking SSL may not be the highest risk. But with vSphere 6.0 and my Toolkit script, replacing SSL certificates is easier than it used to be. So strongly consider taking the time to understand the new deployment methods, assess your business requirements, then take steps to secure your environment. It’s of little use to secure your SSL connections if you give the ESXi root and vCenter admin passwords to everyone.

Next up in the series will be vCenter upgrade and deployment best practices, in Part 4. You can check that out here.

VMTurbo Win a Home Lab

Disclaimer: VMTurbo is a sponsor of this blog.

Do you need a free home lab? Then register for the upcoming VMTurbo 5.1 webinar and get a chance to win a free home lab. Three lucky winners will be chosen at random so carve out an hour of your day and see what’s new with VMTurbo 5.1. Register for free here.


Nutanix NOS 1-Click Upgrade

One of the never ending tasks in IT is keeping up with software builds and firmware updates. These are usually a somewhat painful process, may require downtime, and can often get pushed to the back burner of IT life. At Nutanix we recognized that pain point, and starting in NOS 4.0 we’ve introduced one-click upgrades.

What does one click upgrade really mean? It means that NOS can automatically check for newer software versions of NOS, then all from our HTML5 Prism GUI perform a non-disruptive and automated rolling upgrade of our controller VMs. During this time no vMotions are required, no I/O interruption, no host maintenance modes, and no data relocation is required. It’s so easy, even a junior administrator can perform the upgrade during production hours. Of course you probably will do this during a scheduled maintenance window “just in case”, but theoretically you can do it anytime.

So in this blog post I’ll walk you through the process of a “dark site” one click upgrade. What is a dark site? Well it’s one without internet connectivity, such as a classified environment. Nutanix has a lot of dark site customers, so enabling easy upgrades for this user base was a priority. This process is easy, and if your Nutanix cluster has internet connectivity then it’s even simpler as NOS can automatically download new software updates in the background.

NOS Cluster Upgrade

1. First, login to the Nutanix support portal here. As you can see, the support portal has a similar look at feel to our popular Prism interface. Click on Downloads.


2. Since NOS 4.1.1 is brand new, it’s on the splash page. You will need to download two files, the NOS 4.1.1 tarball, plus the upgrade JSON metadata file. You should also review the release notes prior to doing the upgrade. Blind upgrades are not recommended.


3. After both files have downloaded, login to the Prism interface for the Nutanix cluster that you want to upgrade. Click on the gear icon in the upper right corner and select Upgrade Software. As a side note, from the Prism interface you can see all kinds of cluster stats such as hypervisor version, number of hosts and blocks, performance data, storage summary, and health status. All in glorious HTML5.

2015-02-09_10-24-00a4. On the upgrade software window you can now upload the tarball and the metadata file. It also shows what version the cluster is currently at, which is 4.0.1 in my case.


5. During the upload process it provides a progress indicator, and shows which version you are uploading. In my case I’m updating to 4.1.1.


6. Now that the software and been uploaded, you simply click on Upgrade, acknowledge you want to upgrade, then sit back and wait.


7. During the upgrade a preupgrade check is done, which finishes in a couple of minutes.


8. Once the precheck completes it will start doing the rolling software upgrade. You can expand the progress indicator and see the status of each node. You can even click on Nothing to do, and play Tetris while the upgrade is going on. Yes, Nutanix can also keep you entertained.


9. Here you can see the cluster upgrade is now completed. You are also able to drill into upgrade subtasks, so you know exactly where the upgrade process is at.


10. After the upgrade completes you will now see the new NOS version. And if you look carefully, you will see even MORE 1-click upgrades added. This covers hypervisor upgrades, firmware upgrades, and NCC (a health utility).



As you can see, doing software upgrades for the Nutanix NOS just takes a few clicks. There is ZERO down time, no vMotions needed, no I/O interruption, no host maintenance mode, and no fuss. Starting with NOS 4.1.1 there is now also 1-click upgrades for hypervisors, firmware and NCC. Uncompromisingly simple.

vSphere 6.0 Install Pt. 6: Install PSC

Now that we’ve gotten some background and best practices behind us, now it’s time to start the actual software installation. As previously mentioned, in all but the smallest environments it’s recommended to have a dedicated Platform Services Controller (PSC), rather than an embedded one. So first up here will be provisioning a new Windows Server 2012 R2 VM which will be dedicated to the PSC role. No other vCenter roles or services will be installed on here. Then further along in this series we will provision a second VM, which will run all of the vCenter services. So let’s roll up our sleeves and get started with installing the PSC.

Blog Series

vSphere 6.0 Install Pt. 1: Introduction
vSphere 6.0 Install Pt. 2: Platform Services Controller
vSphere 6.0 Install Pt. 3: Certificate Management
vSphere 6.0 Install Pt. 4: vCenter Upgrade Best Practices
vSphere 6.0 Install Pt. 5: ESXi Upgrade Best Practices
vSphere 6.0 Install Pt. 6: Install Windows PSC
vSphere 6.0 Install Pt. 7: Config SQL DBs
vSphere 6.0 Install Pt. 8: Install vCenter (coming soon)

Permalink to this series: vexpert.me/Derek60
Permalink to my Toolkit script: vexpert.me/toolkit60

Platform Services Controller (PSC) Installation

1. From your favorite template provision a new Windows VM. It needs at least 2 vCPUs and I’d give it 4-6GB of RAM. I would recommend Windows Server 2012 R2. Join it to the domain, and patch it. No need to install Flash, as the PSC has no web or Flash based interface. The PSC install is also very small, so I wouldn’t even attach a second drive. Just install it to the C drive.

2. Mount the vCenter 6.0 ISO and start the Autorun. Click on vCenter Server for Windows then click on Install.


2. Click Next on the build screen.


3. Accept the license agreement.


4. On the Select Deployment Type screen, choose Platform Services Controller. Click Next.


5. At this point the installer should auto-populate the system name with the FQDN of the VM you are installing the PSC on. Accept this value and click Next. As soon as you click next you may get a IPv6 warning. Ignore this warning if you aren’t using IPv6 (and who is?).


6. This is the most important configuration screen. First up, now in vSphere 6.0 you can change the SSO domain. Per VMware do *NOT* make this the same domain as your Active Directory. This will create a show-stopper problem. You CAN, however, use a subdomain of your corporate domain. For example, sso.contoso.local. Personally, I would keep it simple and stick with the default.

Now you will need to configure a password for the administrator@vsphere.local account. This password needs to be at least 8 characters, one lower case character, one numeric character and one special character. And the length must NOT exceed 20 characters. Only use visible ASCII characters, meaning you can’t use a space. You are also not allowed to use the single quote (‘) either. Other special characters may cause problems (remembering the SSO 5.1 and 5.5 days) so be sure to test out your “complex” passwords.

Finally, you can change the site name. I would recommend you do change the name, and have it reference your physical location. While it appears that information is not currently used by vSphere, VMware hinted down the road the ‘site’ name may play a more important role.

If you have a current vSphere deployment that is using SSO, you can join that domain here. In my case I’m doing a fresh install, so I created a new domain.


7. On the Configure Ports screen I would leave these all at their default values.


8. Here you can change the installation directory if you wish. The install is so small that I don’t see the need to install on a D drive, for example.


9. On the final screen it shows a summary of the options you’ve selected. Verify everything is OK and click Install. Wait a few minutes for the installation to complete.



As you can see, the PSC installer is very straight forward. Unlike SSO in vSphere 5.1 that needed an external database, the PSC is fully self-contained. Nice! Now that we have the PSC up and running, it is time to install the Windows vCenter server.

© 2015 - Sitemap