Archives for May 2015

vSphere Install Pt. 16: User Solution Certificates

Now that we have vCenter installed, it’s time to update our User Solution certificates for the vCenter services. This is a fairly straight forward process, using the combination of the VMware Certificate Manager tool and my vCenter 6.0 Toolkit. The VMware Certificate manager tool will automatically create the private keys and CSRs for each user solution certificate. My toolkit will then take the CSRs and submit them to your enterprise CA and also create the chained PEM files the VMware toolkit needs to install the certificates. Then we flip back to the VMware tool to let it actually install the certificates. I decided against duplicating functionality between my Toolkit and the VMware tool, so there’s  little flipping back and forth.

If you are using the VMCA, then that’s even easier, as we can fully rely on the VMware tool to update the required certificates. I’ll go over all of the scenarios here.

Also take note that you need at least version 0.85 of my vCenter toolkit for this article to work properly. So download it, or a newer version, from the permalink below.

Blog Series

vSphere 6.0 Install Pt. 1: Introduction
vSphere 6.0 Install Pt. 2: Platform Services Controller
vSphere 6.0 Install Pt. 3: Certificate Management
vSphere 6.0 Install Pt. 4: vCenter Upgrade Best Practices
vSphere 6.0 Install Pt. 5: ESXi Upgrade Best Practices
vSphere 6.0 Install Pt. 6: Install Windows PSC
vSphere 6.0 Install Pt. 7: Config SQL DBs
vSphere 6.0 Install Pt. 8: Toolkit Configuration
vSphere 6.0 Install Pt. 9: SSL Templates
vSphere 6.0 Install Pt. 10: Install VCSA PSC
vSphere 6.0 Install Pt. 11: VMCA as Subordinate
vSphere 6.0 Install Pt. 12: PSC Machine Certificate
vSphere 6.0 Install Pt. 13: Directory Services Certificate
vSphere 6.0 Install Pt. 14: Windows vCenter Install
vSphere 6.0 Install Pt. 15: VCSA vCenter Install
vSphere 6.0 Install Pt. 16: User Solution Certificates

Permalink to this series:
Permalink to my Toolkit script:

User Solution Certificates with VMCA

1. Open a command prompt and run the ‘certificate-manager’ tool from C:\Program Files\VMware\vCenter Server\vmcad. If you are using the VCSA, open a bash shell and go to the /usr/lib/vmware-vmca/bin directory.

2. Select Option 6, Replace Solution user certificates with VMCA certificates.


3. Enter your SSO password.

4. Enter the IP address of your external PSC. Confirm you want to replace the certificates using the VMCA. Wait a couple of minutes for the procedure to complete.


User Solution Certificates with Custom Certificates

1. Open a command prompt and run the ‘certificate-manager’ tool from C:\Program Files\VMware\vCenter Server\vmcad. If you are using the VCSA, open a bash shell and go to the /usr/lib/vmware-vmca/bin directory.

2. From the main menu select Option 5. Enter your SSO password and PSC IP address.

3. Select Option 1 from the sub menu, to generate CSRs and keys. Enter a directory path of C:\Certs. If you are using the VCSA, enter an appropriate local directory.


4. If you look in the C:\Certs directory you will see a bunch of files created. If you are using the VCSA, copy all of the created files down into C:\Certs.


5. Open a new PowerShell window and launch my vCenter 6.0 Toolkit. Select Option 5 from the main menu, “User Solution Certificate Menu”.

6. If you are using an Online Microsoft CA then select Option 1, Mint User Solution certificates with an online Microsoft CA. Wait a few seconds, and all of the CSRs will be submitted to your online CA and the certificates downloaded. If your CA requires certificate approval, go to your CA approve the certificates, then select Option 2 to resume the download.

7. If you look at the C:\Certs directory you will now see several subdirectories, one for each corresponding CSR. Skip to Step 10 if you are an online Microsoft CA user. 2015-04-25_13-05-578. If you need to manually submit all of the CSRs to your CA (offline Microsoft CA, or third-party CA), then save each minted certificate as a base-64 encoded non-chained file with the following names in the C:\Certs directory:


9. From my User Solution Certificate menu select option 3, which will create your PEM files and move your certificate files into their own directory. Only use this option if you manually downloaded your CRT files from your CA.

10. If you are using the VCSA, copy the new folders in C:\Certs up to the appliance. Also, upload the chain.cer file as well.

10. Back in the VMware Certificate Manager tool select Option 2, Import Custom certificates… Input all of the requested file names, using the “.cer” and “.key” filenames for the corresponding option. Note: Due to a bug, if you try and use the “chain.cer” file for the signing certificate, the operation may fail at 0% and rollback. So until they fix the bug, use the “root64.cer” file for the last response. 2015-05-02_17-22-32

11. Type Y to continue with the replacement. Wait until the process is completed.


Replacing the user solution certificates is not a difficult process, if you combine my Toolkit script with the VMware certificate manager. Even with the multiple CA VMware bug, there’s an easy workaround .

VMworld Ticket Sweepstakes

Just like last year, VMTurbo is giving away a select number of free tickets to VMworld. They cover the conference fee, you cover travel and hotel. A great deal for those where your company won’t send you, or you are an independent consultant. There are three drawings, one each on May 29, June 19, and July 10th. Entry is free, so try to secure yourself a ticket today! Use this link for entry.

Channel 9 Ignite 2015 Session Downloader

As you know if you’ve been following my blog this week, Ignite 2015 took place in Chicago with hundreds of great sessions. In fact, this year all but one or two sessions were spot on. You can easily download all of the great Channel 9 recordings using the PowerShell script you can download here. The conference just ended today, so it might take a few days before Channel 9 gets all of the recordings up. Happy downloading!

Ignite 2015: Encryption, Certificates and PKI

Session: BRK3130

Note: This was a great beginner level session for those not familiar with encryption, certificates or PKI. If you are in that boat, I would urge you to find the session video and watch the whole presentation. If you are a security professional and already know about these topics, then the content is probably too basic. I didn’t capture all the content below, but just took down some highlights what was covered.

Why am I here? Thanks to the NSA. Thanks to Edward Snowden. SharePoint, Lync, Exchange all  need to be secure.

Shows screens of RDP SSL warnings, and browser SSL warnings.

Are you still using passwords? Phishing and fraud, password fatigue, pass the hash attacks

IoT (Internet of things) is adding new concerns of authentication (connected cars, medical, industrial sensors)

Non-repudiation – Ability to bind a human to a digital document

Privacy – Hot topic over the last 2 years due to NSA and Snowden. Challenges are not new.

Encryption – Encryption at rest, in transit, challenges: weak algorithms

Encryption at rest – Bitlocker, EFS, SQL TDE

Encryption in transit – SSL/TLS, IPsec, Office 365 message encryption

Azure RMS – AD RMS for On-Premises. Protect documents from Birth to end of life. Protection regardless of location.

Speaker goes over symmetrical, asymmetrical encryption, hardware security modules (HSM) technologies such as AES and shows how they work.

What is hashing? Uniquely identify a stream of data. It’s a one way function.IMAG0425

Use the tool IIS Crypto to disable/enable and change the order that ciphers are use. FREE.

Good ideas: Remove RC4, reorder suites, Update to 2012 R2, research ECC vs. RSA

Talks about Certificate Authorities, certificates, and their basic properties. Also discusses path of trust, and where to find certificates in Windows.

CA Lifetime planning: End certs – 2 years, intermediate CA – 4 years, root CA – 8 years. Renew certificates when 50% of their life has expired.

S/MIME – For Email encryption and digital signatures

Ignite 2015: Windows Hello

Session: BRK2324

  • Shared secrets are easily breached
  • Passwords are easily replayed and phished
  • See previous “Microsoft Passport” session I blogged about for more info
  • Security without convenience is dead in the water
  • Keys are ideally generated in hardware TPM, software as last resort
  • Single unlock gesture provides access to multiple credentials
  • Browser support via JS/Webcrypto APIs to create and use Passport users

Windows Hello

  • Supports biometric authentication
  • Convenient device logon and strong user authentication
  • Enterprise level security and access to high impact data and resources via Microsoft passport
  • Consistent inbox user enrollment

Biometric Steps

  • Enrollment Steps – Face, iris, and fingerprint share the same design
  • Usage – Authentication
  • Recovery – User can delete enrollment data. Stored strictly on local device.

Enrollment – Find a face, discover landmarks, detect head orientation, build & secure vector based template

Recovery – After 5 failures it falls back to PIN or another auth method. After 32 failures the TPM is locked.

There’s an option to improve face recognition where it will take additional data points

It can also use fingerprints and will use between 21 and 40 points, all stored locally on the device

Only supports a single face mapped to a single account. No multiple faces for a single account.

Authentication vs. Identification

  • Not every biometic modality is created equal
  • False acceptance rate
  • False rejection rate
  • Liveness and anti-spoofing – Can detect dead fingers and high res photos
  • Windows hello demonstrates false rejection rate of 1/100000
  • Windows Hello False rejection rate is 2-4%
  • Windows Hello requires liveness detection and anti-spoofing
  • Microsoft has captured 13K faces for a representative sample

Microsoft Hello Camera can work without visible light. It operates on IR. Speaker demod showing a picture and phone to the camera and it did not work.

Microsoft goal is to make biometics non-susceptible to spoofing, offline attacks, etc.



Ignite 2015: Benchmarking SQL AlwaysOn

Session: BRK3557: Baselining and Benchmarking AlwaysOn Availability Groups

In this session the speaker went through what SQL AlwaysOn availability groups is, and why the customer wanted to use it. Then he went through how he setup his testing, RAID levels, and listed the SQL perform stats that he monitored during the benchmarking. The speaker used a scripted run of SQLIO to perform his benchmark tests. He covered SQL IO sizes, number of threads, and how to scale up to simulate the customer’s environment.

He went into a long discussion about max threads, and how the type of query affects how many threads are spawned. SQL has a max number of worker threads, so understanding how many threads you are spawning when doing at-scale testing is important. He also tuned the cost threshold for parallelism to control the number of spawned threads.

In  the end, he was successful in performing at-scale benchmarks and the customer’s system was implemented successfully. Be sure to check out the session recording for all of the gory details.



Ignite 2015: Remotely managing Nano Server

Session: BRK3455

Note: This session had very densely packed slides and lots of demos. So I’ve changed things up and just included screenshots for this write up. If you want to run Nano I encourage you to check out the video recording to see all of the demos.

Voice of the customer: Reboots impact my business; Server images are too big; Infrastructure requires too many resources; Security impact

Demos that Server Manager GUI “just works” against Windows Server Nano 2016

Remotely Managing Nano Server:













Ignite 2015: Stretching failover clusters in WS2016

Session: BRK3487

Note: This session was jam packed with slides, text, and diagrams. The speaker was also flying through the material, so it was impossible to attempt to keep up. The session was very good, and quite technical. So if you deal with clustering in your daily job, check out the session recording for a boatload of good info.

  • Stretch clusters can achieve low RPO and RTO
  • Disaster avoidance is the new trend
  • Considerations when stretching clusters: Networking, storage

Recommendations: Adjust intra-node heartbeat thresholds; understand

Cloud Witness in Windows Server 2016

  • Leverages Azure as arbitration point
  • Quorum configuration achieved without an extra site
  • Writes a single blob per cluster
  • Costs on Azure is extremely low…in terms of pennies
  • Newly recommended quorum option

Storage Considerations

  • Storage replica is a brand new feature in WS2016
  • Block-level, volume-based synchronous & async using SMB 3.1.1
  • Any Windows volume, any fixed disk storage, any storage fabric
  • Baked into need for third party storage

Hyper-V and General use file server are the main use cases for the tech preview. Not for SoFS.

Requirements & Recommendations

  • Datacenter edition & Azure stack SKUs only
  • Requires Active Directory (no schema updates, just Kerberos)
  • >1Gb network between servers
  • Disks: Must be GPT not MBR.
  • Free space on logs on NTFS/ReFS volume
  • Disk physical sector sizes must be the same (e.g. can’t mix 512e & 4K)
  • Network latency: 5ms round trip
  • Reality: 30-50Km apart
  • Network bandwidth is based on IO of the app and IOPS
  • Log volumes recommended on Flash (SSD, NVMe, etc.)
  • These are *strong* recommendations
  • Supports running inside a VM

Ignite 2015: VMM Overview & Roadmap

Session: BRK2473

Note: This session was 50% about what’s new in VMM 2012 R2, with 15-20 minutes on what’s new in VMM 2016. My take away is that MS is trying to listen to customers and make the product easier to use. But don’t expect any radical changes in VMM (which I think are needed) …just specific feature updates to keep up with the Hyper-V platform. They didn’t stay very long on the VMM 2016 slide, so I didn’t capture everything. See the session recording if you want the full scoop.

SCVMM 2012

Update Rollup 6 was just released – New functionality added

VMM team is now shipping new features in URs, versus having to wait for an entire new release

Microsoft made a point of including user and automated feedback into the design of VMM, and bug fixes.

New Improvements in UR5/UR6:

  • DHCP extension update
  • New Linux OS versions added
  • Maintenance mode behavior fixed
  • Improved performance over WAN links
  • Quicker VM deletion
  • SQL 2014 support
  • Integrate SAN remote replication with ASR
  • New management of vSphere 5.5
  • Added Azure & AWS connectivity & VM support
  • ..many other on the list

Want to get early drops?

VMM 2016:

  • Ease of use – workflow for host and storage cluster creation; simplified logical switch creation and deployment; Flexible bare mental provisioning; Improved diff disk managment
  • Security and Infrastruture – Deploy guarded hosts, manage guarded hosts, protect tenant secrets, improved state consistency
  • Expanded fabric management – Storage replication automation using Azure site recovery; Scale-out file server with SAN storage automation; storage QoS policy management.

Ignite 2015: Hyper-V 2016

Session: BRK3461

Note: The focus of this session is on what’s new in Hyper-V technical preview 2. It will NOT cover all the new features, or features in future server builds. The presenter all flipped through the slides very fast, so I didn’t get all of the details. I recommend you watch the video if this topic interests you.

Nano server is the recommended deployment model for Hyper-V

Virtual Machine Protection

  • Trust is the biggest blocker to cloud adoption
  • MS wants customers to know their data is secure
  • Virtual TPM and secure boot with Linux (Ubuntu 14.04 or later and SUSE)
  • Shielded Virtual Machines – Supports bitlocker inside of the VM, plus other features


  • Storage QoS
  • Can set a policy that caps the IOPS across multiple VMs and they share the policy
  • Great for service providers
  • Host resource protection: Dynamically identify VMs that are not playing well and reduce their resource allocation. Can help protect against malware taking over resources.


  • Today, if you have a temp network outage the hyperV cluster will panic and fall apart in a very bad way. If the storage outage goes above 60 seconds, I/Os will fail and the guest OS will likely crash.
  • Virtual machine storage resiliency – VM is paused/suspended until storage access resumes
  • Virtual machine cluster resiliency – 4 minute timeout for cluster services being stopped, with automatic healing. Another resiliency feature for flapping cluster services due to HW issues, and the host will be quarantined and VMs live migrated off after a certain period.

Shared VHDX

  • Going to allow host based (agent free) backups with shared VHDXs
  • Now you can back up cluster as easy as standalone servers
  • Now allows online resizing of shared VHDXs
  • New VHDX type: VHDS

Replica support for hot add of VHDX. When you add a new disk it added it’s into the non-replicated set.

Runtime resize of memory – For Ws2016 and Windows 10, you can increase/decrease the runtime memory while the VM is running.

Hot add/remove of network adapters. Applicable to Generation 2 VMs only.

Rolling cluster upgrade

  • You can now upgrade a 2012 R2 Hyper-V to WS Tech Preview 2 with no downtime, no new hardware, and ability to rollback.

Operational Improvements

  • Production checkpoints – Uses VSS instead of saved state to create checkpoint. Fully supported in production. FINALLY!

PowerShell Direct to Guest OS

ReFS Accelerated VHDX Operations – Instant fixed disk creation and merging of checkpoints. “Instantly” create fixed disks in about 3 seconds of almost any size. Merging checkpoints happens without data being copied.

Changing how we handle VM servicing

  • Integration components are now distributed via Windows update

Evolving Hyper-V Backup: New architecture plus change block tracking is now native

VM Configuration files: VMCX and VMRS. Now a binary format efficient at scale