New Veeam v8 Nutanix Guides Available

Last year I blogged about the creation of Veeam v7 best practice guides for Nutanix plus VMware and Hyper-V. Those guides have been very popular, and helped a number of our customers and SEs. One of the number one compliments of Veeam that I hear from the field is now simple it is to install and configure, just like Nutanix. No professional services are needed, or days long installs. Click through the install wizard, and you are practically ready to do backups. Competing products can be vastly much more complicated to install and configure.

I’m proud to announce that both guides (VMware and Hyper-V) have been refreshed for some of the new features in Veeam v8. Veeam v8 was a major release, with hundreds of new features and enhancements. Luca Dell’Oca from Veeam spearheaded the updates, which are now available for immediate download.

Veeam and Nutanix Best Practice Guide for VMware

Veeam and Nutanix Best Practices Guide for Hyper-V

So if you are a Nutanix customer and either currently use Veeam, or are looking at Veeam, please downloads the guides and review the contents. I’d also like to take a minute to address one point of discussion brought up in the guides. The VMware guide recommends the use of ‘network mode’ backups versus hot-add. There’s a common misperception that network mode is somehow dog slow, and only hot-add should be used. When using 10G NICs, backup speeds with network mode should not be a problem. In fact, hot-add mode can take 2-3 minutes per VM just to perform the hot-add operation and multiply that by hundreds of VMs and that’s hours of waiting. For quick incremental backups, that can dramatically slow down the job progress.

Thus after in house testing and collaboration with Veeam, we are recommending network mode backups. If you are having backup performance issues even after following the guides, I encourage you to open a support ticket with either Nutanix or Veeam, and our support professionals can get to the bottom of your particular issue. Backups are complicated and everyone’s infrastructure is different, so minor tweaks may be needed to optimize backup throughput. Also, keep up to date with the latest Veeam updates, as newer updates are faster and more stable. The same applies to keeping up with Nutanix NOS updates, as we’ve seen significant performance increases with many of our releases. The beauty of software defined storage!

vSphere Install Pt. 16: User Solution Certificates

Now that we have vCenter installed, it’s time to update our User Solution certificates for the vCenter services. This is a fairly straight forward process, using the combination of the VMware Certificate Manager tool and my vCenter 6.0 Toolkit. The VMware Certificate manager tool will automatically create the private keys and CSRs for each user solution certificate. My toolkit will then take the CSRs and submit them to your enterprise CA and also create the chained PEM files the VMware toolkit needs to install the certificates. Then we flip back to the VMware tool to let it actually install the certificates. I decided against duplicating functionality between my Toolkit and the VMware tool, so there’s  little flipping back and forth.

If you are using the VMCA, then that’s even easier, as we can fully rely on the VMware tool to update the required certificates. I’ll go over all of the scenarios here.

Also take note that you need at least version 0.85 of my vCenter toolkit for this article to work properly. So download it, or a newer version, from the permalink below.

Blog Series

vSphere 6.0 Install Pt. 1: Introduction
vSphere 6.0 Install Pt. 2: Platform Services Controller
vSphere 6.0 Install Pt. 3: Certificate Management
vSphere 6.0 Install Pt. 4: vCenter Upgrade Best Practices
vSphere 6.0 Install Pt. 5: ESXi Upgrade Best Practices
vSphere 6.0 Install Pt. 6: Install Windows PSC
vSphere 6.0 Install Pt. 7: Config SQL DBs
vSphere 6.0 Install Pt. 8: Toolkit Configuration
vSphere 6.0 Install Pt. 9: SSL Templates
vSphere 6.0 Install Pt. 10: Install VCSA PSC
vSphere 6.0 Install Pt. 11: VMCA as Subordinate
vSphere 6.0 Install Pt. 12: PSC Machine Certificate
vSphere 6.0 Install Pt. 13: Directory Services Certificate
vSphere 6.0 Install Pt. 14: Windows vCenter Install
vSphere 6.0 Install Pt. 15: VCSA vCenter Install
vSphere 6.0 Install Pt. 16: User Solution Certificates

Permalink to this series:
Permalink to my Toolkit script:

User Solution Certificates with VMCA

1. Open a command prompt and run the ‘certificate-manager’ tool from C:\Program Files\VMware\vCenter Server\vmcad. If you are using the VCSA, open a bash shell and go to the /usr/lib/vmware-vmca/bin directory.

2. Select Option 6, Replace Solution user certificates with VMCA certificates.


3. Enter your SSO password.

4. Enter the IP address of your external PSC. Confirm you want to replace the certificates using the VMCA. Wait a couple of minutes for the procedure to complete.


User Solution Certificates with Custom Certificates

1. Open a command prompt and run the ‘certificate-manager’ tool from C:\Program Files\VMware\vCenter Server\vmcad. If you are using the VCSA, open a bash shell and go to the /usr/lib/vmware-vmca/bin directory.

2. From the main menu select Option 5. Enter your SSO password and PSC IP address.

3. Select Option 1 from the sub menu, to generate CSRs and keys. Enter a directory path of C:\Certs. If you are using the VCSA, enter an appropriate local directory.


4. If you look in the C:\Certs directory you will see a bunch of files created. If you are using the VCSA, copy all of the created files down into C:\Certs.


5. Open a new PowerShell window and launch my vCenter 6.0 Toolkit. Select Option 5 from the main menu, “User Solution Certificate Menu”.

6. If you are using an Online Microsoft CA then select Option 1, Mint User Solution certificates with an online Microsoft CA. Wait a few seconds, and all of the CSRs will be submitted to your online CA and the certificates downloaded. If your CA requires certificate approval, go to your CA approve the certificates, then select Option 2 to resume the download.

7. If you look at the C:\Certs directory you will now see several subdirectories, one for each corresponding CSR. Skip to Step 10 if you are an online Microsoft CA user. 2015-04-25_13-05-578. If you need to manually submit all of the CSRs to your CA (offline Microsoft CA, or third-party CA), then save each minted certificate as a base-64 encoded non-chained file with the following names in the C:\Certs directory:


9. From my User Solution Certificate menu select option 3, which will create your PEM files and move your certificate files into their own directory. Only use this option if you manually downloaded your CRT files from your CA.

10. If you are using the VCSA, copy the new folders in C:\Certs up to the appliance. Also, upload the chain.cer file as well.

10. Back in the VMware Certificate Manager tool select Option 2, Import Custom certificates… Input all of the requested file names, using the “.cer” and “.key” filenames for the corresponding option. Note: Due to a bug, if you try and use the “chain.cer” file for the signing certificate, the operation may fail at 0% and rollback. So until they fix the bug, use the “root64.cer” file for the last response. 2015-05-02_17-22-32

11. Type Y to continue with the replacement. Wait until the process is completed.


Replacing the user solution certificates is not a difficult process, if you combine my Toolkit script with the VMware certificate manager. Even with the multiple CA VMware bug, there’s an easy workaround .

VMworld Ticket Sweepstakes

Just like last year, VMTurbo is giving away a select number of free tickets to VMworld. They cover the conference fee, you cover travel and hotel. A great deal for those where your company won’t send you, or you are an independent consultant. There are three drawings, one each on May 29, June 19, and July 10th. Entry is free, so try to secure yourself a ticket today! Use this link for entry.

Channel 9 Ignite 2015 Session Downloader

As you know if you’ve been following my blog this week, Ignite 2015 took place in Chicago with hundreds of great sessions. In fact, this year all but one or two sessions were spot on. You can easily download all of the great Channel 9 recordings using the PowerShell script you can download here. The conference just ended today, so it might take a few days before Channel 9 gets all of the recordings up. Happy downloading!

Ignite 2015: Encryption, Certificates and PKI

Session: BRK3130

Note: This was a great beginner level session for those not familiar with encryption, certificates or PKI. If you are in that boat, I would urge you to find the session video and watch the whole presentation. If you are a security professional and already know about these topics, then the content is probably too basic. I didn’t capture all the content below, but just took down some highlights what was covered.

Why am I here? Thanks to the NSA. Thanks to Edward Snowden. SharePoint, Lync, Exchange all  need to be secure.

Shows screens of RDP SSL warnings, and browser SSL warnings.

Are you still using passwords? Phishing and fraud, password fatigue, pass the hash attacks

IoT (Internet of things) is adding new concerns of authentication (connected cars, medical, industrial sensors)

Non-repudiation – Ability to bind a human to a digital document

Privacy – Hot topic over the last 2 years due to NSA and Snowden. Challenges are not new.

Encryption – Encryption at rest, in transit, challenges: weak algorithms

Encryption at rest – Bitlocker, EFS, SQL TDE

Encryption in transit – SSL/TLS, IPsec, Office 365 message encryption

Azure RMS – AD RMS for On-Premises. Protect documents from Birth to end of life. Protection regardless of location.

Speaker goes over symmetrical, asymmetrical encryption, hardware security modules (HSM) technologies such as AES and shows how they work.

What is hashing? Uniquely identify a stream of data. It’s a one way function.IMAG0425

Use the tool IIS Crypto to disable/enable and change the order that ciphers are use. FREE.

Good ideas: Remove RC4, reorder suites, Update to 2012 R2, research ECC vs. RSA

Talks about Certificate Authorities, certificates, and their basic properties. Also discusses path of trust, and where to find certificates in Windows.

CA Lifetime planning: End certs – 2 years, intermediate CA – 4 years, root CA – 8 years. Renew certificates when 50% of their life has expired.

S/MIME – For Email encryption and digital signatures

Ignite 2015: Windows Hello

Session: BRK2324

  • Shared secrets are easily breached
  • Passwords are easily replayed and phished
  • See previous “Microsoft Passport” session I blogged about for more info
  • Security without convenience is dead in the water
  • Keys are ideally generated in hardware TPM, software as last resort
  • Single unlock gesture provides access to multiple credentials
  • Browser support via JS/Webcrypto APIs to create and use Passport users

Windows Hello

  • Supports biometric authentication
  • Convenient device logon and strong user authentication
  • Enterprise level security and access to high impact data and resources via Microsoft passport
  • Consistent inbox user enrollment

Biometric Steps

  • Enrollment Steps – Face, iris, and fingerprint share the same design
  • Usage – Authentication
  • Recovery – User can delete enrollment data. Stored strictly on local device.

Enrollment – Find a face, discover landmarks, detect head orientation, build & secure vector based template

Recovery – After 5 failures it falls back to PIN or another auth method. After 32 failures the TPM is locked.

There’s an option to improve face recognition where it will take additional data points

It can also use fingerprints and will use between 21 and 40 points, all stored locally on the device

Only supports a single face mapped to a single account. No multiple faces for a single account.

Authentication vs. Identification

  • Not every biometic modality is created equal
  • False acceptance rate
  • False rejection rate
  • Liveness and anti-spoofing – Can detect dead fingers and high res photos
  • Windows hello demonstrates false rejection rate of 1/100000
  • Windows Hello False rejection rate is 2-4%
  • Windows Hello requires liveness detection and anti-spoofing
  • Microsoft has captured 13K faces for a representative sample

Microsoft Hello Camera can work without visible light. It operates on IR. Speaker demod showing a picture and phone to the camera and it did not work.

Microsoft goal is to make biometics non-susceptible to spoofing, offline attacks, etc.



Ignite 2015: Benchmarking SQL AlwaysOn

Session: BRK3557: Baselining and Benchmarking AlwaysOn Availability Groups

In this session the speaker went through what SQL AlwaysOn availability groups is, and why the customer wanted to use it. Then he went through how he setup his testing, RAID levels, and listed the SQL perform stats that he monitored during the benchmarking. The speaker used a scripted run of SQLIO to perform his benchmark tests. He covered SQL IO sizes, number of threads, and how to scale up to simulate the customer’s environment.

He went into a long discussion about max threads, and how the type of query affects how many threads are spawned. SQL has a max number of worker threads, so understanding how many threads you are spawning when doing at-scale testing is important. He also tuned the cost threshold for parallelism to control the number of spawned threads.

In  the end, he was successful in performing at-scale benchmarks and the customer’s system was implemented successfully. Be sure to check out the session recording for all of the gory details.



Ignite 2015: Remotely managing Nano Server

Session: BRK3455

Note: This session had very densely packed slides and lots of demos. So I’ve changed things up and just included screenshots for this write up. If you want to run Nano I encourage you to check out the video recording to see all of the demos.

Voice of the customer: Reboots impact my business; Server images are too big; Infrastructure requires too many resources; Security impact

Demos that Server Manager GUI “just works” against Windows Server Nano 2016

Remotely Managing Nano Server:













Ignite 2015: Stretching failover clusters in WS2016

Session: BRK3487

Note: This session was jam packed with slides, text, and diagrams. The speaker was also flying through the material, so it was impossible to attempt to keep up. The session was very good, and quite technical. So if you deal with clustering in your daily job, check out the session recording for a boatload of good info.

  • Stretch clusters can achieve low RPO and RTO
  • Disaster avoidance is the new trend
  • Considerations when stretching clusters: Networking, storage

Recommendations: Adjust intra-node heartbeat thresholds; understand

Cloud Witness in Windows Server 2016

  • Leverages Azure as arbitration point
  • Quorum configuration achieved without an extra site
  • Writes a single blob per cluster
  • Costs on Azure is extremely low…in terms of pennies
  • Newly recommended quorum option

Storage Considerations

  • Storage replica is a brand new feature in WS2016
  • Block-level, volume-based synchronous & async using SMB 3.1.1
  • Any Windows volume, any fixed disk storage, any storage fabric
  • Baked into need for third party storage

Hyper-V and General use file server are the main use cases for the tech preview. Not for SoFS.

Requirements & Recommendations

  • Datacenter edition & Azure stack SKUs only
  • Requires Active Directory (no schema updates, just Kerberos)
  • >1Gb network between servers
  • Disks: Must be GPT not MBR.
  • Free space on logs on NTFS/ReFS volume
  • Disk physical sector sizes must be the same (e.g. can’t mix 512e & 4K)
  • Network latency: 5ms round trip
  • Reality: 30-50Km apart
  • Network bandwidth is based on IO of the app and IOPS
  • Log volumes recommended on Flash (SSD, NVMe, etc.)
  • These are *strong* recommendations
  • Supports running inside a VM

Ignite 2015: VMM Overview & Roadmap

Session: BRK2473

Note: This session was 50% about what’s new in VMM 2012 R2, with 15-20 minutes on what’s new in VMM 2016. My take away is that MS is trying to listen to customers and make the product easier to use. But don’t expect any radical changes in VMM (which I think are needed) …just specific feature updates to keep up with the Hyper-V platform. They didn’t stay very long on the VMM 2016 slide, so I didn’t capture everything. See the session recording if you want the full scoop.

SCVMM 2012

Update Rollup 6 was just released – New functionality added

VMM team is now shipping new features in URs, versus having to wait for an entire new release

Microsoft made a point of including user and automated feedback into the design of VMM, and bug fixes.

New Improvements in UR5/UR6:

  • DHCP extension update
  • New Linux OS versions added
  • Maintenance mode behavior fixed
  • Improved performance over WAN links
  • Quicker VM deletion
  • SQL 2014 support
  • Integrate SAN remote replication with ASR
  • New management of vSphere 5.5
  • Added Azure & AWS connectivity & VM support
  • ..many other on the list

Want to get early drops?

VMM 2016:

  • Ease of use – workflow for host and storage cluster creation; simplified logical switch creation and deployment; Flexible bare mental provisioning; Improved diff disk managment
  • Security and Infrastruture – Deploy guarded hosts, manage guarded hosts, protect tenant secrets, improved state consistency
  • Expanded fabric management – Storage replication automation using Azure site recovery; Scale-out file server with SAN storage automation; storage QoS policy management.
© 2015 - Sitemap