VMworld 2016: vCenter Performance

Session: INF8108

Teaser: 5.5 to 6.0 is 300% faster for vCenter operations From 6.0 to 6.5(?) is another 100% higher.

HTML5 vs. Flex client: Shows a chart with dramatically faster HtML5 performance over Flash client.

VCSA vs. windows – When you have a datacenter with heavy load the VCSA far out performs Windows vCenter.

Why move to multiple vCenters?

  • Concurrency
  • Business separation?
  • Geography
  • VDI vs. server workloads
  • Large inventory
  • If VC is 70% CPU or memory, split it

Future: External load balancer will be built-in when using multiple vCenter servers. No more external load balancer needed.


  • When? If latency is 40ms or greater between sites.

PSC Performance Considerations

  • Default size of 2 vCPU and 4GB is sufficient for a majority of customers

vCenter Server Performance Considerations

  • vCenter can accept 2000 concurrent sessions – hard limit
  • VPXD can handle 640 tasks before they get queued – another hard limit
  • Per-host and per-datastore limits: A host can perform up to 8 provisioning operations at once.
  • A datastore can perform up to 128 vMotions at once
  • A datastore can perform up to 8 storage vMotions at once
  • A 10Gb NIC allows a host to do 2x more vMotions than a 1Gb NIC
  • Latency between vCenter and hosts (ROBO) is not a huge issue.
  • Latency between vCenter and the database can impact performance (>10ms)

Impact of Changing Stats level on DB/network traffic

  • Changing between level 1 and level 2, there is a 4x increase in storage/network usage
  • Changing from level 2 to level 3 is another 2x increase

Database Performance

  • Occasional 3-4s query time is fine
  • 10s or more queries are BAD

Is something slow?

  • Check memory/heap size of vSphere-client process on vCenter
  • Memory/CPU of machine running web browser
  • Are plugins functioning?
  • Connection between browser and vCenter
  • vCenter CPU should not exceed 70% on average (spikes are perfectly normal)

If using Windows vCenter, use the SysInternals Process explorer to map Java processes to vCenter services

For VCSA use VIMtop to look at performance

If you increase VCSA memory, heap sizes will automatically be increased upon reboot.


VMworld 2016: What’s new with vSphere

Session: INF8375

What happened since VMworld 2015?

  • End of availability of C# client in next major release
  • HTML5 web client fling
  • 6.0 U2 – Q1 2016
  • Pricing and packaging changes – No more vSphere “enterprise” edition, or vSOM standard and enterprise.
  • End of sale of vSphere 5.0 and 5.1 (August 24, 2016)
  • Open source of PowerShell by Microsoft. Future PowerCLI for Mac/Linux.
  • Unix to Linux Migration

Tech Previews

  • vCenter Server Appliance migration tool from Windows vCenter
  • vCenter Install, Upgrade and Patching – Enhanced patching (VUM replacement)
  • vCenter High availability – RTO < 10 minutes (active/passive)
  • VM Level encryption – Encrypted VMDKs and configuration files
  • Automation – Predictive capability in DRS. Evolves DRS to use vRealize operations data.
  • Proactive HA – Detects potential host issues and evacuate host prior to failure.
  • Network aware DRS
  • HTML5 Client

New Friends Coming Online

  • vSphere Integrated Containers – Native docker interface, container management portal, container registry.
  • VMware Integrated Open Stack 3.0
  • Photon Platform – Web-scale enterprise container infrastructure. Scales to 1000s of nodes, 1Ms container

VMworld 2016: VM and App Protection

Session: INF8939

4 Step Program for Success

  • Define – Gather requirements – RPO/RTO
  • Research and design – look at various technologies
  • Acquire and implement – Document
  • Test and operate – Continuous testing, continuous research

Disaster recovery and business continuity

  • DR is recovery of data
  • BC is the full business process of recoverying

Define Requirements

  • What are you  trying to protect? apps, VMs, DBs, etc.
  • What are you protecting against? data loss, data corruption, disaster, etc.
  • What is your RPO? zero, minutes, hours, days
  • What is your RTO?
  • How long do I need to keep data? retention policy, archiving, etc.

Protection Tiers

  • Tier 1 – mission critical
  • Tier 2 – Required for longer term business continuity
  • Tier 3 – Nice to have but not required

Tape Backup

  • Cheapest medium
  • RPO of hours to days
  • RTO – depends on how much data
  • Good for archival/long term retention

Hardware Snapshots

  • Snap/restore data in seconds from GB to TB
  • Application consistent storage snapshots – not needed for all VMs
  • Data on primary storage can be expensive

Array Replication

  • Async or sync
  • Only changed data sent
  • Flexible RPO options
  • RTO is based on how data is restored

Site Recovery Manager

  • Integrates with vSphere for site failover
  • Able to test and re-test
  • Requires array integration

Continuous Data Protection

  • Flexible RPO options
  • RTO based on amount of data
  • Only changed data sent


  • May be appliance or software based
  • Most  integrate with traditional backup
  • Integrates with CDP


  • Typically continuous backup so low RPO
  • Backup and recovery limited by bandwidth
  • May have longer recovery times
  • Can take a long time to seed backups

vSphere Metro Cluster

  • Zero RPO/RTO (time to restart apps is  not zero)
  • Great for site protection
  • Layer 2 stretching
  • No application specific backup/restore

Fault Tolerance

  • Limited in supported vCPUs
  • Requires high bandwidth between hosts
  • Does not protect against OS/App failures


VMworld 2016: Extreme Performance: DRS

Session INF8959

300,000 vCenter deployments, 94% with DRS enabled

Quick Facts

  • Faster power on for large clusters: 6.5 is 3x faster than 5.5 and 6.0.
  • 5x lower CPU utilization in 6.5 than previous versions
  • 6.5 has better VM placements

DRS ensures resource availability – DRS does this in two ways

  • Effective initial placement – Use the right host
  • Efficient load balancing – Moving VMs to different hosts
  • DRS collects 20 VM performance metrics and 5 host metrics – CPU ready time, memory swapped, memory active, shared memory, CPU used max, CPU used average
  • Application performance is the primary goal of DRS

Factors Impacting DRS Performance

  • Migration threshold – Left makes it less aggressive
  • Rules — Too many rules may prevent DRS balancing the cluster
  • Reservations, limits, shares – Do not set reservations unless absolutely necessary
  • VM Overrides – Custom DRS settings for a VM.

DRS Faults

  • When DRS tries to fix something, but can’t.

DRS Performance Case Studies

  • Case 1:  How does DRS react to spikes in workload? DRS reacts to spikes and will move loads.
  • Case 2: Does DRS prefer moving heavy or light VMs? DRS prefers to move medium workloads to restore balance faster.
  • Case 3: Why is memory utilization not balanced? DRS considers active memory+ 25% of idle memory.  It will not perfectly “balance” memory across all hosts.


  • Always right size your VMs
  • Occasional swapping is not bad, constant swapping is bad


VMworld 2016: vSphere Encryption Deep Dive

External Threats

  • Nation states, profit motive, highly skilled, social engineering

Internal Threats

  • Snowden.
  • Admins who abuse privileges
  • Physical access to data

VMware’s Vision for security – Secure Access, Secure Infrastructure, Secure Data

VM Encryption Preview

  • Encryption managed via storage policies – Encryption done in ESXi kernel, uses AES-NI, and uses XTS-AES-256.
  • No modification within the guest. VM agnostic.
  • Policy driven. Full support of vMotion and vMotion is encrypted.
  • Uses an external KMS (KMIP compliant)
  • VMDKs are encrypted along with external files such as VMX, snapshots, etc.

Who manages VM encryption?

  • Security admin will manage your KMS and keys
  • Subset of vSphere admins will manage encryption within vSphere

vCenter RBAC has been enhanced for granular encryption control. For example, prevent admins from downloading encrypted VMDKs or opening a console to an encrypted VM.

Key Managers

  • KMIP 1.1 compliant key managers
  • Tested a variety such as Thales, HyTrust, etc.

Key Management Best Practices

  • KMS keys are pushed to all hosts for HA purposes
  • Multiple key managers are supported
  • Expired keys will not be used for new encryption operations. No deep re-encryption needed with new VM key. Shallow re-key operation.
  • No KMS means no booting of encrypted VMs
  • KMS needs to be as reliable as DNS. It must be highly available.

Core Dumps

  • Core dumps are encrypted with a host key
  • Logs are not encrypted
  • You can re-encrypt the core dump with a password (e.g. GSS support needs)
  • Always collect support bundle with a password
  • Uses OpenSSL for core re-keys

Backup, Restore and VM Best Practices

  • SAN mode backups are not supported (use hot-add).
  • No API changes for backup products
  • Backup proxy VM must be encrypted.
  • Backup service account needs cryptographer.directaccess permission
  • Backup data is not backed up encrypted
  • Have a policy in place to re-encrypt a restored VM
  • Backup solution should provide its own encryption solution
  • Don’t encrypt vCenter or your PSCs

Encrypted vMotion

  • 3 modes: Disabled, Opportunistic, Required
  • Configure vMotion encryption from vCenter GUI
  • One-time usage key for each vMotion
  • Set vMotion encryption via PowerShell as well


VMworld 2016: What’s new in PowerCLI

Session INF8092

Note: If you missed it, Microsoft open sourced PowerShell and will release it for MacOS and Linux. VMware will release PowerCLI modules for Linux and MacOS, so you can run the same scripts on all three platforms.

PowerCLI Technical Preview

  • No more snap-ins – Much more simple to install and maintain
  • Horizon View module – Now distributed as a module. Now remote server enabled. Full API support.
  • VSAN commandlets added
  • Cross center vMotion support – vSphere 6.0 and higher
  • Improved Virtual Disk Management – Disks are now first class objects.
  • vSphere DSC (desired stated configuration) module. Luc Dekens wrote a script that takes existing vSphere config state and writes a DSC file.
  • Parallelism – Typical for big jobs. Start-job cmdlet

VMworld 2016: What’s new in Horizon 7

Session EUC8404

Innovations in Horizon 7:

  • Makes Desktop and App management easy
  • Just-in-time desktops – VM fork technology. Drive down storage costs by 30%. Deliver apps instantly. Streamline OpEX by 50%. No more lengthy recompose operations. Faster provisioning, better performance, simplified desktop administration. Significantly lower load on vCenter (1/14th). No refresh, no recompose, no rebalance.
  • Smart Policies – Ties the desktop to the user so you can customize based several attributes. Sets policies like cut/paste, mounting local disks, no local printing, inhibit certain apps from running, change PCoIP profile based on WiFi usage, etc.
  • True SSO – With WorkSpace One you can authenticate with many different methods, a certificate is generated, and logs into Windows without having to provide username/password to Windows.
  • Infrastructure Updates – Cloud Pod Architecture (CPA) – Combines multiple horizon environments into a single federation. PODs managed independently. Uses global load balancing. New in 7 is increased scale (25 pods across 5 sites), more flexible entitlements, improved failover support, integration with VMware identity manager.
  • Access Point – Hardened appliance for external user access. SLES 12 Linux appliance. Pass-through authentication, smart card support, SAML pass-through, RADIUS/SecureID support. Blast over port 443. Enhanced health status.
  • Automation Improvements – Integrated scripting for Horizon. Integrated with VMware PowerCLI for an end-to-end experience.
  • User Experience – New remoting protocol (Blast Extreme). PCoIP will be supported into the future. Blast uses H.264, GRID optimized, uses less bandwidth, TCP or UDP, adapts better to lossy networks, better battery life.
  • Blast Extreme and PCoIP feature parity – Most zero clients can’t support the Blast protocol. But same common feature rich user experience. You do not lose any client features using Blast Extreme.
  • WAN Optimization – When operating in TCP mode, the SteelHead appliance can perform optimization.
  • vRealize Operations for Horizon with Blast Extreme – Stats like RTT, FPS, throughput, etc.
  • Horizon 3D Graphics Update – Supports 4K resolution up to 4 monitors. Supports K1, K2, M6, M10, M60 cards.
  • RDS Hosted apps have significantly evolved to support a lot of use cases and closed most of the gaps with Citrix

VMworld 2016: Architecture Future of Network Virtualization

Session: NET8193R. Bruce Davie, CTO Networking

Software developers need to be treated as a first class customer. The developer is king.

Network virtualization is the bridge to the future.

Network architecture today: Data plane, control plane, management plane, cloud consumption. Distributed data plane, centralized control.

Management Plane Availability

  • Developers need access to the management plane and it needs higher availability than in years past
  • New: The scalable persistence of memory
  • Write and read scalability
  • Durability
  • Shrink-wrapped
  • Consistent snapshots
  • Atomic transactions
  • Driving innovation: Distributed, shared log – No single point of failure

Control Plane Evolution

  • Heterogeneity – Hypervisors, gateways, top-of-rack switches, public cloud workloads, containers
  • Scalability – Thousands of hypervisors, 10,000s of logical ports
  • Central control plane – Generalized instructions that doesn’t need to understand heterogenity
  • Local control plane – Hypervisor specific controls (vSphere, KVM, Hyper-v, AWS, Azure, etc.)

What about non-virtualized workloads?

  • NSX has solutions for this problem

High-performance Data Plane

  • x86 processors can forward hundreds of millions of packets a second
  • DPDK – Data path development kit from Intel.
  • active-active edge cluster
  • Active-hot-standby for stateful services

Takeaway: Developers are key, and need to make them successful.

Extend NSX to the public cloud – VMware is starting with AWS

Network virtualization for containers – Put a vSwitch in the guest OS