VMware Tools 10.2.0 Released

Hot off the press is a new version of VMware tools, 10.2.0. This has a number of bug fixes, and an updated OS support list. For security geeks, VMware tools finally supports ASLR (Address Space Layout Randomization). There’s also a Windows 10 fall creator’s (1709) fix tucked in there as well. You can find the full release notes here.

One cool feature of this release, is a VIB for vSphere 5.5, 6.0 and 6.5. This lets you update the ‘baked in’ tools of the ESXi hypervisor. This VIB can be pushed via VMware Update Manager (VUM).

Direct downloads are available here.

Windows 10 Credential Guard and VMware Workstation 14

Microsoft has been very busy adding new security features to Windows 10. It seems that each release gets something new, or existing features are enhanced. For enterprises, one of the great new-ish features is Windows Defender Credential Guard. What is Credential Guard? It uses VBS (virtualization based security) to help mitigate pass-the-hash or pass-the-ticket attacks. I wrote a how-to blog article many years ago on how one can ‘root’ your Windows 7 PC and ultimately compromise your whole network including domain controllers. It was scary easy. Windows 8 was supposed to make it harder, but Windows 10 with fall creator’s update (1709) has really raised the bar.

But until the release of VMware Workstation 14, you couldn’t easily test these new features in a virtual environment. However, Workstation 14 has explicitly added support for VBS in hardware v14, and the UEFI firmware supports secure boot. This now allows one to test Windows Defender Credential Guard. The whole process is fairly easy. But a few requirements must be met: 1) VMware Workstation 14 (or later) 2) Windows 10 Enterprise edition (no home/pro) 1709 (Fall Creator’s update) 3) Physical host that is modern enough to support virtualization extensions.

Workstation 14 Credential Guard Configuration

Let’s get started with configuring the VM hardware on Workstation 14 to appropriately support VBS and secure boot.  I’ll assume you are familiar with Workstation basics. VM size just for basic testing can be 1 vCPU and 2GB of RAM.

  1. Create a new virtual machine using the custom (advanced) wizard.
  2. Select hardware compatibility: Workstation 14.x
  3. Select ‘I will install the operating system later’
  4. Select ‘Microsoft Windows 10 x64’ guest operating system
  5. Select a VM name and location that you desire
  6. Select UEFI and secure boot firmware type
  7. Choose your processors/core that you desire
  8. Choose the memory configuration you desire
  9. Choose the network connection type you desire
  10. Leave the SCSI controller type and virtual disk type
  11. Create a new virtual disk
  12. Allocate sufficient storage and split as needed
  13. Chose a disk file name that you desire
  14. Click Finish
  15. Edit the VM settings and click on the Options tab
  16. Click on Advanced and check the box next to Enable VBS

Now that your VM hardware is properly configured, next, install Windows 10 Enterprise Edition 1709. I won’t go through that process, as there’s nothing special to do until it’s fully installed and you have a desktop. Once you have a desktop, come back to this post and resume the configuration. Be sure to grab the latest VMware tools, which has updates for Windows 10 Fall creator’s update, here.

Windows 10 Credential Guard Configuration

1. Press the Windows key and type system information.
2. Scroll down on the summary page and look at Virtualization-based security. It should show not enabled.
3. Press the Windows key and type features.
4. Scroll down to Hyper-V, Hyper-V Platform, and check Hyper-V Hypervisor.

5. Wait for the feature to be added, but do NOT reboot.
6. Open gpedit.msc. Navigate to Computer Configuration, Administrative Templates, System, Device Guard.
7. Enable the Turn on Virtualization Based Security policy.
8. Select the options below, or enable UEFI lock for a production environment to prevent remote manipulation of these settings. You can also turn on the UEFI memory attributes table if you wish.

9. Close gpedit and do a gpupdate /force from the command line.
10. Restart the VM. Open System Information and on the summary page scroll down to the very bottom. Verify virtualization-based security is running.

Summary

As you can see from this post, enabling Windows Defender Credential Guard is pretty easy. Workstation 14 supports it out of the box. VBS is a new feature of Hardware Version 14, which vSphere 6.5 does not support. So any support for VBS would come in a future vSphere update. Workstation often foreshadows upcoming vSphere features, so I wouldn’t be surprised to see it in the next version.

VMworldl 2017: What’s new in storage

Session: SER1317BU

Faster storage need faster networks
-10/25/40 NICs are now the norm
-Protocol stack needs to be upgraded with new storage protocols
-Performance of AFA depends on network connection with low latency
-32Gbps FC is shipping, 64Gbps is coming

NVMe – A logical device interface to NVM devices
-PCIe is the physical interface and NVMe is the protocol
-up to 64K queues and 64K queue depth
-All major OSes support NVMe

NVMeoFabric
-Allows large number of external NVMe drives into external storage
-Aims for no more than 10 microseconds latency overhead compared to local NVMe

vSphere 6.5 Features
-VMFS 6.0
*meta data is 4K aligned
*supports 64-bit addressing
*NO in-place upgrade from VMFS 5.0. See KB 2147824

Automatic Unmap in 6.5
-Automatic unmap support when VM is deleted/migrated
-Space reclimation requests from guest OS which supports UMAP
-Only automatic unmap on arrays with UNMAP granularity LESS than 1MB
-Background impact is minimal: set to 25MB/sec max
-Future: Possibly throttle/accelerate UNMAP rate based on array load

High Capacity Drives in 6.5
-Support 512e drives
-Requires VMFS 6
-vSphere 6.0 supports physical mode RDMs mapped to 512e drives
-FAQ: KB2091600

New Scale Limits
-512 LUNs & 2048 paths
-If using 8 paths per LUN, you can now have 256 LUNs

NFS 4.1 Plug-in and strong crypto & HW acceleration support
-NFS 4.1 supported since 6.0
-HW acceleration (VAAI) now supported
-Stronger crypto with AES
-Supports IPv6
-Better security with NFS 4.1

Virtual NMVe in 6.5
-NVMe 1.0 device emulation
-Hot add/remove
-Multi-Q support – 16 queues with 4K depth

VMworld 2017: Advanced ESXi troubleshooting

Session: SER2965BU

Note: This session had a number of log examples and what to look for. Review the session slides for all the details. EXCELLENT session!

Goal: 7 log files, 7 ESXi commands, 7 config files for enhanced troubleshooting

7 Important Log Files

-Host abruptly rebooted – vmksummary.log
-Slow boot issues – /var/log/boot.gz .  You can also enable serial logging (Shift + o)
-ESXi not responding – hostd.log & hostd-probe.log
-VM issues – vmware.log
-Storage issues – vmkernel.log
-Network and storage issues – vobd.log
-HA issues – fdm.log   /opt/vmware/fdm/prettyprint.sh hostlist | less

7 ESXi Commands
-Monitor & configure ESXi – esxcli
-VMkernel sysinfo shell command – vsish get /bios; /hardwareinfo;


-Manage ESXi * VM config – vim-cmd

-VMFS volumes & virtual disks – vmkfstools
-Detailed memory stats – memstats
-network packet capture – pktcap-uw
-monitoring – esxtop

7 Configuration Files
-/etc/vmware/esx.conf – storage, networking, HW info
-/etc/vmware/hostd/vminventory.xml – VM inventory
-/etc/vmware/hostd/authorization.xml – vCenter to ESXi host connection
-/etc/vmware/vpxa/vpxa/cfg – vCenter and ESXi connectivity
-/etc/vmware/vmkiscsid/iscsi.conf – iSCSI configuration
-/etc/vmware/fdm – HA config
-/etc/vmware/license.cfg – license configuration

VMworld 2017: DR with VMware on AWS

Session: MMC2455BU, GS Khalsa

Legacy (physical) DR solutions are not adequate – Long RTOs, lots of surprises, unreliable
vSphere is an enabler for DR – consolidation, hardware independence, encapsulation (VM is a file)

Long distance DR solutions with async replication
-Active/passive
-Active/Active
-bi-directional failover
-Shared site recovery

Metro DR Solutions with sync replication
-Availabiity – Zero RPO/RTO
-Mobility – active/active datacenters
-Disaster avoidance

DR to the cloud with AWS
-Co-located DR costs are high
-DR to the cloud is less expensive

VMware Cloud on AWS
-Managed SDDC stack running on AWS
-Consistent operational model enables hybrid cloud
-Leverage cloud economics
-Goals of DR: Deliver as a service, build on VMware (SRM, vSphere replication, etc.)
-Working on flexible SRM pairing – Decouple on-site upgrade from VMC/AWS
-Loosening version dependencies across vCenter, SRM & vSphere Replication releases
-Working on major UI improvements – HTML5 and “clarity” UI standard
NEW: SRM Appliance based on photon OS

GS then shows a number of video demos showing the full SRM configuration, setup, and failover process. Anyone familiar with SRM will be accustomed to the same workflow, but with a nice new coat of paint on the GUI.

 

 

 

VMworld: PowerCLI What’s New

Session: SER2529BU Alan Renouf

PowerCLI Overview
-623 cmdlets and counting
-PowerCLI is 10 years old
-Name change – VMware PowerCLI
-Move-VM now includes cross vCenter vMotion
-Automate everything with VSAN
-Independent disk management cmdlets – new-vdisk, get-vdisk, copy-vdisk, move-vdisk
-VVOL replication cmdlets
-New Horizon View module
-SPBM cmdlets
-More inventory parameters
-DRS cluster groups and VM/host rule cmdlets

Install: install-module VMware.PowerCLI

Release Frequency
-Less features, but more often
-Less wait on bug fixes
-Focused on your input

PowerCLI 6.5.2
-New ‘inventoryLocation’ parameter – move-vm, import-vapp, new-vapp
-Mount a content library ISO with new-CDDrive
-Fixes and enhancements

Multiplatform Fling
-Photon OS, Mac OS, Linux, Docker

VMware Cloud on AWS?
-Works exactly the same as on-site vCenter

Endless Possibilities
-Content library – more cmdlets to come
-Parameter auto-complete
-vSphere REST API high-level cmdlets
-Powershell DSC (desired state config) – Chef, Puppet, Ansible, Saltstack
-New vSphere Client and Rest API support for Onyx (automated code generator)
-PowerCLI multiplatform 6.0

Community Projects
(FREE) OpBot – Connects vCenter to slack. Download: http://try.opvizor.com/opbot
(NEW!!) PowerCLI Feature request page: https://vmwa.re/powercli

 

VMworld 2017: Architecting Horizon 7 & Apps

Session: ADV1588BU

Note: This session had a multitude of complex architecture diagrams which I did not capture. See the session slide deck, after VMworld, for all the details.

Why? –Business objective/drivers
How? Meet requirements
What? Design and build
Deliver Build and integrate
Validate Met requirements?

Design Steps

  1. Business drivers & use case definition
  2. Services definition
  3. Architecture principles and concept
  4. Horizon 7 component design
  5. vSphere 6 design
  6. Physical environment design
  7. Services integration
  8. User experience design

Use a repeatable model when scaling up:

 

Physical Environment Considerations
-AD
-GPO
-DHCP
-Licensing

Identity Management

Profiles and User Data
-Folder redirection
-Mandatory profile
-User environment manager

AppVolumes
-AppStack replication
-Single site or multiple site
-Use writeable volumes very sparingly

VMware Horizon Apps

Speaker goes over a highly detailed reference architecture with lots of complex slides. And he goes over the LoginVSI setup, both hardware and software.

VMworld 2017: vSphere 6.5Host Resources Deep Dive Pt. 2

Session: SER1872BU Frank Denneman, Niels Hagoort

Note: This was a highly technical session with lots of diagrams. Best bet is to get Frank and Niel’s book for all the details.

Compute Architecture: Shows a picture of a two NUMA node server. Prior to Skylake processors, two DIMMSs per memory channel are optimal. Skylake processors increased the number of memory channels and have a maximum of 2 DIMMS per channel.

QPI Memory performance: 75ns local latency, but 132ns latency to other NUMA node

Quad channel local memory access: 76GB/s. Remote access will be noticeably slower.

vNUMA exposes the physical NUMA architecture to a VM. vNUMA ‘kicks in’ when a VM has more than 8 vCPUs and if the core count exceeds the physical CPU package. ESXi will then evenly split the vCPUs across the two physical CPU packages.

If you use virtual socks, mimic the physical CPU package layout as much as possible. This allows the OS to optimally manage memory and the cache.

“PreferHT” can be useful, see KB 2003582. This forces the NUMA scheduler to count hyperthreads as cores. Use this setting when a VM is more memory intensive vs. CPU intensive.

What if the vCPUs can fit in a socket, but VM memory cannot? numa.consolidate=FALSE can be useful.

One AHCI storage IO needs 27K CPU cycles. If you want a VM do do 1M IOPS, you need 27GHz of CPU power.

NVMe 1 I/O needs 9.1K CPU cycles, which is vastly less than AHCI storage.

With 3D crosspoint, it can max I/O performance at a very low queue depth. This makes it quite useful as a caching tier in vSAN.

CPU Utilization vs. Latency

Workload latency sensitive? No, then tune CPU for power savings. Yes, then tune for lowest latency. SAP HANA, for example, could benefit from low latency.

Interrupt coalescing, is enabled by default on all modern NICs. This can increase packet latency. You can increase ring buffers by using KB2039495, which can help with dropped packets.

Polling vs. interrupts

Pollmode driver (DPDK) can optimize network I/O performance.

Low CPU utilization = higher latency
Higher CPU utilization = lower latency

vSphere 6.5 as vRDMA, which can significantly boost network throughput.

VMworld 2017: Virtualizing AD

Session: VIRT1374BU: Matt Liebowitz

AD Replication
-Update sequence number (USN) tracks updates and are globally unique
-InvocationID – Identifies DC’s instance in the AD database
-USN + InvocationID = Replicable transaction

Why Virtualize AD?
-Fully supported by Microsoft
-AD is friendly towards virtualization (low I/O, low resource)
-Physical DCs waste resources

Common objections to virtualizing DCs
-Fear of stolen vmdk
-Privilege escalation – VC admins do not need to be domain admins and vice versa
-Must keep xx role physical – no technical or support reason. Myth
-Timekeeping is hard in VMs

Time Sync
-VM guest will get time re-set with vMotion and resuming from suspend. If there’s a ESXi host with bad time/date, it can cause weird “random” problems when DRS moves DCs around.
-There’s a set of ~8 advanced VMX settings to totally disable time sync from guest to ESXi host. Recommended for AD servers. See screenshot below.

Virtual machine security and Encryption
-vSphere supports VMDK encryption
-Virtualization based security – WS2016 feature – supported in future vSphere version

Best Practices

Domain Controller Sizing
USN Rollback
Happens when a DC is sent back into time (e.g. snapshot rollback)
-DCs can get orphaned if this happens since replication is broken
-If this happens, it’s a support call to MS and a very long, long process to fix it

VM Generation ID
-A way for the hypervisor to expose a 128-bit generation ID to the VM guest
-Need vSphere 5.0 U2 or later
-Active Directory tracks this number and prevents USN rollback
-Can be used for safety and VM cloning

Domain Controller Cloning
-Microsoft has an established process to do this, using hypervisor snapshots.
-Do NOT hot clone your DCs! Totally unsupported and will cause a huge mess.

VMworld 2017: Extreme Performance

Session: SER2724BU

Performance Best Practice Guide for vSphere 6.5 guide is now out. Download now!

Baseline best practices
-Use the most current release
-HW selection makes a difference
-Refer to best practice guides
-Evaluate power management
-Rightsize your workloads
-Keep hyperthreading enabled
-Use DRS to manage contention
-Do NOT use resource pools – more harm than good
-Monitor oversubscription
-Use paravirtualized drivers

Monitoring
-Compute: Contention – CPU ready, co-stop
-Memory: Oversubscription – balloon, swap
-Storage: Service time – device and kernel latency

vNUMA
-Poor NUMA locality (N%L)
-pNUMA does not match vNUMA
-VM config should match physical topology (don’t make wide VMs)
-Don’t create a VM with a larger vCore count than pCores

Keep things up to date
-Virtual hardware can make a performance difference
-38 changes were made in vHW 11 alone
-Use latest vHW

Power Management
-New in 6.5 is %A/MPERF in ESXtop to see power management. Over 100% means turbomode.
-“Balanced” mode allows turbomode
-Always set BIOS to “os controlled”
-High performance caps turbo opportunity – good for large VMs – required for latency sensitive workloads
-“high performance mode” should be used for benchmarking since it results in the most stable results

Hyper-threading
-25% more performance, approximately
-Latest processes may be higher performance