TechEd 2014: Building a web-scale Private Cloud

Session: DCIM-B395 Building a Web-Scale Private cloud (without losing your cool) by Steve Poitras, Nutanix (@StevenPoitras). This was a good session showing how the combination of the Windows Azure pack, Hyper-V, and Nutanix enable large scale-out private clouds. You can check out the full video and slide deck of the session here on Channel 9. Steve had a lot of diagrams and good tables that you should check out, which aren’t captured in my session notes.

Why are we here today?

  • Web-scale cloud is here
  • IT needs to be much more agile
  • Users want services on demand and IT needs to keep up with it
  • We want to spend time doing other things, like value add services instead of supporting legacy services


  • Define web-scale
  • Focus is architecting a web-scale solution using Hyper-V on the Nutanix platform

Why web scale?

  • Inherent complexity in legacy infrastructure
  • Lack of availability
  • Unpredictable scaling
  • What’s in with web scale: Smart software, commodity hardware, local attached storage, highly distributed software, scale one x86 server at a time, heavily automated and resilient
  • What’s out: Customized hardware (ASICs), islands of resources, unused overprovisioned systems, frequent north-south traffic, lots of manual intervention

Web-scale made possible with Nutanix

  • Nutanix was founded in 2009
  • Shipping fourth generation operating system (NOS 4.0)
  • Hundreds of customers in a variety of vertical markets
  • Built for massive scale (3 to thousands of nodes)
  • Uses high performance commodity hardware
  • The secret sauce is the software (controller VM) which services all I/Os and enables web-scale functionality
  • Supports Hyper-V 2012 R2, ESXi and KVM

Nutanix under the Covers

  • Hypervisor sees the Nutanix distributed file system (NDFS) as one or more SMB 3.0 file shares
  • Supports features like snapshots, dedupe, web-scale out, and disaster recovery
  • Locally shared storage is comprised of both flash and spinning disks
  • Variety of models (compute heavy, storage heavy, etc.)
  • Mix and match models within the same cluster




The diagram below shows a multi-tenant example using Nutanix and the Windows Azure pack. The solution is modular, scales, is Microsoft Fast Track certified, and Nutanix published a full Reference Architecture guide which will be downloadable in the near future.


Steve then goes on to describe various VM classifications (light, medium, heavy) and shows how many VMs can fit on various Nutanix models. He then describes one of the service pods, and that it looks like. Steve then goes into more depth on pool design and multi-availability zone deployments and shows how the solution can scale out with Nutanix is the foundational building block. Again, for the full slide deck and details check out the MSDN Channel 9 video here. The session provides a good overview how the Azure pack + Nutanix = highly web-scale like infrastructure for your private cloud.




TechEd 2014: Hunting Malware with Sysinternals Tools

Session DCIM-B368: Malware hunting with Sysinternals Tools. This was a great session by Mark Russinovich on how to use his Sysinternals tools to find and rid your system of malware. He had a number of demos showing exactly how his tools find, and then can be used to remove the malware. Check out the Channel 9 video to see all of his great demos.


  • The top 4 AV products detect less than 40% of all malware
  • Malware cleaning steps: Disconnect from network, identify malicious process/drivers, terminate identified processes, identify and delete malware autostarts, delete malware files, reboot and repeat.

Identify Malware Processes

  • Investigate process that: have no icon, have no description or company name, packed, live in user profile, open TCP/IP endpoints, suspicious DLLs
  • Don’t use task manager, use Process Explorer
  • A lot of malware uses randomly generated names
  • “Search online” in process explorer is not that useful these days
  • Pink processes host Windows services (background processes). Blue processes run as the user.
  • Cyan color is Metro apps
  • red/green show processes that are launched and terminated
  • Packed executables are shown in purple. Packed can mean compressed or encrypted
  • Add the “verified signers” column to the display view

Image Verification

  • Most all Microsoft code is digitally signed
  • New: VirusTotal Integration into Process Explorer
  • Add “virus total” column
  • Sigcheck -e -u -vr -s  c:\ (file versioning tool)
  • Strings: check the memory image for suspicious strings

Terminating Processes

  • First put the process to sleep (suspend), then terminate it

Cleaning Autostarts

  • Use “autoruns”
  • Tell autoruns to only show images not signed by Microsoft
  • Malware has started using WMI to start processes

Tracing Malware Activity

  • If in doubt run Process Monitor
  • Filtering is the key to using process monitor
  • Category is “write” is the best filter

TechEd 2014: Software defined storage in WS2012 R2

Session: DCIM-B349. Software defined storage with Windows Server 2012 R2 and System Center 2012 R2. This was a jam packed session with tons of content on each slide. Great in-depth talk about what’s new in the 2012 R2 wave which came out last year. I only captured 25% of the slide content below, so be sure to check out the Channel 9 video and slide deck when they get posed, for all the goodies.

Storage Enhancements

  • New approach to storage: File based storage (SMB3) other Ethernet networks. Cost effective storage.
  • Faster enumeration of SMI-S storage providers
  • Virtual Fibre Channel integration in SC 2012 R2
  • SC can now leverage ODX for fast VM copy operations
  • Investments in Fibre Channel switch discovery and pulling that into VMM. Shows a demo of creating a FC zone in VMM. Also shows provisioning a LUN from with VMM from a Fibre Channel array. You can configure a LUN in a service template, so all VMs get access to the LUN.

Focused Scenarios for 2012 R2 Wave

  • Reducing CAPEX and OPEX

Infrastructure-as-a-Service Storage Vision

  • Dramatically lowering the costs and efforts of delivering IaaS storage services
  • Disaggregrated compute and storage – Independent management and scale at each layer
  • Industry standard servers, networking and storage – Inexpensive networks, inexpensive shared JBOD storage
  • Microsoft is heavily investing in the SMB protocol and will use this going forward as the basis of storage
  • Overall objective is to reduce cost. The cheapest storage is the storage you already own.
  • Ability to use “Spaces” with low cost JBOD
  • Ability to manage the full solution within System Center

Storage Management in System Center 2012 R2

  • Insight, Flexibility, Automation
  • Storage Management API (SM-API)
  • New architecture for 10x faster enumerations
  • Capacity management, scale-out-file-server, and a lot more

Guest Clustering with shared virtual disks

  • Guest clustering with commodity storage
  • Sharing VHDX files
  • VM presented a shared virtual SAS disk

iSCSI Target Server

  • VHDX support
  • Support up to 64TB LUNs
  • Dynamically grow LUNs

File Based Storage

  • SMB Direct support (uses RDMA)
  • 50% improvement for small IO workloads

Scale out File Server

  • SMB session management for back-end IO distribution

Live Migration

  • SMB as a transport for live migration
  • Delivers performance using RDMA – so no CPU hit on the host
  • Adds compression (75% faster)

SMB Bandwidth Management

  • Restrict bandwidth for different workloads (e.g. file copy, live migration, storage access)

Data Deduplication

  • Can dedupe open files – VDI scenarios is a good use case
  • Good for high reads, low write VHDXs
  • Added support for CSV

Storage Spaces

  • Optimized data placement – Pool consists of both HDDs and SSDs with automated tiering
  • Write-back cache – Smooths out workload IOPS

TechEd 2014: Network tuning for specific workloads

Session: DCIM-B344, Network Turning for Specific Workloads. This was a great session, with a ton of Q&A during and after the main presentation was over. I’d highly encourage you to watch the full video on Channel 9 when it is uploaded to get all of the goodies. The session notes below are a small fraction of the gold nuggets that were discussed in the session. Confused about VMQ, RSS, vRSS, SMB multi-channel performance, virtual switches, NIC teaming and when to use what feature? Be confused no more after watching the video.


  • Socket is a NUMA node, and within the node is a core. On the core you have logical processors (with hyper-threading), on which you have virtual processors for VMs.


Problem 1: Enterprise physical web server and file server. Large volume of incoming packets, but one core is highly utilized.

Solution: Enable RSS on the server. RSS is for physical servers only. NIC spreads the network traffic by TCP/UDP flows across different cores to enhance performance and balance processor utilization.

Problem 2: A VM is deployed and the incoming packet processing is saturating a limited set of cores.

Solution: Virtual machine queue. VMQ spreads traffic per vNIC. RSS is disabled on the pNIC when a virtual switch is defined. A single core is bottlenecked at 4-5 Gbps of traffic, depending on processor speed. VMQ is enabled by default, so no manual configuration is needed. Number of queues depends on the physical NIC properties. New NICs have more queues (64+ not uncommon).

Problem 3: A VM has a large number of incoming packets, such as a web server. The workload is limited to using one vCPU. This is only for VMs with >3 Gbps of traffic. Less traffic can be serviced by a single core without any additional configuration.

Solution: vRSS can be used on WS2012 R2 VMs. This spreads traffic across multiple vCPUs. Flows are moved if a CPU has 90% or higher utilization. MS states they have seen line rate up to 40Gbps to a VM using vRSS with a 40 Gbps NIC. vRSS must be manually enabled inside of the VM.

Problem 4: A highly latency sensitive application, such as high speed financial trading.

Solution: Use SR-IOV. Bypasses the virtual switch, and directly connects the VM to the hardware NIC. Only for use with trusted VMs, since switch security is bypassed. Rarely used, but available for these very limited cases.

NIC Teaming

Windows Server 2012 R2 has a new dynamic NIC teaming mode. Continuously monitors traffic distribution. Actively adjusts traffic based on observed load. Download the Windows Server 2012 R2 NIC teaming guide here.

TechEd 2014: What’s new in Exchange

Microsoft TechEd 2014 is in full swing and this is the first general session that I’m attending. It will cover what’s new in Exchange, both in Office 365 and on-prem. As usual during conference this is live blogging, directly from content presented in the session. So editing is minimal, and please excuse any grammar issues.

Email Challenges

  • Too much stuff in my inbox
  • Important emails get buried
  • People keep sending documents as attachments
  • Hard to keep up with legal requirements
  • Need a better way of sharing data

Historically major Exchange releases are every three years. But the cloud is changing that. The same Exchange team handles both Office 365 and on-prem code bases. Microsoft is all about Cloud First delivery, with Office 365. Features continuous innovation and rapid feedback. Eventually everything in Exchange cloud will end up in Exchange on-prem. These come as cumulative updates and service packs, plus major new releases (which will be in 2015). Major new on-prem releases will still be every ~three years.

Future of Exchange

Email will have to be more social, more open, smarter.

Security and Compliance

  • Compliance Center
  • S/MIME in OWA
  • DLP document fingerprinting
  • Office 365 message encryption
  • DLP Policy tips in OWA for devices

Compliance center features across Exchange, SharePoint and Lync from one location. Targeted for end of calendar year 2014. eDiscovery, auditing, data loss prevention, retention policies, retention tags and journal rules. Demo showed document fingerprinting. You upload a sample file, such as a contract, then define rules. For example, set different rules for sending within the organization, or externally (with more restrictions). Demo continued to show a DLP policy tool tip when a contract was attached to an email, and this is now available in OWA.

Two new features: DLP for data at rest, and bring DLP to SharePoint (no dates mentioned).

Work Smarter

  • “Clutter” view
  • People View
  • Outlook web app search enhancements
  • Document collaboration
  • outlook web app rich content
  • App enhancements

Demo shows a ‘clutter’ button that filters in the inbox view to show only emails that need to be filtered. It will learn as you delete or otherwise mark emails. Demo also showed off a People view, where he clicked on a person and the mailbox view was filtered to show only emails from that person. The view learns which people are most important and only shows them. Demo shows easier to use search, where it shows search suggestions when you start typing in the search field. It also lists search refiners in the left pane when results are shown, to further narrow down the results.

Document Collaboration

Making attachments smarter. Deep integration with OneDrive for business. Send an attachment as a OneDrive for business link. Access it anytime, anywhere from any device. Edit attachments and reply in a single process. View the attachment in side-by-side view with the email. Multiple people can edit the same document. When attaching a file it will now allow you to send the whole file, or automatically upload to OneDrive and just send a link. It will then be automatically shared from OneDrive. Demo also shows the ability to drag and drop images into an OWA email (no more manual attaching), plus the ability to create tables and in-line preview of links in emails.

Social Email

  • Groups
  • Group email experience
  • Group calendar experience

One Groups System: One identity system (Azure Active Directory) is the master for group identity and membership across Office 365. Covers Exchange, SharePoint and Yammer (Lync in the future). Demo shows OWA, and a new Groups section on the left. Groups can be public or private. Shows a threaded conversation, with inline previews of web links. Also shows the side-by-side view of previewing docs such as a spreadsheet. The group conversation view looks very much like Yammer, and in fact you can view the conversation in yammer. A very seamless experience. Demo shows a group calendar and side-by-side view with a person’s own calendar. Demo also shows calendar search, which is new. Demo shows the GUI experience to create a new group and adding members. They also demoed the group experience from an Android phone, which looked like the OWA experience.

Additional information


This was a good high level session, which showcases the rapid feature development of the Exchange platform. All new features first debut in Office 365, then on a much more measured pace make it into the on-prem version of Exchange. Seeing the rapid feature enhancements and deep integration with Yammer, SharePoint, OneDrive, and Lync really provide a compelling story when compared to other hosted services such as those offered by Google. Gmail innovation and seamless integration with Office products just isn’t in the same ballpark.