VMworld 2013: Software Defined Storage the VCDX Way

This was a great “put your architecture cap on” session by two well known VCDX’s, Wade Holmes and Rawlinson Rivera. Software defined <insert virtualization food group here> is all the rage these days. Be it SDN (networking), SDS (storage), SDDC (datacenter) or software defined people. Well maybe not quite at the people stage but some startup is probably working on that.

Given the explosion of SDS solutions, or those on the near horizon, you can’t put on your geek hat and just throw some new software storage product at the problem and expect good results. As an engineer myself “cool” new products always get my attention. But an IT architect has to look at SDS from a very different perspective.

This session gave an overview of the VCDX Way for SDS. I took a different approach to this session’s blog post from most other ‘quick publish’ VMworld session notes. Given the importance of SDS and the new VMware products, I’ve made this post a lot longer and tried to really capture the full breadth of the information presented by Rawlinson and Wade.

Session Introduction

How do you break down the silos in an organization? How do you align application and business requirements to storage capabilities? In the “old” days you matched up physical server attributes such as performance, high availability and performance to a specific workload. Big honking database servers, scale out web servers, or high IOPS email systems.

In the virtual era you gained flexibility, and can better match up workloads to pools of compute resources. Now it is much easier to implement various forms of high availability, scale-out performance, and greatly increase provisioning speed. But some subsystems like storage, even with tools like storage IO control, VASA, and storage DRS were blunt instruments trying to solve complex problems. Did they help? Absolutely, are they ideal? Not at all.

The final destination on the journey in this session is software defined storage (SDS). The remainder of this session covered the “VCDX Way” to SDS. This methodology enables efficient technology solution design, implementation and adoption to meet business requirements. I’ve heard from several people this week the array of storage solutions is nearly bewildering and so following the methodology can help you make your way through the SDS maze and ultimately be very successful in delivering solid solutions.


  • Gather business requirements
  • Solution Architecture
  • Engineering specifications
  • Features: Availability, Manageability, Performance, Recoverability, Security

Software-Defined Storage

Software defined storage is all about automation with policy-driven storage provisioning backed by SLAs. To achieve this storage control logic is abstracted into the software layer. No longer are you tied to physical RAID sets, or using blunt instruments like a VMFS datatore to quasi match up application requirements with performance, availability, and recovery requirements.

The control plane needs to be flexible, easy to use and automatable like crazy. The presentation slide below shows Storage Management with SDS of “tomorrow”. At the top level is the policy-based management engine, better known as the control plane. Various data servers are then offered, such as replication, deduplication, security, performance, and availability. In the data plane you have the physical hardware which would be a traditional external storage array, or the new fangled JBOD scale-out storage tier.

software defined storage

Three Characteristics of SDS

  • Policy-Driven control plane – Automated placement, balancing, data services, provisioning
  • App-centric data services – Performance SLAs, recoverability, snapshots, clones, replication
  • Virtualized data plane – Hypervisor-based pooling of physical storage resources

Solution Areas – Availability

Availability is probably one of the first storage properties that pops to mind for the average IT when you think about storage. RAID level and looking at the fault domain within an array (such as shelf/cage/magazine availability) are simple concepts. But those are pre-SDS concepts that force VMs to inherit the underlying datastore and physical storage characteristics. The LUN-centric operational model is an operational nightmare and the old way of attempting to meet business requirements.

If you are a vSphere administrator then technologies such as VAAI, storage IO control, storage DRS, and storage vMotion are tools in your toolbox to enable meeting application availability and performance requirements. Those tools are there today for you to take advantage of, but were only the first steps VMware took to provide a robust storage platform for vSphere. You also need to fully understand the fault domains for your storage.

Take into account node failures, disk failures, network failures, and storage processor failures. You can be assured that at some point you will have a failure and your design must accommodate it while maintaining SLAs. SDS allows the defining of fault domains on a per-VM basis. Policy based management is what makes VM-centric solutions possible.

Instead of having to define characteristics at the hardware level, you can base it on software. VM storage profiles (available today) is an example of a VM-centric QoS capability. But those are not widely used. Think about how you scale a solution and the cost. Cost constraints are huge, and limit selection. Almost nobody has an unlimited budget, you carefully need to initial capital costs, as well as future expansion and operational costs.

Solution Areas – Management

Agility and simplified management are a hallmark of SDS, enabling easy management of large scale-out solutions. The more complex a solution is, the more costly it will be over the long term to maintain. In each release of vSphere VMware has been introducing building blocks for simplified storage management.

The presenters polled the audience and asked how many were using VASA. Only a couple of people raised there hand. They acknowledged that VASA has not seen wide adoption. In the graphic below you can see VMware’s progression from a basic set of tools (e.g. VASA 1.0), to the upcoming VSAN product (VASA 1.5), to the radically new storage model of vVOLs (VASA 2.0). No release data for VVOLs was mentioned, but I would hope they come to fruition in the next major vSphere release. VSAN is a major progression in the SDS road map, and should be GA in 1H 2014.

software defined storage management

The speakers ran through the VSAN VM provisioning process, and highlighted the simple interface and the ability to define on a per-VM level the availability, performance and recoverability characteristics you require. As stated earlier, we are now at the stage where we can provide VM-centric, not datastore or LUN centric, solutions. Each VM maintains its own  unique policies in the clustered VSAN datastore.

Management is not just about storage, but about the entire cloud. Think about cloud service provisioning which is policy-based management for compute, networking and storage resources. Too many options can become complex and difficult to manage. Personally, I think VMware still has room for improvement in this area. VSAN, Virsto, vVOLS, plus the myriad of third-party SDS solutions like PernixData, give customers a lot of options but can also be confusing.

Solution Areas – Performance

Clearly storage performance is a big concern, and probably the most common reason for slow application performance in a virtualized environment. Be it VDI or databases or any other application key performance indicators are  IOPS, latency and throughput. Applications have widely varying characteristics, and understanding them is critical to matching up technologies with applications. For example, is your workload read or write intensive? What is the working set size of the data? Are the IOs random or sequential? Do you have bursty activity like VDI boot storms?

With VMware VSAN you can reserve SSD cache on a per-VM basis and tune the cache segment size to match that of the workload. These parameters are defined at the VM layer, not a lower layer, so they are matched to the specific VM workload at hand. VMware has recently introduced new technologies such as Virsto and Flash Read Cache to help address storage performance pain points. Virsto helps address the IO blender effect by serializing writes to the back-end storage, and remove the performance penalty of snapshots, among other features. 20130829_124601 The VMware VSAN solution is a scale-out solution which lets you add compute and storage node in blocks. There were several sessions at VMworld on VSAN, so I won’t into more details here. 20130829_124250

Solution Area – Disaster Recovery

Disaster recovery is extremely important to most businesses, but is often complex to configure, test, and maintain. Solutions like SRM, which use array-based replication, are not very granular. All VMs on a particular datastore have the same recovery profile. This LUN-centric method is not flexible, and complex to manage. In contrast, future solutions based on vVOLS or other technologies enable VM-level recovery profile assignment. Technologies such as VMware NSX could enable pre-provisioning of entire networks at a DR site, to exactly match those of the production site. The combination of NSX and VM-level recovery profiles will truly revolutionize how you do DR and disaster avoidance.


Solution Area – Security

Security should be of concern in a virtual environment. One often overlooked area is security starting at the platform level by using a TPM (trusted platform module). TPM enables trusted and measured booting of ESXi. Third party solutions such as Hytrust can provide an intuitive interface to platform security and validate that ESXi servers only boot using known binaries and trusted hardware.

I make it a standard practice to always order a TPM module for every server, as they only cost a few dollars. How does this relate to SDS? Well if you use VSAN or other scale-out storage solutions, then you can use the TPM module to ensure the platform security of all unified compute and storage blocks. On the policy side, think about defining security options on a per-VM basis, such as encryption, when using vVOLs. The speakers recommended that if you work on air-gapped networks, then looking a fully converged solutions such as Nutanix or Simplivity can increase security and simplify management.


Example Scenario

At the end of this session Wade and Rawlinson quickly went through a sample SDS design scenario. In this scenario they have a rapidly growing software company, PunchingClouds Inc. They have different application tiers, some regulator compliance requirements, and short staffed with a single storage admin.


The current storage design looks like the model fibre channel SAN with redundant components. The administrator has to manage VMs at the LUN/datastore level.


At this point you need to do a full assessment of the environment. Specifications such as capacity, I/O profiles, SLAs, budget and a number of other factors need to be thoroughly documented and agreed upon by the stakeholders. Do you have databases that need high I/O? Or VDI workloads with high write/read ratios? What backup solution are they currently using?


After assessing the environment you need to work with the project stakeholders and define the business requirements and constraints. Do you need charge back? Is cost the primary constraint? Can you hire more staff to manage the solution? How much of the existing storage infrastructure must you re-use? All of these questions and more need to be thoroughly vetted.


After thorough evaluation of all available storage options, they came up with the solution design as shown in the slide below. It consists of a policy-based management framework, using two isolated VSAN data tiers, but also incorporates the existing fibre channel storage array.



The SDS offers a plethora of new ways to tackle difficult application and business requirements. There are several VMware and third-party solutions on the market, with many more on the horizon. In order to select the proper technologies, you need a methodical and repeatable process, “The VCDX Way”, to act as your guide along the SDS path. Don’t just run to the nearest and shiniest cool product on the market and just hope that it works. That’s not how an enterprise architect should approach the problem, and your customers deserve the best-matched solution as possible so that you become a trusted solution provider solving business critical needs.

VMworld 2013: Top 10 vSphere Storage Things to Know

Twitter: #STO5545, Eric Siebert (HP)

This was the best session of the day, and highly informative. Eric Siebert is a fast talker, and his slides were packed to the gills with details (and I didn’t even capture 25% of the content). This is the first session that I grabbed a few screenshots from and have incorporated them into my session notes. Please pardon the perspective skew in the photos. If you attended VMworld and have questions about vSphere storage, check out the whole deck. It is a goldmine of good information if you are not well versed in the topic. If you are a storage SME and are a consultant, this is an excellent reference deck to use in customer engagements.

Storage is fundamental to virtualization, and so often it’s not designed properly. And in VDI environments in particular, engineering the right storage solution is no easy task when done on a large scale. Eric covers the top 10 areas in storage that you need to consider to successfully implement a virtualized storage solution. This is just the tip of the iceburg, but a great resource.

If there’s one area of virtualization that can be a resume generating event (RGE), I would say that has to be storage. Unless you like getting pink slips, pay attention to how you design storage!

Why is Storage Critical

  • Shortage of any one resource prevents higher VM density
  • Storage is the slowest and most complicated of the four food groups
  • Available storage resources drive host VM density

#1 File or Block?

  • File or block and how do I decide?
  • Storage protocols are a method to get data from host to a storage array
  • Different protocols are a means to the same end
  • Each protocol has its own pros and cons
  • Review the two charts below for a good comparison of protocols


20130827_170601The Speed Race

  • Bandwidth is not about speed, it is about the size of the highway
  • Many factors influence performance and IOPS – Bandwidth, cache, disk speed, RAID, bus speeds, etc.

Things to consider:

  • VMware won’t tell you which protocol to use
  • vVols will level the playing field between NFS and block
  • Some applications tend to favor one specific protocol (e.g. VDI for NFS)
  • VMFS evolves with each release and new features usually first appear in block (e.g. VAAI)

#2 Storage Design Considerations

  • Must meet the demands: Integration (SRM, VASA, VAAI, etc.); High performance; High Availability; high efficiency
  • Flash, SSD, and I/O accelerators can enhance performance and eliminate bottlenecks
  • Avoid using extents to grow VMFS
  • VAAI (ATS) does not mean unlimited VMs per datastore
  • Use the PVSCSI controller
  • Avoid upgrading datastores to VMFS-5. Create new stores and migrate
  • LUNs don’t have unlimited IOPS
  • Use jumbo frames for iSCSI/NFS
  • Always isolate network storage traffic
  • Always read vendor best practices for your array
  • Always read vSphere performance best practices papers (vendor recommendations should take priority over generic VMware settings)

#3 Choosing between RDMs and VMFS

  • RMDs have limited value and more difficult to manage
  • Use RDMs only for practical reasons (e.g. clustering)
  • Use VMFS in most cases
  • VMware testing shows almost no performance benefit for RDMs
  • This is an old argument and pretty much put to bed…VMFS is the way to go for block

#4 Storage Performance

  • IOPS and Latency are key performance indicators
  • Constantly monitor IOPS and latency to spot bottlenecks
  • Monitor throughput
  • Baseline your system to know what’s normal and what’s not
  • Disk latency stats: GAVG, KAVG, DAVG
  • GAVG less than 20ms
  • GAVG or KAVG under 1ms
  • High DAVG indicates problem with storage array
  • HP announced a vCOPS integration pack for 3PAR which will GA later this year
  • Use esxtop to monitor storage performance
  • vSCSIStats is a good CLI utility
  • Storage Plut-in for vCenter server


#5 Why high available storage is critical

  • Single point of failure for most environments
  • When storage fails, all hosts and VMs connected to it fail as well
  • Protect at multiple levels: adapter, path, controller, disk, power, etc.
  • vSphere Metro Storage Cluster vs. vCenter SRM

#6 The role of SSDs in a vSphere Environment

  • SSDs should be a strategic tier, just don’t throw them in
  • Combine SSD and HDD for balance in performance and cost
  • SSDs wear out, so monitor closely

#7 Impact of VAAI

  • Enables integration with array-specific capabilities and intelligence
  • Fewer commands and I/O sent to the array
  • vSphere attempts VAAI commands every 16,384 I/Os

#8 Why Thin is In

  • Almost every VM never uses it’s full disk capacity
  • Starting thin is easy, staying thin is hard
  • Saves money
  • SCSCI UNMAP has been greatly improved (but not automatic) in vSphere 5.5

#9 vSphere Storage Features vs Array Features

  • Let your array do auto-tiering
  • Array thin provisioning is more efficient
  • Lots more on slides that I could write down fast enough..see the slide deck

#10 Benefits for VSAs

  • Less expensive than a shared storage array
  • Enables HA, vMotion, etc.
  • Lower capital cost
  • Will become more popular in the future, but does have some down sides

VMword 2013: vSphere 5.5 Space Reclaimation

Twitter: #STO4907. Aboubacar Diare (HP), Abid Saeed (VMware)

This session discussed how storage reclamation for block works in vSphere 5.0 thought vSphere 5.5. vSphere 5.5 has major improvements in how freed storage space is reclaimed. While not totally automated, it’s much more efficient and easier to kick off the reclamation process. I had to leave early to catch my next session, so I missed the performance impact part of the session.


  • Problem Statement
  • What is UNMAP
  • UNMAP in ESXi 5.x
  • Test Configuration
  • Results and considerations
  • Summary

Problem Statement

  • VMFS datastore freed capacity from VM deletion is not automatically released from the back-end storage array

What is UNMAP?

  • A SCSI primitive that is used with back-end storage to release LBAs back to the array
  • ESX 5.0 introduced the UNMAP command, and only supported on block storage
  • Does your device support UNMAP?  esxcli storage core device vaai status get -d
  • UNMAP is part of thin provisioning feature of the storage array
  • ESXi 5.0 GA: UNMAP is synchronous but disabled in ESXi 5.0 U1
  • ESXi 5.0 Ux and 5.1 Ux – Asynchronous method using vmkfstools -y
  • ESXi 5.5 – Asyncrhonous – can run from ESXi cli once and you are good to go

UNMAP is ESXi 5.0

  • ESXi 5.0 GA: User deletes VM or does storage vMotion, VMFS frees the blocks in VMFS, ESX sends UNMAP, storage frees physical blocks
  • ESXi 5.0 Patch 2: No UNMAP commands sent

Space Reclaimation in ESXi 5.0 Ux, 5.1 and 5.5

  • Step 1: User deletes VM/VMDK, VMFS frees the blocks in VMFS metadata
  • Step 2: User runs vmkfstools/esxcli command, then VMFS inflates a temp file of zeros, ESX sends UMAP commands to that temp file block range
  • vmkfstools reserves the free capacity (60% by default) when you unmap
  • VMFS free capacity is not the same as storage free capacity
  • Max balloon file size is 2TB.
  • ESXi 5.1 Update x will create multiple 2TB allocation files for datastores with more than 2TB free

Space Reclamation in ESXi 5.5

  • Uses esxcli, so you can run it remotely via server name and datastore name
  • Reclaim unit: Reclaim 100% of the space in small chunks. The chunk size is specified with reclaim_unit
  • Relclaim default is 200MB chunks
  • Unmap command is sent to the entire free capacity of the datastore, so storage free capacity always equals VMFS free capacity
  • Since only 200MB are reclaimed at a time, you are safe since the datastore won’t run out of free blocks for other VMs
  • Ran tests for reclaim units: 200MB to 12GB. Sweet spot seems to be 1.6GB to 12GB

TechEd: Storage Management with VMM 2012 R2 (MDC-B344)

This session focused on both the platform storage enhances in Windows Server 2012 R2 in addition to VMM 2012 R2. Microsoft was very up front that the 2012 release baked in a huge amount of technology into the platform (OS), but not all of it was exposed through VMM 2012 and even in SP1. In the R2 release both the platform and VMM have been more fully integrated and a lot of new features added. Going forward Windows and System Center will ship on the same schedule. Within Microsoft the OS and System Center teams have been re-aligned into the same org. Just like VMware ships the hypervisor and the mangement suite at the same time, Microsoft is now on the same cadence.

I didn’t get a screenshot, but the presenter had a slide showing the storage features in every version of VMM dating back to 2007. Starting with 2012 there was an explosion in features, with more added in SP1 (shipped in January 2013) and a lot more in R2. The pace at which Microsoft is enhancing the hypervisor and management stack is pretty astounding.

This session was supposed to be heavy on demos, but the speaker’s VPN connection back to the mother ship was not behaving. For his storage demos he was going to use a 3PAR to demonstrate the fibre channel LUN provisioning features in VMM 2012 R2, and NetApp for the SMB 3.0 file share demo. VMM has a lengthy list of storage arrays which are natively supported. If you are a 3PAR customer, you will need 3.1.2 MU1 for full VMM 2012 R2 support.

Storage Management Pillars

  • Insight: end to end mapping, pool, volume and file share classification, monitoring, standards based
  • Flexibility: Provisioning of pools, LUNs, file shares, scalable, allocation and assignment, FC zoning, zone aliases
  • Automation: Rapid provisioning, scale out file server, disaster recovery, bare metal Hyper-V host provisioning, ODX

R2 Enterprise Storage Management

  • More optimized storage discovery (e.g. a 3PAR with hundreds of disks) or VMAX with thousands of LUNs
  • Real-time updates for out of band changes using CIM indications
  • Fibre channel fabric discovery and zone provisioning and activation of zone sets
  • Support for Hyper-V virtual fibre channel
  • ODX optimized virtual machine deployments (copy VM from library)
  • Rapid provisioning using difference disks

Storage Provisioning for Tier 1 Application Demo

  • Fibre Channel switches
  • Hyper-V Host with 2 FC ports
  • Service template to model computer with two virtual HBAs

New to VMM 2012 R2

  • 10x faster SMI-S enumeration
  • Management of scale-out file server underlying spaces storage
  • Added remoting and cluster-awareness for managing storage spaces
  • Abilitity to assign storage and fabric classification at the volume or SMB share level. Allows finer grain SLA control.
  • Fully support iSCSI targets for storage
  • Support for SMB 3.02 (new to WS2012 R2)
  • Spaces provisioning: Discovery of physical spindles, storage pool creation and deletion, mirror and parity spaces creation and deletion
  • Capacity management: pool/volume/file share classification; file share ACL management
  • Scale-out file server deployment: bare metal deployment, creation of scale-out file server cluster, add/remove nodes, file share management

San Diego VMUG: Infrastructure Convergence with SimpliVity

This San Diego VMUG session was put on by Gabriel Chapman of SimpliVity. If you’ve never heard of SimpliVity, you aren’t alone. They are a start-up and emerged from stealth mode last year, and have been shipping product for about a month. Their OmniCube is an all-in-one 2U platform consisting of compute, memory, storage, ESXi, inline de-dupe and compression, SSD, SATA and a management layer.

You no longer have to manage separate computing and storage devices. Multiple OmniCubes can be federated for high-availability and remote data replication. Two key tenants of the platform are simplicity and cost efficiency. It was a high level session, so it didn’t delve into a lot of technical details. But I did grab a few notes from the presentation:

The Promise of Convergence

  • Consolidation, greater flexibility, Ease of use
  • VMware enables convergence, but it’s not all roses. Increased complexity, siloed management (storage, network, compute)

Efficient Data Management Features of the OmniCube:

  • Data efficiency – De-dupe and compression
  • Data mobility- Rapid clones, replication
  • Management – vCenter plug-in, globally managed
  • Deployment – Simple install, VMware integrated, scale out
  • Obsolete – LUN provsioning, RAID

SimpliVity OmniCube

  • 3x lower acquisition cost, power and operating costs
  • Combines compute, networking, storage (SSD and SATA)
  • Real-time inline data dedupe/compression
  • Global management
  • Each cube has a standard configuraiton of memory, disk, CPUs

This looks like an interesting product for certain use cases. Could be great for branch offices where you may not have highly skilled engineers, or smaller shops that want to virtualize but don’t have virtualization and storage gurus. Or maybe departments in large companies that need to do their own thing, and want something easy, affordable, and fully converged.

San Diego VMUG: VMware Backup Tips from Veeam

At the San Diego VMUG this session was presented by Rick Vanover from Veeam. He covered some tips and tricks for doing VMware backups. Of course the session was Veeam focused, and highlighted features of their backup software.

Tips for efficient VM Backups

  • Take a modern approach
  • We have a robust platform with vSphere, use it
  • Seek an approach built for virtualization – Remove burdens of agents inside VMs, easy on VM administrator, and scalable
  • Do not put the backup data where the VMs are running (e.g. SAN). Storage can fail.

Disk-Based Backup Flexibility

  • Many people do disk-to-disk backups with VMs
  • vSphere gives you a framework with many options (VDDK, VADP, direct SAN access, etc.)
  • Hardware dedupe appliances are more efficient than backup software dedupe

Upgrades with preparation

  • A virtual lab can help you ensure that a critical upgrade will go smooth as planned
  • Restore VMs to a sandbox and test upgrades without affecting production

Know where Bottlenecks Exist

  • Backuping up VMs have many moving parts
  • It’s important to know what may be slowing down your backups
  • Source storage, network, disk target, CPU resources, backup window

Veeam Explorer for Exchange

  • Restore Exchange items directly from the Veeam backup
  • Search and browse across one or more Exchange databases
  • Recover emails, contacts, calendar items, tasks, etc.
  • No agent needed, free in all editions of Veeam Backup and Replication

VMworld 2012: vSphere 5 Storage Best Practices INF-STO2980

Speaker: Chad Sakac, Vaughn Stewart

This was a really great session! Had two of the best sessions on the last day of VMworld.

  • Each protocol has different configuration considerations
  • Majority of customers use block protocols (iSCSI, FC, FCoE)
  • NetApp: NFS 50% usage, block the other 50% from autosupport data
  • Best flexibility come from a combination of VMFS and NFS
  • Key Point 1: Leverage Key documents
    • VMware technical Resource Center
    • FC SAN Config Guide
    • iSCSI SAN Config Guide
    • Best practices for NFS storage
    • Key partner documents – Best practices
  • Key Point 2: Setup Multipathing Right
    • vSphere Pluggable Storage Architecture (SATP) has several components
    • Don’t be inclined to make changes to the defaults..makes it more complicated and adds risk to the design. Don’t change the claim rules or I/O defaults.
    • PSP – Path Selection Policy
      • Fixed – Used commonly on active-active arrays
      • MRU – Default for many active-passive arrays
      • Round Robin – Default in vSphere 5.1 for EMC VNX/VMAX.
    • MPP
  • ALUA – Asymmetric Logical Unit Access. Common on mid-range arrays like NetApp and EMC VNX, and many other brands. Not true active/active for all paths and all LUNs.
    • Active – Optimized
    • Active – Non-optimized
    • Standby
    • Dead – APD – Target/array toally dead
    • “Gone away” – PDL – Can reach the array, but device such as LUN went away
  • Multi-Pathing with NFS
    • Significantlly different multi-pathing architecture than block protocols
    • NFSv3 is very basic in terms of understanding of multi-pathing
    • Must rely on switching technology for link aggregration
    • Single TCP connection from the ESXi for data and control information
    • Active/Passive path today until a future release of vSphere with NFS4
    • Use vendor specific vCenter plug-ins to enhance NFS support/configuration
  • Microsoft Cluster Service
    • Unsupported Storage Configuration – FCoE, iSCSI, NFS, Round Robin PSP, NPIV
    • Vendor support: 3rd party MPPs or Guest connected storage
    • Use iSCSI in guest – works very, very well (storage partners support this)
    • vSphere 5.1 has expanded support – up to 5 node cluster support
  • NFS Best practices
    • Use vCenter plug-ins, always! Automates configuration and tweaks
    • You can use FQDNs now and it will work
    • NetApp Cluster-Mode requires one IP per datastore
  • Jumbo Frames?
    • Recommendation is to NOT use jumbo frames. Adds more complexity, and performance increase is very marginal.
    • Stick with standard block sizes.
  • Optimize I/O
    • Misalignment of filesystems results in additional work on storage conroller to satisfy IO request
    • Affects VMFS and NFS datastores
    • Align the guest partitions! Automated in fresh installs of Windows Server 2008/Win7 and later
    • Linux will likely not align partitions. Must manually align paritions.
    • EMC UBerAlign – Free tool
    • NetApp – Data Ontap 8.1.1 – VSC plug-in Migrate and Optimize
  • Leverage Plug-Ins (VAAI and VASA)
    • 5.1 changes: vCenter Client plug-ins, NFS assists, block assists. See Chad’s Blog post
  • Keep It Simple
    • Use large capacity datastores
    • Avoid extents
    • Avoid RDMs
    • Array end – Use thin volumes and LUNs
  • Storage DRS
    • Use it! Even if in manual mode, it will make recommendations

Free Veeam VMware Management Pack for MS System Center 2012

For those of you that don’t already have a good monitoring solution for your VMware infrastructure, and use MS Operations Manager, you need to check out the Veeam Management Pack. It’s a good solution for single pane monitoring of your MS and VMware infrastructure. They are currently running a special for new customers that you can get a 10 socket license pack for free here.

Official announcement is below:

FREE 10 sockets of Veeam Management Pack

The Veeam Management Pack 10-Pack – a free VMware monitoring solution exclusively for new Veeam MP customers worldwide who are using Microsoft System Center 2012.

The Veeam Management Pack 10-Pack includes:

  • A free 10-socket license of the Veeam Management Pack for deep VMware monitoring in System Center 2012
  • One full year of maintenance and support

What is Veeam Management Pack for VMware?

The Veeam Management Pack provides scalable, fault-tolerant and agentless VMware infrastructure monitoring and management directly in Microsoft System Center.

Veeam MP enables you to:

  • Protect investments in System Center with integrated VMware monitoring
  • Manage physical and virtual infrastructure from one console
  • Eliminate the cost of additional monitoring frameworks


To qualify for this offer, you must be new to the Veeam MP and have System Center 2012 or plans to deploy it soon.

Get your free 2 socket Veeam Backup 6.0 License key

Veeam is once again running a Christmas special where you can get a free NFR (not for resale) Veeam Backup 6.0 license key for 2 sockets which is good for 1 year. You can select VMware, Hyper-V, or both. All you need to do is fill out this form and wait for the email with the license key.

For home labs or just trying out Veeam without their more limited timed trial versions, this is a great opportunity. Even if you don’t think you will use the key, I’d grab one anyway since you never know what may come up over the next year where it could come in handy.

Veeam is targeting the offer at certified VMware professionals such as VCP, but they don’t require any identifying information.

vSphere 5.0 Storage Improvements

If you a regular follower of my blog, you will probably notice I’m a bit of a storage geek. VAAI, FCoE, WWNs, WWPNs, VMFS, VASA and iSCSI are all music to my ears. So what’s new to vSphere 5.0 storage technologies? A LOT. That team must have been working over time to come up with all these great new features. Here’s a list of the high level new features, gleaned from a great VMware whitepaper that I have a link to at the end of this post.

VMFS 5.0

  • 64TB LUN support (with NO extents), great for arrays that support large LUNs like 3PAR.
  • Partition table automatically migrated from MBR to GPT, non-disruptively when grown above 2TB.
  • Unified block size of 1MB. No more wondering what block size to use. Note that upgraded volumes retain their previous block size so may want to reformat old LUNs that don’t use 1MB blocks. I use 8MB blocks, so I’ll need to reformat all volumes.
  • Non-disruptive upgrade from VMFS-3 to VMFS-5
  • Up to 30,000 8K sub-blocks for files such as VMX and logs
  • New partitions will be aligned on sector 2048
  • Passthru RDMs can be expanded to more than 60TB
  • Non-passthru RDMs are still limited to 2TB – 512 bytes

There are some legacy hold-overs if you upgrade a VMFS-3 volume to VMFS 5.0, so if at all possible I would create fresh VMFS-5 volumes so you get all of the benefits and optimizations. This can be done non-disruptively with storage vMotion, of course. VMDK files still have a maximum size of 2TB minus 512 bytes. And you are still limited to 256 LUNs per ESXi 5.0 host.

Storage DRS

  • Provides smart placement of VMs based on I/O and space capacity.
  • A new concept of a datastore cluster in vCenter aggregates datastores into a single unit of consumption for the administrator.
  • Storage DRS makes initial placement recommendations and ongoing balancing recommendations, just like it does for compute and memory resources.
  • You can configure storage DRS thresholds for utilized space, I/O latency and I/O imbalances.
  • I/O loads are evaluated every 8 hours by default.
  • You can put a datastore in maintenance mode, which evacuates all VMs from that datastore to the remaining datastores in the datastore cluster.
  • Storage DRS works on VMFS and NFS datastores, but they must be in separate clusters.
  • Affinity rules can be created for VMDK affinity, VMDK anti-affinity and VM anti-affinity.

Profile-Driven Storage

  • Allows you to match storage SLA requirements of VMs to the right datastore, based on discovered properties of the storage array LUNs via Storage APIs.
  • You define storage tiers that can be requested as part of a VM profile. So during the VM provisioning process you are only presented with storage options that match the defined profile requirements.
  • Supports NFS, iSCSI, and FC
  • You can tag storage with a description (.e.g. RAID-5 SAS, remote replication)
  • Use storage characteristics or admin defined descriptions to setup VM placement rules
  • Compliance checking

Fibre Channel over Ethernet Software Initiator

  • Requires a network adaptor that supports FCoE offload (currently only Intel x520)
  • Otherwise very similar to the iSCSI software initiator in concept

iSCSI Initiator Enhancements

  • Properly configuring iSCSI in vSphere 4.0 was not as simple as a few clicks in the GUI. You had to resort to command line configuration to properly bind the NICs and use multi-pathing. No more! Full GUI configuration of iSCSI network parameters and bindings.

Storage I/O Control

  • Extended to NFS datastores (VMFS only in 4.x).
  • Complete coverage of all datastore types, for high assurance VMs won’t hog storage resources

VAAI “v2”

  • Thin provisioning dead space reclamation. Informs the array when a file is deleted or moved, so the array can free the associated blocks. Compliments storage DRS and storage vMotion.
  • Thin provisioning out-of-space monitors space usage to alarm if physical disk space is becoming low. A VM can be stunned if physical disk space runs out, and migrated to another datastore, then resume computing without a VM failure. Note: This was supposed to be in vSphere 4.1 but was ditched because not all array vendors implemented it.
  • Full file clone for NFS, enabling the NAS device to perform the disk copy internally.
  • Enables the creation of thick disk on NFS datastores. Previously they were always thin.
  • No more VAAI vendor specific plug-ins are needed since VMware enhanced the T10 standards support.
  • More use of the vSphere 4.1 VAAI “ATS” (atomic test and set) command throughout the VMFS filesystem for improved performance.

I’m excited about the dead space reclamation feature, however, there’s no mention of a tie-in with the guest operating system. So if Windows deletes a 100GB file, the VMFS datastore doesn’t know it, and the storage array won’t know it either so the blocks remain allocated. You still need to use a program like sdelete to zeroize the blocks so the array knows they are no longer needed. You can check out even more geeky details at Chad Sakac’s blog here.

Hopefully VMware can work with Microsoft and other OS vendors to add that final missing piece of the puzzle for complete end-to-end thin disk awareness. Basically the SATA “TRIM” command for the enterprise. Maybe Windows Server 2012 will have such a feature that VMware can leverage.

Storage vMotion

  • Supports the migration of VMs with snapshots and linked clones.
  • A new ‘mirror mode’, which enables a one pass block copy of the VM. Writes that occur during the migration are mirrored to both datastores before acknowledged to the OS.

 If you want to read more in-depth explanations of these new features, you can read the excellent “What’s New in VMware vSphere 5.0 – Storage” by Ducan Epping here.