VSP3111: Nexus 1000v Architecture, Deployment, Management

This session focused on the Cisco distributed virtual switch, the Nexus 1000v. The speaker was very knowledgeable and a great presenter. Lots of great details, but as fast as he was going I didn’t get all of the details. You can check out the his blog at jasonnash.com.


  • The VSM is a virtual supervisor module, which acts as the brains of the switch just like a physical switch.
  • The VEM is a virtual ethernet module, which is in essence, a virtual line card that resides on each ESXi host.
  • VSM to VEM communications are critical and you have various deployment options
    • Layer 2 only: Uses two to three VLANs and is the default option, and the most commonly deployed architecture.
    • Layer 3: Utilizes UDP communications over port 4785, so it can be routed
  • When in layer 2 mode you need to configure the control, management and packet networks
    • Management: End point that you SSH into to manage the VSM and maintains contact to vCenter. Needs to be routable.
    • Control: VSM to VEM communications (This is where most problems occur.)
    • Packet: Used for CDP and ERSPAN traffic
  • Nexus 1000v deployment best practices
    • Locate each VSM on different datastores
    • You CAN run vCenter on a host that utilizes the N1K DVS
    • ALWAYS, ALWAYS run the very latest code. Latest as of Sept 1, 2011 is 1.4a, which does work with vSphere 5.0.
    • Don’t clone or snapshot the VSM, but DO use regular Cisco config backup commands
    • Always, always deploy VSMs in pairs (no extra licensing cost, so you are dumb not to do it).
  • Port profile types
    • Ethernet profile: Used for physical NICs and are used as uplinks out of the server. These use uplink profiles.
    • vEthernet profile: Exposed as port groups in vCenter and is the most common type of administrative change made in the VSM.
  • Uplink teaming
    • N1Kv supports LACP, but the physical switch must support it as well.
    • vPC-HM – Requires hardware support from the switch and more complex to troubleshoot
    • vPC-HM w/ MAC pinning – Most common configuration and easy to setup/troubleshoot.
  • On Cisco switches enable BDPU filter and BDPU guard on physical switch ports that connect to N1K uplinks.
  • Configure VSM management, control, packet, Fault Tolerance, vMotion as “system” VLANs in the N1K so they are available at ESXi host boot time and don’t wait on the VSM to come up.
  • For excellent troubleshooting information check out Cisco DOC 26204.
  • You can also check out the N1KV v1.4a troubleshooting guide here.
  • The network team may prefer to use the Nexus 1010, which is a hardware appliance that runs the VSMs. This removes the VSM from the ESXi hosts, and could be better for availability, plus the network guys can use a serial cable into the 1010. You would deploy 1010s in pairs, and they have bundles that really bring down the price.
  • You can deploy multiple VSMs on the same VLANs, but just be sure to assign each VSM pair a different “DOMAIN” ID.

Not mentioned in this session are additional Cisco products that layer on top of the 1000v, such as the forthcoming Virtual ASA (firewall), a virtual NAM, and the virtual secure gateway. The ASA is used for edge protection while the VSG would be used for internal VM protection.

SUP1010: Cisco and VMware Innovating Together

This session was a ‘super session’ which is basically a vendor touting their wares and how well they integrate with VMware. To that end Cisco went through a number of announcements and innovations that are really industry leading. High points of this session were:

  • Policy based┬ámanagement is key to deploying clouds. Policies can include security, storage, network, and compute resources.
  • Clouds must be multi-tenant per-server, elastic, and automatic.
  • Network, compute and storage have now finally converged
  • Key tenants to clouds are: Open management, mixed vendor environments, industry standards and partner solutions.
  • Datacenter designs must have a 10-15 year design life
  • Policy driven management is fundamental and Cisco has implemented this via service profiles
  • 2011 Cisco achievements: Virtual ASA (firewall), #1 in VMmark 2.1 performance, enhanced VMdirectPath for VM mobility
  • Cisco UCS has a 21% performance advantage over same-core configurations from other vendors (tested in 4 node, 4 socket config)
  • New virtual interface card has dual 40Gbps ports and supports upto 256 PCIe interfaces for high-density VDI, multi-tenancy where you can link virtual ASA policies to a service profile.
  • vCloud Director now has integration with UCS for automated provisioning and configuration
  • VXLAN is a game changer for the networking industry. The Nexus 1000v will support VXLAN in beta in September 2011, and vCloud director will support VXLAN as well.
  • VXLAN supports up to 16 million interfaces, up from the 4096 VLANs.
  • VXLAN is the next generation “VLAN” concept and enables VM mobility across the cloud regardless of physical location.
  • VLANs are end of life!
  • VXLAN has been submitted to the IETF as a standard
  • Virtual ASA firwall appliance was announced yesterday
    • For multi-tenant datacenters
    • Uses the Nexus 1000v vPath technology
    • Same features as the physical ASAs
  • VM live migration across datacenters
    • Maintain security policies during and post-migration
    • Workload mobility
  • In 2011 Cisco and VMware
    • Integrated the Nexus 1000v with vCD
    • Enhanced UCS autodeploy with service profile integration
    • Overdrive network API
    • Integrate vCD and vShield manager with OverDrive
    • vShield Edge and N1K beta in Sept 2011
  • Future integration: OverDrive for network management, virtual ASA for security, N1K: a complete stack
  • 44 vendors have written products for the UCS XML integration API
  • Cisco UCS is now the #2 US blade manufacturer, after just 2 years in the market
  • Cisco is working on a virtual WAAS
  • ASA will provide tenant-level security down to the VM

In short, I think Cisco is leading the way with unified computing and the other major players (HP, IBM, Dell, etc.) have a lot of catching up to do. No solution stack is perfect, but looking at the currently shipping products and their integration roadmaps, I think Cisco “gets it,” It will be interesting to see how the other vendors respond since they are arguably lagging in both vision and shipping products.

Cisco leaks details of virtual ASA Firewall appliance

Hot off the presses is a Cisco ‘announcement’ of a virtual ASA product that is in the works, although no details were released about pricing and availability. It will leverage the capabilities of the Nexus 1000v DVS, and is more for edge protection (North-South traffic), vice internal traffic (East-West) which their VSG product is better suited for. You can see the full blog post from Cisco here.

New Cisco Nexus 1000v 4.0(4)SV1(3)

A few days ago Cisco released a new version of their virtual Nexus 1000v switch, version 4.0(4)SV1(3). You can read the release notes here. Major new features include:

– GUI configuration setup which creates VMware port groups, VLANs, turns on SSH, adds the plug-in to vCenter, and other nifty things.

– ERSPAN Type III support.

– Virtual NAM support. Monitors ERSPAN and NetFlow data sources.

Additional documentation can be found here. Overall, this is a very minor point release, unlike 1(2) which had some major new features.

New Nexus 1000v Release Notes and updated VEM

A few weeks ago Cisco revised their release notes for the Nexus 1000v 4.0(4)SV1(2) version. This is not a new version of the 1000v VSM appliance, just updated documentation. The updates clarify limitations with vMotion, VMware Lab Manager, ACL limitations, and NetFlow limitations. You can check out the new release notes here.

The good news in the revised notes is that vMotion of the VSM is now fully supported. However, you don’t want both the active and standby VSM on the same physical ESX host. Makes sense!

Cisco has released an updated 1000v download package, which appears to have an updated VEM component. The VSM component remains unchanged since the original 4.0(4)SV1(2) release.

VEM v110-

You can download the updated installer package here. If you just want to download the latest VEM, you can get it from VMware here and selecting VEM from the drop-down. VEMs seem to get updated in conjunction with each new public build of ESX(i).

Cisco Nexus 1000v hits v1.2

Cisco has released a minor update to the Nexus 1000v. They have a video showing some new features. Most of them are security related, and are summarized below. It appears to me VDS is designed to interoperate with the vShields Zones technology in vSphere.

  • Layer 3 control A VSM can be Layer 3 accessible and control hosts that reside in a separate Layer 2 network
  • Virtual Service Domain (VSD) Virtual service domains (VSDs) allow you to classify and separate traffic for network services. Interfaces within a VSD are shielded by a service VM (SVM) that provides a specialized service like a firewall, deep packet inspection (application aware networking), or monitoring.
  • iSCSI Multipath The iSCSI multipath feature sets up multiple routes between a server and its storage devices for maintaining a constant connection and balancing the traffic load.
  • DHCP Snooping DHCP snooping acts like a firewall between untrusted hosts and trusted DHCP server.
  • Dynamic ARP Inspection Dynamic ARP Inspection (DAI) validates ARP requests and response.
  • MAC Pinning If one or more upstream switches do not support port channels, you can use MAC pinning to assign each Ethernet port member to a particular port channel subgroup.
  • Static Pinning You can use vPC-HM to configure a port channel subgroup so that traffic is forwarded only through its member ports by assigning (or pinning) one of the following to the subgroup: vEthernet interface, the Control VLAN e Packet VLAN.

Update: Good information from Cisco here, including a PDF of the new features. VSM installation Guide is here. VEM installation guide is here. Cisco also has a host of new documentation for v1.2 you can find here (on the left side).

Cisco Nexus 1000v Documentation

Cisco has a good set of documentation for the Nexus 1000v. I’ve provided the direct links below. The compatibility guide seems to be frequently updated, so always be sure to download the latest version prior to deployment. For a good overview of the 1000v and all of the components, review the deployment guide.

General Information
Cisco Nexus 1000V Release Notes, Release 4.0

Cisco Nexus 1000V and VMware Compatibility Information, Release 4.0

Cisco Nexus 1000V Series Switches Deployment Guide

Install and Upgrade
Cisco Nexus 1000V Software Installation Guide, Release 4.0
Cisco Nexus 1000V Virtual Ethernet Module Software Installation Guide, Release 4.0

Configuration Guides
Cisco Nexus 1000V License Configuration Guide, Release 4.0
Cisco Nexus 1000V Getting Started Guide, Release 4.0
Cisco Nexus 1000V Interface Configuration Guide, Release 4.0
Cisco Nexus 1000V Layer 2 Switching Configuration Guide, Release 4.0
Cisco Nexus 1000V Port Profile Configuration Guide, Release 4.0
Cisco Nexus 1000V Quality of Service Configuration Guide, Release 4.0
Cisco Nexus 1000V Security Configuration Guide, Release 4.0
Cisco Nexus 1000V System Management Configuration Guide, Release 4.0
Cisco Nexus 1000V High Availability and Redundancy Reference, Release 4.0

Reference Guides
Cisco Nexus 1000V Command Reference, Release 4.0
Cisco Nexus 1000V MIB Quick Reference

Troubleshooting and Alerts
Cisco Nexus 1000V Troubleshooting Guide, Release 4.0
Cisco Nexus 1000V Password Recovery Guide
Cisco NX-OS System Messages Reference

If you are starting out your adventure with the Nexus 1000v, these documents should be required reading. At first the whole concept of a virtual switch running on a hypervisor may seem complex and a bit daunting. However, once you wrap your head around the various pieces it actually makes a lot of sense and may not be as complicated as you think.

vShield Zones and Cisco Nexus 1000v

So today at VMworld I learned that the current version of the Cisco Nexus 1000v is NOT compatible with the vSphere vShield technology. I didn’t get an approximate release date on when a compatible version would be released. Stay tuned!

9/1 Update: I talked to a VMware lead for vShields and he said Cisco is in the process of making changes to the 1000v to allow it to function with vShields. Basically they are adding a service port to the 1000v which can channel all traffic to the vShields VM and back into the 1000v. They are in the process of testing the update, and likely by year’s end it will be released.

HP Flex-10 and Cisco Nexus 1000v reference documents and video

Cisco has created a good PowerPoint slide deck on how to configure the Cisco Nexus 1000v with the HP Virtual Connect Flex-10 blade interconnect. You can download the slides here. They also created a video which you can see here.

Over the coming months I’ll be working integrating both technologies with VMware vSphere 4.0. So you can count on more posts in the future on this topic.

Oh, and in case you didn’t know HP sells 1Gb copper SFP modules for their Flex-10 10Gb interconnect. So you aren’t required to have an upstream 10Gb switch. They don’t make this too clear in the QuickSpecs, but they do have a 1Gb RJ-45 SFP buried document that you can purchase.

Update: Cisco pulled the slides from the original link. But, Cisco has uploaded an even newer version to the VMware community forums here. So check that out..much cleaner presentation than the older version.

Cisco Nexus 1000v demo videos

Today Cisco released two videos on the Cisco Nexus 1000v, going into a lot of good deals about how it works. I’d recommend anyone considering the Enterprise Plus edition of VMware vSphere and the Cisco 1000v to look at the videos.

If you want to view them in HD, so you can clearly see the screenshots, take a look at their facebook page. Look on the LEFT side of the page for the two videos with the HD icon. The primary videos in the middle of the page are low-resolution and VERY hard to read.