Short answer: You can never have too many! Longer answer: It depends. As many people know, VMware ESX is NIC hungry. Between the kernel console, Vmotion, and various production networks, it can take six or more phyiscal NICs to provide full redundancy. If you want to do Microsoft clustering, throw in another two NICs for the private heartbeat. If you want to virtualize TMG or UAG (formerly ISA and IAG, respectively) you might need even more NICs. Doing iSCSI or other IP storage? Best practices would recommend even more NICs. Here a NIC, there a NIC, everywhere a NIC, eeee–iiii–eee–iii–oooo!
In vSphere 4.0, VMware now has a feature called VM Direct Path IO. VM Direct Path IO allows a VM to directly talk to the underlying hardware, bypassing the hypervisor and reducing overhead allowing for maximum performance. In vSphere 4.0 though you lose Vmotion with Direct Path, which is a major down side.
So back to the NIC question. How do you get eight or more NICs per physical server? Various companies have solutions…or not. My current project is focused on blade servers, so let me talk about that for a bit. When investigating blade servers, you quickly realize some vendors are more virtualization oriented than others.
For example, with the IBM BladeCenter you can’t have more than six physical NICs if you also want a fibre channel HBA. This is true even on their newest Intel Nehalem blade server the LS22. If you are using IP storage and don’t need a fibre channel HBA, you can increase the NIC count.
HP has a technology called Virtual connect Flex-10, which let’s you reconfigure the two on-board 10Gb NICs into four FlexNICs each, for a total of eight NICs out of the box without using any mezzanine slots. Throw in a dual 10Gb mezzanine card and you have up to another eight FlexNICs, for a total of sixteen per physical server. In addition, you can customize the bandwidth of each NIC in 100Mbps increments. Want a wicked fast multi-gigabit Vmotion network, go for it! Pretty cool! The ESX hypervisor sees each FlexNIC as a unique physical NIC. Eight and sixteen NICs give you good flexibility in designing your ESX hosts and follow best practices. But why stop with sixteen NICs?
Recently Cisco announced their UCS blade servers, which supports SR-IOV via a new Intel NIC chip which features VT-c. SR-IOV is Single-Root Input/Output virtualization. This is a similar concept to HP’s FLex-10, but goes much further. Instead of being limited to dividing the 10Gb bandwidth into four NICs, Cisco supports 128 NICs. Now you are thinking, why do I need 128 NICs per physical server?
As I mentioned earlier, vSphere 4.0 is supporting VM Direct Path IO. With 128 NICs, you could assign each VM its own physical NIC(s), bypassing the hypervisor and allow near-native I/O performance. Cisco assured me that in the future both VMware and Cisco were working on technology to allow Vmotion and Direct Path IO to be used together. When? Who knows.
Given that NICs and I/O performance are key features for virtualization, when evaluating hardware virtualization solutions it really pays to define your requirements and carefully look at what the various vendors can provide. Cisco is brand new to the blade market and won’t be shipping their solution until June, so they currently have zero market share.
HP is currently the blade server industry leader, but will face some stiff competition from Cisco. If you have long hardware refresh cycles and haven’t yet made the plunge into blade servers, be sure to put Cisco on your short list of vendors. If you want a product with a long track record and shipping today, HP provides a compelling solution.
Just a quick note on costs. While I haven’t done any pricing of the Cisco UCS solution, there is a good blog by Brad Hedlund discussing Cisco vs. HP and some refuting some claims made by Engerena.
