Improving network performance for Hyper-V R2 virtual machines on HP blade servers

How can we dramatically improve the network communication between two Hyper-V R2 virtual machines in HP blade servers?

Many of our customers have started using HP BladeSystem c7000 enclosures with ProLiant G6 or G7 blade servers. For the network interconnect they use HP Virtual Connect Flex-10 or FlexFabric blade modules which are ideally connected to the outside world via 10Gb Ethernet network switches. In a less ideal world multiple 1Gb connections can be combined to form a fatter trunk to redundant 1Gb Ethernet core switches.

So much for the cabling! As soon as we dive into the blade enclosure, all network communication stays within the confines of the blade enclosure with its multi-terabit signal backplane.

So how on earth can two virtual machines on two different physical Hyper-V R2 blade servers within this same enclosure only communicate at the speed of only 20 to 30MB per second? Conversely, how can we get them back to a much more acceptable speed? If you want to find out, I invite you to read on.

Let me first explain how the different components work together.

In the following diagram we see an HP c7000 Blade Enclosure with several blade servers, two Virtual Connect Flex-10 or FlexFabric blade modules which are each connected to a core Ethernet switch.

A Hyper-V R2 server cannot have enough network adapters. With the latest generation of blade servers we don’t need to fill up the enclosure with a switch pair for each network module or mezzanine. The dual-port 10Gb Ethernet onboard modules can be split into 2 x 4 FlexNICs. Speeds can be dynamically assigned in 100Mb increments from 100Mb to 10Gb.

image

So the Parent Partition on the Hyper-V R2 server sees 8 network adapters at the speed set in the Virtual Connect server profile. We set up at least three NIC teams for management, virtual machines and cluster communication (heartbeat, live migration, cluster shared volumes). The onboard network adapters in the blade server are from Broadcom and the teaming software used is HP Network Configuration Utility (NCU).

Until last week we used the NCU 10.10.0.x teaming software which allowed us to use native VLANs (see previous blogs). What the older versions have in common is the ultra low speed of VM to VM communication with this particular combination of hardware.

Because we wanted to find out we setup a test configuration with above configuration. The Hyper-V servers were Hyper-V Server 2008 R2 with SP1 (release candidate).

The network performance tests were conducted with NTttcp. This is a multi-threaded, asynchronous application that sends and receives data between two or more endpoints and reports the network performance for the duration of the transfer.

We setup two Windows Server 2008 R2 Virtual Machines with 1 vCPU, 1Gb of memory, 1 Virtual Machine Bus Network Adapter (with IC 6.1.7601.17105) and two VHD’s, one dynamic VHD for the OS and one fixed-sized VHD for the test. No changes were made to the physical or virtual network adapter in terms of TCP and other hardware offloads. We simply kept the defaults.

Test 1: VM to VM on same Hyper-V R2 host in same blade enclosure

Broadcom driver on host 5.2.22.0
Teaming software on host NCU 10.10.0.x
Teaming type Network Fault Tolerance with Preference Order
Network speed 2 x 4Gb (Effectively 4Gb)
Total MB Copied 1342
Throughput in Mbps 3436
Result Excellent
Method Average of 10 tests

Test 2: VM to VM on two different Hyper-V R2 hosts in same blade enclosure

Broadcom driver on host 5.2.22.0
Teaming software on host NCU 10.10.0.x
Teaming type Network Fault Tolerance with Preference Order
Network speed 2 x 4Gb (Effectively 4Gb)
Total MB Copied 1.342
Throughput in Mbps 319
Result Awful
Method Average of 10 tests

Via my HP contacts we learnt that HP has been working to improve network performance specifically for HP BladeSystem, Virtual Connect and the FlexNIC network adapters. It turned out that the slow speeds occurred on the LAN on Motherboard (LOM) B, C and D only. So LOM 1:a and LOM 2:a appeared to perform well. If you don’t split your networks into multiple FlexNICs you wouldn’t have noticed any degradation. However, in a Hyper-V cluster environment you need many more networks.

In this iSCSI connected server we use three teams:

  1. LOM 1:a + 2:a Management_Team (Domain access; managing Hyper-V hosts)
  2. LOM 2:a + 2:b VM_Team (for Virtual Machine communication)
  3. LOM 3:c + 3:c Cluster_Team (Live Migration, Cluster Shared Volumes)

The remaining two ports are used to create a MPIO bundle for connection to the iSCSI network.

image

Because the VM_Team is on LOM B, we suffered very low performance when two VM’s living on different Hyper-V hosts.

To see if the newly built drivers and teaming software (released on december 19th) we updated the Broadcom drivers and the NIC teaming software. The same tests were executed to see the difference.

Test 1: VM to VM on same Hyper-V R2 host in same blade enclosure

Broadcom driver on host 6.0.60.0
Teaming software on host NCU 10.20.0.x
Teaming type Network Fault Tolerance with Preference Order
Network speed 2 x 4Gb (Effectively 4Gb)
Total MB Copied 1342
Throughput in Mbps 4139 (+21,7%)
Result Excellent
Method Average of 10 tests

Test 2: VM to VM on two different Hyper-V R2 hosts in same blade enclosure

Broadcom driver on host 6.0.60.0
Teaming software on host NCU 10.20.0.x
Teaming type Network Fault Tolerance with Preference Order
Network speed 2 x 4Gb (Effectively 4Gb)
Total MB Copied 1342
Throughput in Mbps 1363 (+426%)
Result Good
Method Average of 10 tests

Although we haven’t tested and compared all LOMs we feel quite confident that network bandwidth is now distributed in a more even way across the different FlexNICs.

8 Comments

  1. Eli's Gravatar Eli
    December 22, 2010    

    what about G7 with Emulex onboard CNA card ?

    will NCU 10.20.0 make the difference or it the Broadcom driver ?

    • December 22, 2010    

      Don’t have the gear to try that one out myself. The performance improvement was mentioned in the NCU update, so its worth your try. Please report back if you have! Thx.

  2. Gaston's Gravatar Gaston
    December 23, 2010    

    how do you perform the Throughput test? Can you explain me what tool did you use?

  3. Matthew's Gravatar Matthew
    February 20, 2011    

    Hi, you mentioned that your HP contacts acknowledged a throughput issue with the other LOM NIC’s. Do you happen to have any reference material or post you can point me to so I can read more? Your blog post is the only reference I can find on the issue.

    Do you know if it was silently fixed in future firmware releases because the release notes do not shed any light on the issue.

    • February 21, 2011    

      There was mention of performance improvements in the accompanying readme but in general these bugs are kept silent until fixed and new ones introduced. It takes a bit of inside information I’m afraid ;-)

  4. Jim's Gravatar Jim
    March 5, 2011    

    I have the ability to test with 460C G7 and the Flex Fabric modules. Would you please post the command line used fro NTTTCP?

  5. March 9, 2011    

    Hi Jim, it would be cool to see the difference between BL460 G6 and G7.

    I used NTttcp (http://msdn.microsoft.com/en-us/windows/hardware/gg463264.aspx) to test network performance with the following parameters:

    NTttcpr -m 1,0, [ip address sender] -a 6 -fr -f

    Hope to hear from you soon!

    Hans Vredevoort

  1. Improving network performance for Hyper-V R2 virtual machines on HP blade servers « UP2V on December 22, 2010 at 21:34
  2. Tweets that mention Improving network performance for Hyper-V R2 virtual machines on HP blade servers – Hyper-V blog by Hans Vredevoort -- Topsy.com on December 22, 2010 at 22:23
  3. Windows Server 2008 R2 SP1 and HP Network Teaming testing results – Hyper-V blog by Hans Vredevoort on March 17, 2011 at 13:36

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code class="" title="" data-url=""> <del datetime=""> <em> <i> <q cite=""> <strike> <strong> <pre class="" title="" data-url=""> <span class="" title="" data-url="">

Our Sponsors





Powered by