Powered by System Center
// Main banner ad on homepage
Hyper-v
NIC Teaming, Hyper-V switch, QoS and actual performance | part 2 – Preparing the lab
Jan 9th
This blog series consists of four parts
- NIC Teaming, Hyper-V switch, QoS and actual performance | part 1 – Theory
- NIC Teaming, Hyper-V switch, QoS and actual performance | part 2 – Preparing the lab
- NIC Teaming, Hyper-V switch, QoS and actual performance | part 3 – Performance
- NIC Teaming, Hyper-V switch, QoS and actual performance | part 4 – Traffic classes
Preparing the lab
The test lab consist of two servers (HP Proliant DL 360 G5, nothing fancy but it will give a good picture on the processor demand). Each server contains a dual port 10Gb NIC and a quad port 1Gb NIC. The NICs have RSS and VMQ support. The quad port 1Gb NIC in the servers are directly connected to each other. This will give the best picture since a switch configuration might interfere with the results.
Performance is influenced by a lot of factors. For example, copying a large file between the servers will not be very representative. Server 2012 supports SMB multichannel, whereby multiple TCP streams are used for a single file copy. This requires Physical NICs with RSS support. SMB multichannel will work with NIC teaming since RSS is exposed through the team on the default interface. The Hyper-V switch does not support RSS and does not expose it to upper level protocols. SMB Multichannel will not function for the vNICs. A file copy initiated from a vNIC is single TCP stream. NIC Teaming is designed a single TCP stream to assign to a single team member. When the file is written to the destination, disk I/O can also impact the performance.
Luckily there are some good tools available for measuring bandwidth. During the tests JPerf will display detailed information on the bandwidth, Performance Monitor shows the load distribution and Task Manager gives insight into the processor load and distribution.
NTttcp, IPerf and JPerf
In my initial test I used NTttcp, that was rewritten by Microsoft in 2008. Microsoft is using an updated version of NTttcp that enables additional parameters, but this updated version is not publicly available. Therefore I resorted to IPerf. IPerf is a commonly used network testing tool that can create multiple TCP streams and measure the bandwidth of a network connection. IPerf can run as a server or as a client. The server listens on port 5001 and one or multiple clients can send a single TCP stream or multiple TCP streams to the server. IPerf was originally created for Linux, but there are compiled version for Windows publicly available. I have used a graphical front end for IPerf called JPerf. JPerf gives some nice graphs but requires Java so I wouldn’t recommend installing it on your production servers. If you want to run the same test in your production environment you can use the compiled version of IPerf (which will leave no footprint on the server) or create two virtual machines and install JPerf inside them.
Installation
If you want to use the command line version of IPerf (no footprint) copy the content of the compiled IPerf version to your server. For JPerf you will need to install Java first. JPerf does not require a separate IPerf file. You can just copy the content of JPerf to your server. Before you can run JPerf you will need to add the path to javaw.exe to the Path variable.
In the System Properties of your server open the Advanced tab and select the Environment Variables. Search for the Path variable and (if you installed Java in the default folder) add ;C:\Program Files (x86)\Java\jre7\bin to the end of Path variable.
Now you can open JPerf by running jperf.bat located in the root of folder you copied.
To configure JPerf as receiver select Server as IPerf Mode and click Run IPerf. IPerf listens on port 5001 by default. This port should be allowed in the firewall. With the Num Connections value of 0 IPerf will keep listening on port 5001 after a successful run.
To configure JPerf as sender select Client as IPerf Mode. In the server address specify the IP address of the server where JPerf is in listening mode. During the test I concluded that JPerf will only function on interfaces with a default gateway configured.
NIC Teaming, Hyper-V switch, QoS and actual performance | part 1 – Theory
Jan 8th
This blog series consists of four parts
- NIC Teaming, Hyper-V switch, QoS and actual performance | part 1 – Theory
- NIC Teaming, Hyper-V switch, QoS and actual performance | part 2 – Preparing the lab
- NIC Teaming, Hyper-V switch, QoS and actual performance | part 3 – Performance
- NIC Teaming, Hyper-V switch, QoS and actual performance | part 4 – Traffic classes
One of the basics of every Hyper-V configuration is networking. Set aside the missing flexibility, the choices for a Hyper-V cluster design in Windows Server 2008 R2 were clear. A dedicated network interface for each type of traffic (management, cluster, live migration). With this configuration in production NICs were underutilized most of the time and when you needed the bandwidth it was capped at the maximum of a single interface. In the (rare) case of a NIC dying on you there was no failover. In Windows Server 2008 R2 there was no NIC Teaming support. For load balancing and failover the only option was resorting to the NIC Teaming software provided by the hardware vendor.
From experience I can say that a lot of customers were having trouble designing their networking in a Windows Server 2008 R2 cluster correctly. Problems with 3rd party NIC Teaming, live migration over VM networks, not enough physical adapters, you name it, we’ve seen the most “creative” configurations.
Most customers are stuck in the Windows 2008 R2 thinking pattern. This is understandable as Microsoft strongly recommended that each network in a Windows 2008 R2 Hyper-V cluster had its own dedicated physical NIC in each host.
In Windows Server 2012, NIC Teaming is delivered by Microsoft out of the box. The official term is NIC Teaming, it is also referenced as Load Balancing and Failover (LBFO). NIC Teaming is an integral part of the Windows 2012 operating system. With NIC Teaming you can team multiple NICs into a single interface. You can mix NICs from different vendors, as long as they are physical Ethernet adapters and meet the Windows Logo requirement. The NICs must operate at the same speed. Teaming NICs operating at different speeds is not supported. But flexibility comes with complexity and many choices.
With Hyper-V in Windows Server 2012 it is even possible to create a Hyper-V switch on top of a NIC team. The Hyper-V switch is a full-fledged software based layer 2 switch with features like QOS, port ACLs, 3rd party extensions, resource metering and so on. You can create virtual adapters and attach them to the Hyper-V switch. These developments provide us with the proper tools to create converged fabrics.
What to expect
Usually the first thing tested after initial configuration is copying a large file between two hosts. With a Hyper-V Switch configured on a NIC team composed of two 10Gb adapters you might expect the file to copy with (2 x 10 Gbits / 8 =) 2.5 GBytes per second. When you copy the file you find that actual throughput is a lot lower (about 400 MB/s to 800 MB/s).
The first reaction : it doesn’t work!!
Let me clarify. It’s a little more complicated than just combining two 10Gb NICs and expecting a 2.5 GB/s file copy. It is possible to get these bandwidth results but you need to understand that there are a lot of factors of influence on the actual throughput.
Before we dive in to testing first we will have to look at the choices provided by Windows Server 2012 and how the inner workings of these choices are of influence on the actual bandwidth.
Theory
Transmission Control Protocol
TCP is one of the main protocols in the TCP/IP suite.
Transmission Control Protocol (TCP) is a transport protocol (layer 4). TCP provides reliable, ordered delivery of a stream of octets. TCP provides the mechanism to recover from missing or out-of-order packets. Reordering packets generates great impact on the throughput of the connection. Microsoft’s NIC Teaming (or any other serious NIC Teaming solution) will try to keep all packets associated with a single TCP stream on the same NIC to minimize out-of-order packets.
Hardware
There are some NIC hardware functionalities you should be aware of.
Receive side scaling
Receive side scaling (RSS) enables the efficient distribution of network receive processing across multiple processors.
It is possible to specify which processors are used for handling RSS requests. You can check if your current NIC hardware has RSS support by running the following PowerShell Get-SmbServerNetworkInterface
Virtual machine queue
Virtual machine queue (VMQ) creates a dedicated queue on the physical network adapter for each virtual network adapter that requests a queue. Packets that match a queue are placed in that queue. Other packets, along with all multicast and broadcast packets, are placed in the default queue for additional processing in the Hyper-V switch. You should enable VMQ for every virtual machine (and it is enabled by default). The new WS2012 feature, D-VMQ, will automatically assign the queues to the right VMs as needed based on their current activity.
Note Hyper-threaded CPUs on the same core processor share the same execution engine. RSS and VMQ will ignore hyper-threading.
Receive Side Coalescing
Receive Side Coalescing (RSC) improves the scalability of the servers by reducing the overhead for processing a large amount of network I/O traffic by offloading some of the work to network adapters.
For advanced configuration of these NIC hardware features Microsoft has released a great document on performance tuning guidelines for Windows Server 2012.
Hyper-V Protection with Data Protection Manager 2012 SP1
Dec 27th
In the quiet days between Christmas and New Year, I had some time to research how DPM2012 SP1 performed with protecting guests on a Windows Server 2012 Hyper-V cluster using CSV v2.0.
According to the SP1 release notes we can expect improved backup performance of Windows Server 2012 Hyper-V over CSV deployments with the following benefits:
-
900% improvement in Express Full backups
-
Parallel backups
-
No performance difference between backups from CSV owners and non-owners
Let me first point out that my setup is based on the following configuration:
Storage Server
-
HP ProLiant BL460c Gen8
-
Windows Server 2012 Datacenter with iSCSI Target Server Role enabled
-
Dual-port 10Gb HP FlexFabric 554FLB (Emulex) Adapter
-
Converged Fabric network configuration
Hyper-V Cluster Nodes
-
HP ProLiant BL460c Gen8
-
Windows Server 2012 Datacenter with Hyper-V role and Failover Cluster feature enabled
-
10Gb HP FlexFabric 554FLB (Emulex) Adapter
-
Converged Fabric network configuration
DPM 2012 SP1 Server
-
Windows Server 2012 Hyper-V Virtual Machine
-
Windows Server 2012 Datacenter
-
1 x 10Gb virtual network adapter (synthetic)
Network Configuration
Both on the iSCSI Target Server and Hyper-V cluster nodes, the two 10Gb network adapters have been teamed using a switch independent teaming mode with Hyper-V Port as the load balancing algorithm. A Hyper-V Extensible Switch is connected to the NIC Team and several virtual networks have been configured using the Converged Fabric method of Windows Server 2012. Each network has a minimum bandwidth Quality of Service configured on the virtual switch level. On the backend, the servers use a Virtual Connect Flex-10 interconnect.
HP Storage Platforms and Windows Server 2012
Dec 12th
[Updated: see 
]
HP bought 3PAR several years ago and as I expected the recently announced HP 3PAR StoreServ 7000 series are intended to replace HP EVA. There is even a data migration solution to make the migration as easy as possible. Of course I looked at what 3PAR has to offer in terms of features in Windows Server 2012 and integration with Hyper-V and System Center Virtual Machine Manager 2012.
The 3PAR StoreServ 7000 datasheet tells us this:
In Windows® Server 2012 environments, built-in, fine-grained virtualization, system-wide striping, and support for multi-tenancy give you the ability to consolidate mixed workloads onto a single HP 3PAR StoreServ 7000 system. With Windows Server 2012 Offload Data Transfer (ODX) and the HP 3PAR StoreServ 7000, you can migrate large files—such as databases or video files—up to seven times faster with near-zero network impact due to zerodetection capability integrated into the HP 3PAR ASIC. HP 3PAR Thin Technologies and Windows Server 2012 automatic reclamation of storage automate storage growth and shrinkage, while HP 3PAR Adaptive Optimization Software delivers the right QoS to the right data at the right time in Windows Server 2012 environments.
Not only does the new 3PAR StoreServ 7000 support Offloaded Data Transfer (ODX) but also Storage Management Initiative Specification (SMI-S) and a new Web Services API. SMI-S is what Microsoft has promoted vigorously and nobody seemed to believe in. At the end of the day most storage vendors finally comply with this SNIA standard. In our Microsoft Private Cloud Computing book, a good part of the storage fabric chapter is devoted to SMI-S integration in System Center Virtual Machine Manager 2012.
Lower end and non-comparable storage systems such as Dell’s EqualLogic only promise 2x improvement on file copy with ODX. The 7x promise with 3PAR’s ODX capability sounds good but of course we have to verify this in practice. At TechEd we were shown video’s comparing a 10GB regular copy (in about 3 minutes at 80-90MB/sec) versus a 10 second copy with ODX. I’m not sure if Microsoft used NetApp or EMC for the demo.
HP’s P4500 requires SANiQ v10.0 which is finally available and at a minimum will support Windows Server 2012. Many customers who invested in this platform and plan to upgrade to Windows Server 2012 Hyper-V will no doubt be very disappointed to learn that there is no ODX, no SMI-S, no UNMAP, no Dedupe. At least I cannot find any references to these important integrations for Windows Server 2012, Hyper-V and System Center. A 60-day trial for HP StoreVirtual VSA and documentation can be obtained here:
See an older blog on ODX:
http://www.hyper-v.nu/archives/hvredevoort/2012/04/windows-8-storage-hyper-v-part-4-offload-data-transfer-odx/
And on SMI-S:
http://www.hyper-v.nu/archives/hvredevoort/2011/03/vmm-2012-fabric-deep-storage-integration/
@WorkingHardInIT pointed out that the Dell EqualLogic number for ODX might even be 5x the speed of a regular non-offloaded copy. I based my number on a demo I had seen. In a Dell Techcenter blog on Windows Server 2012 and EqualLogic an increase in copy time of 5x is mentioned. As I stated for the 3PAR number, it is always best to verify these numbers in your own practice.
Windows Azure Services for Windows Server
Nov 9th
Many businesses are changing their perspective on cloud services. A lot of arguments that prohibit acceptance of the cloud slowly start to crumble. Internet connectivity is a common good, bandwidths are increasing and even the legal aspects are dealt with. Instead of a threat the cloud now empowers possibilities. Microsoft anticipated on this change years ago with numerous game changing developments. They have set the bar with their public cloud offerings like Office365.com, Outlook.com and Bing.com just to name a few worldwide deployed services. They have even raised the bar further with their cloud operating system Windows Azure. Providing these services to millions of people gave Microsoft great insight.
Leveraging the knowledge learned through their public cloud offerings, Microsoft has created a great platform for the private cloud with Windows Server 2012 and System Center 2012. This scalable solution provides businesses an elastic environment of pooled resources with self-service capabilities and usage based metering.
Microsoft now takes it to the next level by bringing Windows Azure to Windows Server. This solution mainly focusses on service providers but if you do not consider yourself one, please continue reading. The new service has also immense potential for enterprise organizations and might even trigger IT organization to think about providing hosted services.
Windows Azure for Windows Server consists of a Service Management Portal and a Service Management API. Microsoft has released a beta of Windows Azure for Windows Server with two services and will continue to add services over time. The first two services are high density website hosting and virtual machine provisioning and management. I will focus on the virtual machine provisioning service in this blog.
Microsoft provides an end to end solution enabling service providers to create an IaaS (Infrastructure as a Service) offering. This end to end solution consists four layers.
- Windows Server 2012: The virtualization layer
- System Center VMM 2012 SP1: The management layer
- Service Provider Foundation for SCVMM: Multitenancy for SCVMM and a REST ODATA API
- Windows Azure For Windows Server (WA4WS): The Service Management Portal and API
Private Cloud
Windows Server 2012 is the most cloud capable Operating System ever released. Combined with System Center Virtual Machine Manager 2012 as the management tool, they form the building blocks of the modern private cloud, providing scalability and availability features to meet even the most critical environments. Service Providers running hundreds or even thousands of workloads for their customers have these requirements. Leveraging the power of the private cloud, Microsoft enables multitenancy with the Service Provider Foundation (SPF). SPF also creates an industry standard Restful ODATA web service that developers can use for programming to System Center VMM 2012. This means that service providers can retain their current portal solutions and still benefit from the private cloud provided by Microsoft.
Windows Azure
Windows Azure runs tens of thousands of customers and hundreds are added to the platform each day. As you can imagine the management portal, the primary tool for this platform, is used heavily. Since the launch of Windows Azure on February 1, 2010 the management portal has evolved to fulfill a diversity of requirements, from browser independency to easy administrating, monitoring and diagnostics. Microsoft has taken these developments to complete their end to end offering for service providers and created a Service Management Portal combined with a Service Management API for Windows Server that are now available in Beta.
State of the VDI and SBC Union
Oct 17th
Project Virtual Reality Check is a initiative by Ruben Spruijt and Jeroen van de Kamp to investigate VDI infrastructure compared to a virtualized SBC infrastructure (among other things). You can read more about VRC here:
http://www.projectvrc.com/project-vrc-objectives
So far, Project Virtual Reality check has been a massive undertaking, which generated much interest over the years. The results have been presented at all the big technology events over the world and their findings and best practices have been published on different occasions. However, one thing was clear: many discussions in the VDI and SBC space are not just about performance best practices and product comparisons.
As a result the VRC team decided to boot up the first edition of the Project Virtual Reality Check “State of the VDI and SBC union” survey. http://www.brianmadden.com/blogs/jeroenvandekamp/archive/2012/09/26/announcing-project-vrc-s-quot-state-of-the-vdi-and-sbc-union-quot-survey.aspx
It was the VRC Team’s aim to ask all the relevant questions, both functional and technical. These questions range from “What are the most important design goals set for this environment”, to “Which storage is used”, to “How are the VM’s configured”. The questions are comprehensive, and relevant to everyone in building VDI and SBC environments. The aim of Project VRC is to repeat the survey at least once a year. This will allow us to see how our industry is changing in practice.
Within a couple of weeks more than 600(!) people started the survey:
https://nl.surveymonkey.com/s/ProjectVRCsurvey2012
The VRC team asked us to help promote this survey, which we are glad to do. Within the first week already more than 600 people have started the survey. So if you are in any way involved in the Hyper-V and VDI community, please go ahead and participate in this survey!
If you want to contact the guys behind VRC:
Jeroen van de Kamp | @TheJeroen
Ruben Spruijt | @Rspruijt
Intelligent Automatic Pagefile Setting in Windows Server 2012 Hyper-V
Oct 6th
UPDATED 10/8/2012
Pagefile
If you install Windows Server 2008 R2 with Hyper-V, the pagefile will be automatically managed, which usually has a 1:1 ratio with physical memory. However VM’s use their own pagefile and do not use the host paging mechanism. If you have lots of memory, chances are your pagefile will be way too big.
If we compare two servers with 16GB of memory each, one installed with Hyper-V R2 and the other installed with Windows Server 2012 Hyper-V we clearly see different numbers. Both servers have run for several days.
In the figure to the left you will see that Windows has 16GB allocated but recommends 24GB (1:1,5)
In the figure to the right you see that Windows Server 2012 is much more intelligent and allocates considerably less memory for paging. It is fairly consistent with the best practice in R2 to set a fixed maximum for the pagefile between 4 and 6GB.
Although I haven’t found any best practices yet for the pagefile in Windows Server 2012 Hyper-V, this might suggest that you can use the autopilot for virtual memory in the latest server edition of Windows. Of course we need a little bit more experience in the field to really call this a best practice.
If you are looking for a very thorough blog on pagefile settings in R2, please visit this blog.
Here is another one on technet:
http://blogs.technet.com/b/mghazai/archive/2011/05/26/what-s-the-story-of-pagefile-size-on-hyper-v-servers.aspx
And finally a post about the Hyper-V Dynamic Memory and Host Memory Setting in R2:
http://blogs.technet.com/b/virtualpfe/archive/2011/08/29/hyper-v-dynamic-memory-and-host-memory-reserve-setting.aspx
Host Memory Reserve
The Host Memory Reserve, which reserves memory for the processes in the parent partition can be found in the following registry key:
Registry Key: HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Virtualization
Value Name: MemoryReserve
Value Type: REG_DWORD
According to this post, the Host Memory in R2 is calculated in MB as follows: 384MB + (Memory in GB * 64)
So a 16GB host in R2 will have a default Host Memory Reserve of 384MB + (16 * 64) = 1.408MB
In two 16GB Windows Server 2012 hosts I looked at, the default Memory Reserve was 2048MB. We have to conclude that the formula used for Hyper-V R2 does not apply to Windows Server 2012.
I also looked at a Windows Server 2012 Hyper-V host with only 4GB which had a currently allocated pagefile size of 704MB and a recommended pagefile size of 3.582MB. This small host did not have a Host MemoryReserve entry at all, probably because a default 2GB Memory Reserve would be disproportional to the physical memory.
Update 10/8/2012
I had an interesting discussion about MemoryReserve with Michel Luescher who works for Microsoft Consultancy Services at Microsoft Schweiz. Michel is writing the chapters in Windows Server 2012 Hyper-V Installation and Configuration which deal with this subject.
IMPORTANT NOTE: Do not manually configure the Host MemoryReserve entry and leave this to Windows Server 2012 which will auto-configure this value on an as needed basis
In fact any newly installed Windows Server 2012 server with the Hyper-V role does not have a Host MemoryReserve entry in the registry and we still need to figure out under what circumstances this entry is added to the registry.
Take a look at Michel’s blog (German) on the subject:
http://www.server-talk.eu/2012/10/08/wie-viel-virtual-memory-braucht-ein-hyper-v-host-reloaded/
Win a Microsoft Private Cloud Computing Book
Oct 4th
A few days ago I promised to send someone a free Microsoft Private Cloud Computing book, provided they tweeted or blogged about The Great Big Hyper-V Survey of 2012.
Of course we want as many people to participate in this survey as possible. That’s why we have been rather persistent on Twitter and we are glad so many of you have already given attention to our survey. Several hundred Hyper-V users from all sizes of companies including several Fortune-500 companies have already contributed.
If you haven’t completed the survey yet, let me offer the book as an incentive.
But I haven’t explained the rules yet:
-
Of course fellow authors and colleagues of mine are not eligible to this prize. Sorry Aidan, Patrick, Damian!
-
You must have retweeted my tweets on TGBHSof2012 or you must have named us in your blog
-
Mentions on Facebook are not valid (as I have abandoned Facebook some time ago). Not sorry for Facebook : –)
-
If you retweeted I must be able to find you in this list which keeps track of the retweets:
http://bit.ly/QwKTIT
http://topsy.com/www.hyper-v.nu/archives/hvredevoort/2012/10/the-great-big-hyper-v-survey-of-2012-has-launched/?utm_source=button -
If you blogged and mentioned this page you are eligible:
http://www.hyper-v.nu/archives/hvredevoort/2012/10/the-great-big-hyper-v-survey-of-2012-has-launched/ -
I will announce an answer on the day of the close of the survey: October 31st 2012 19:00 CET
The Great Big Hyper-V Survey of 2012 Has Launched
Oct 1st
You can participate in the survey HERE
Just over a year ago, we asked people to answer 80 questions about their intentions and implementations of Hyper-V and System Center in The Great Big Hyper-V Survey of 2011. We learned a lot about how people were using Windows Server 2008 R2 Hyper-V and System Center 2007-2010. Back then, we knew just 2 things about Windows Server 2012 Hyper-V and System Center 2012 was still pre-release. But now, both are generally available, and we want to learn about:
-
Have you learned about these new technologies?
-
Do you already use them?
-
Are you planning on using them?
-
How do you plan to use them?
Once again, this is a completely independent survey, run by 3 MVPs (me, Aidan Finn, and Damian Flynn), and Microsoft has had no input or involvement. They might help us promote it – because we do know that our findings were read by them and some of the information was a surprise for them.
The goal of the survey is to learn. We’re all bloggers and speakers and we want to deal with what’s relevant. You’re interested in seeing what other people are doing. We all want to learn from each other and we learned a lot last year.
This is a perfect time to speak – if Windows vNext development is like that of Windows Server 2012, then they’ll be spending the next 10-12 months talking, learning, etc. And the same might happen post-SP1 for System Center 2012.
So we ask you to:
-
Respond to the survey and answer all the questions. There are 72 questions. I know, it’s a lot but there’s a lot of stuff to ask about. We’ll only be using complete responses.
-
Share the survey with colleagues, customers, on Twitter, Facebook, blogs, LinkedIn, MySpace, or whatever work-related social network you are on. We got an amazing response last year and we want to beat that. The more responses we can use, the more reliable the data will be.
Thank you in advance for taking the 10-15 minutes to respond to the survey.
BTW, we don’t ask for or want any personal data or email addresses. No individual response will be shared. We will only be sharing aggregate information, e.g. X people responded with Y answer.
The survey will close on December 1st 2012
Hyper-V Support Made Easy in Red Hat Enterprise 5.9
Sep 25th
If you are using Red Hat Enterprise Linux you’ll find great support for Hyper-V as a standard feature in the new minor Linux 5.9 release for which a beta became available recently.
Running Linux distributions with native Hyper-V support will save you the trouble of separately installing Hyper-V Integration Components to provide support for multiple cores and synthetic drives for mouse, video, network and storage. The Hyper-V Linux drivers were recently accepted upstream by the Linux community. For Red Hat Enterprise Linux 5 this means running as a guest on Hyper-V will improve overall performance.
Twitter
RSS