Powered by System Center
// Main banner ad on homepage
Posts tagged Powershell
Set Cluster Live Migration Settings
Apr 18th
Today we, Paul Huijbregts and I, resolved one of the last hurdles in finalizing our automated cluster installation script. This hurdle was to change the priority of the Live Migration settings when creating a Hyper-V cluster.
To change this priority we first tried to use the Set-VMMigrationNetwork PowerShell command. Unfortunately this command can only be used when dealing with non-clustered Hyper-V hosts. So we dug deeper and deeper using different PowerShell commands and BING without any satisfying results.
Then we realized there is something called “the registry” which holds the HKEY_LOCAL_MACHINE\Cluster key. After some more digging we found two registry entries called MigrationExcludeNetworks and MigrationNetworkOrder. These entries hold the IDs and order from the Cluster Networks available in your cluster.
Aha … room for possibilities! So, changing these registry entries would order and select the Cluster Networks in the way you want? Yes it does!
For this we fabricated some PowerShell lines.
$ClusterNetworkLM = Get-Clusternetwork LM
$ClusterNetworkCLUSTER = Get-Clusternetwork CLUSTER
$ClusterNetworkMGMT = Get-Clusternetwork MGMT
$ClusterNetworkISCSI = Get-Clusternetwork ISCSI$includeIDs = $ClusterNetworkLM.id + ";" + $ClusterNetworkCLUSTER.id
$excludeIDs = $ClusterNetworkMGMT.id + ";" + $ClusterNetworkISCSI.idSet-ItemProperty -Path "HKLM:\Cluster\ResourceTypes\Virtual Machine\Parameters" -Name MigrationExcludeNetworks -Value $excludeIDs
Set-ItemProperty -Path "HKLM:\Cluster\ResourceTypes\Virtual Machine\Parameters" -Name MigrationNetworkOrder -Value $includeIDs
The result is very very satisfying as you can see in the screen dump below. We are now able to control the order and the selection of the Live Migration settings in a cluster using the Cluster Network ID’s.
MVMC Automation Toolkit (MAT)
Apr 17th
In one of the break-out session at the Microsoft Management Summit 2013, which was held last week, an automation toolkit was shown to migrate virtual machines from “the other guys” to Hyper-V. The toolkit makes use of Microsoft Virtual Machine Converter and is part of the Solution Accelerator tools.
The Microsoft Virtual Machine Converter provides a Microsoft-supported, freely available, stand-alone solution for converting “the other guys”-based virtual machines and virtual disks to Hyper-V-based virtual machines and virtual hard disks (VHDs)—including conversion from “the other guys” to Hyper-V on Windows Server 2012. Because MVMC has a fully scriptable command-line interface (CLI), it integrates especially well with data center automation workflows such as those authored and run within Microsoft System Center 2012 – Orchestrator. It can also be invoked through Windows PowerShell.
MVMC provides you with:
- A quick, low-risk option for “the other guys” to evaluate Hyper-V.
- Converts “the other guys” virtual machines to Hyper-V virtual machines: The VM Conversion will convert “the other guys”-hosted virtual machines and ensure that the entire configuration, such as memory, virtual processor, and other machine configurations, is also migrated from the initial source. The tool also adds virtual NICs to the deployed virtual machine on Hyper-V.
- Supports a clean migration to Hyper-V with uninstallation of VMware tools on the source virtual machine.
- Provides a wizard-driven GUI, making it simple to perform virtual machine conversion.
- Installs integration services for Windows 2003 guests that are converted to Hyper-V virtual machines.
- Supports conversion of virtual machines from “the other guys” vSphere 4.1 and 5.0 hosts, including those hosted on a vSphere cluster, to Hyper-V. The tool also supports migration of virtual machines to a Hyper-V host that is part of a failover cluster.Note MVMC also supports conversion of virtual machines from “the other guys” vSphere 4.0 if the host is managed by vCenter 4.1 or vCenter 5.0. You have to connect to vCenter 4.1 or 5.0 through MVMC to convert virtual machines on vSphere 4.0.
- Supports offline conversions of “the other guys”-based virtual hard disks (VMDK) to a Hyper-V-based virtual hard disk file format (.vhd file).
- Includes a fully scriptable command-line interface (CLI) for performing machine conversion and offline disk conversion, integrating with data center automation workflows such as those authored and executed within System Center 2012 – Orchestrator. The command line can also be invoked through Windows PowerShell.
MVMC simplifies low-cost, point-and-click migration of Windows 7, Windows Vista, Windows Server 2008 R2, Windows Server 2008, Windows Server 2003 R2 with SP2, and Windows Server 2003 with SP2 guest operating systems from VMware to Hyper-V.
The MVMC Automation Toolkit is a collection of PowerShell scripts that will automate conversions using the MVMC.exe. It is back-ended by a SQL instance (SQL Express will work). You can use it to convert several machines at once, on a single server or across many servers at once.
MVMC Automation Toolkit can be considered as a low budget very easy to use toolkit to automate migrates from “the other guys” to Hyper-V. If you want to know more about this tool you have to watch this session!
Removing a non-existent VMM Library Server
Feb 5th
I admit, you don’t have to remove a non-existent VMM Library Server everyday, but today happened to be such a day. Let me explain what happened. In addition to a SQL Server 2012 AllwaysOn cluster for the VMM database and a VMM 2012 SP1 Failover Cluster, I wanted to also make the VMM Library Server highly available because the management foundation for a production Windows Server 2012 Hyper-V hosted cloud would soon become too big too fail.
Let me concentrate on the VMM Library cluster. I created an ordinary guest based 2-node Windows Server 2012 cluster using the new synthetic virtual Fibre Channel adapters, connected to an EMC VNX5300 storage system. After documenting the WWN’s and WWPNs for both adapters and requesting the SAN admin to create a few disks and to correctly zone the FC devices, creating the failover cluster was really a very quick deal. For some reason after installing EMC PowerPath 5.5 SP1, only one node of the cluster detected the correct Multi-Path Disk Device while the other thought it was dealing with a VRAID SCSI Disk Device.
Because uninstalling the devices or removing and reinstalling EMC PowerPath didn’t do the trick, I just went on to create a standard clustered file server role and add a share to be used for the VMM Library.
NIC Teaming, Hyper-V switch, QoS and actual performance | part 4 – Traffic classes
Jan 14th
This blog series consists of four parts
- NIC Teaming, Hyper-V switch, QoS and actual performance | part 1 – Theory
- NIC Teaming, Hyper-V switch, QoS and actual performance | part 2 – Preparing the lab
- NIC Teaming, Hyper-V switch, QoS and actual performance | part 3 – Performance
- NIC Teaming, Hyper-V switch, QoS and actual performance | part 4 – Traffic classes
With the insights from the results of the tests, it is possible to look at multiple scenario’s for the traffic classes live migration and virtual machine.
Live migration
Live migration moves machines from one host to another without noticeable downtime. This can be live migration within a cluster or moving virtual machines with “shared nothing” live migration. Live migrations uses one TCP stream for control messages (low throughput) and one TCP stream for transfer of virtual machine memory and state (high throughput utilization). When live migration includes migrating the VHD, SMB will be used for that. SMB itself will use one or multiple TCP streams depending on your SMB multichannel settings.
Scenario 1 : Server with two quad port 1Gb NICs
If you have invested in new 1Gb hardware before Windows Server 2012 was available, upgrading your NICs to 10Gb hardware is not a requirement. The NIC Teaming functionality allows for teaming up to 32 physical NICs. It is possible to reuse the dedicated 1Gb NICs you used for your Windows Server 2008 R2 or your (obsolete!!) VMware environment and create a single team.
The disadvantage with VMQ and LBFO based on Address Hash is that all the settings for the individual physical NICs in the team must be identical. Whereas NIC Teaming based on HyperVPorts allows for overlapping processor settings.
I have tested with additional live migration networks with the same metric in Switch Independent / HyperVPorts mode. Each live migration network will get its own port on the Hyper-V switch allowing for distribution of the individual live migration networks amongst the team members on a round-robin basis.
I created single NIC team with 8 1Gb team members in Switch Independent / HyperVPorts. After configuring a Hyper-V switch on top of this NIC team, I created six live migration networks with the same metric.
I also adjusting the maximum number of simultaneous Live Migration settings to ten simultaneous live migrations on each cluster node. Running a live migration of ten virtual machines (ten high throughput TCP streams) resulted in only one team member being utilized.
Live migration will use only one available network for moving virtual machine memory and state. Even if other live migration networks are configured with the same metric.
With 2 quad port NICs it is possible to create a different configuration for more live migration bandwidth without losing all VMQ overlapping. Create two NIC teams. One team with four 1Gb team members in Switch independent / HyperVPorts and one team with four 1Gb team members in LACP / Address Hash (you might even configure two team member per quad NIC in a single team for added redundancy).
The Switch independent / HyperVPorts NIC team is configured with a Hyper-V switch for converged Fabric. The LACP / Address Hash NIC team is dedicated for live migration. Since there is no Hyper-V switch on top of this NIC team, RSS is used for load balancing the individual stream.
NIC Teaming, Hyper-V switch, QoS and actual performance | part 3 – Performance
Jan 11th
This blog series consists of four parts
- NIC Teaming, Hyper-V switch, QoS and actual performance | part 1 – Theory
- NIC Teaming, Hyper-V switch, QoS and actual performance | part 2 – Preparing the lab
- NIC Teaming, Hyper-V switch, QoS and actual performance | part 3 – Performance
- NIC Teaming, Hyper-V switch, QoS and actual performance | part 4 – Traffic classes
Performance
Part 1 of this blog series explained the theory of NIC Teaming, Hyper-V switch and QoS. Theory is essential but we don’t run Hyper-V clusters in theory. We run them in production. Windows Server 2012 NIC Teaming and converged fabric allows for more bandwidth. Live migration and virtual machines are two traffic classes where more bandwidth can be useful. The following tests will look at the possible configurations to get the most bandwidth out of your Hyper-V environment on these traffic classes.
NIC Teaming to NIC Teaming
Now that we have the tools configured we can run our first test. It is a good idea to do this one step at a time so the differences in configuration will show exactly how this influences the results.
The first step is two create a NIC team on each server and connect them directly to each other. I have used the quad port 1Gb NIC on each server to create NIC team.
Each NIC team is configured in LACP / Address Hash. Running IPerf with a single stream results in a bandwidth of 113 MBytes per second.
As stated before the NIC team will force a single TCP over a single team member, so this is the expected result. Opening performance monitor during the test will verify this.
Adding more streams will balance the sessions over the team members. After adding one TCP stream per test, all four team members were active at ten parallel TCP streams.
NIC Teaming, Hyper-V switch, QoS and actual performance | part 2 – Preparing the lab
Jan 9th
This blog series consists of four parts
- NIC Teaming, Hyper-V switch, QoS and actual performance | part 1 – Theory
- NIC Teaming, Hyper-V switch, QoS and actual performance | part 2 – Preparing the lab
- NIC Teaming, Hyper-V switch, QoS and actual performance | part 3 – Performance
- NIC Teaming, Hyper-V switch, QoS and actual performance | part 4 – Traffic classes
Preparing the lab
The test lab consist of two servers (HP Proliant DL 360 G5, nothing fancy but it will give a good picture on the processor demand). Each server contains a dual port 10Gb NIC and a quad port 1Gb NIC. The NICs have RSS and VMQ support. The quad port 1Gb NIC in the servers are directly connected to each other. This will give the best picture since a switch configuration might interfere with the results.
Performance is influenced by a lot of factors. For example, copying a large file between the servers will not be very representative. Server 2012 supports SMB multichannel, whereby multiple TCP streams are used for a single file copy. This requires Physical NICs with RSS support. SMB multichannel will work with NIC teaming since RSS is exposed through the team on the default interface. The Hyper-V switch does not support RSS and does not expose it to upper level protocols. SMB Multichannel will not function for the vNICs. A file copy initiated from a vNIC is single TCP stream. NIC Teaming is designed a single TCP stream to assign to a single team member. When the file is written to the destination, disk I/O can also impact the performance.
Luckily there are some good tools available for measuring bandwidth. During the tests JPerf will display detailed information on the bandwidth, Performance Monitor shows the load distribution and Task Manager gives insight into the processor load and distribution.
NTttcp, IPerf and JPerf
In my initial test I used NTttcp, that was rewritten by Microsoft in 2008. Microsoft is using an updated version of NTttcp that enables additional parameters, but this updated version is not publicly available. Therefore I resorted to IPerf. IPerf is a commonly used network testing tool that can create multiple TCP streams and measure the bandwidth of a network connection. IPerf can run as a server or as a client. The server listens on port 5001 and one or multiple clients can send a single TCP stream or multiple TCP streams to the server. IPerf was originally created for Linux, but there are compiled version for Windows publicly available. I have used a graphical front end for IPerf called JPerf. JPerf gives some nice graphs but requires Java so I wouldn’t recommend installing it on your production servers. If you want to run the same test in your production environment you can use the compiled version of IPerf (which will leave no footprint on the server) or create two virtual machines and install JPerf inside them.
Installation
If you want to use the command line version of IPerf (no footprint) copy the content of the compiled IPerf version to your server. For JPerf you will need to install Java first. JPerf does not require a separate IPerf file. You can just copy the content of JPerf to your server. Before you can run JPerf you will need to add the path to javaw.exe to the Path variable.
In the System Properties of your server open the Advanced tab and select the Environment Variables. Search for the Path variable and (if you installed Java in the default folder) add ;C:\Program Files (x86)\Java\jre7\bin to the end of Path variable.
Now you can open JPerf by running jperf.bat located in the root of folder you copied.
To configure JPerf as receiver select Server as IPerf Mode and click Run IPerf. IPerf listens on port 5001 by default. This port should be allowed in the firewall. With the Num Connections value of 0 IPerf will keep listening on port 5001 after a successful run.
To configure JPerf as sender select Client as IPerf Mode. In the server address specify the IP address of the server where JPerf is in listening mode. During the test I concluded that JPerf will only function on interfaces with a default gateway configured.
NIC Teaming, Hyper-V switch, QoS and actual performance | part 1 – Theory
Jan 8th
This blog series consists of four parts
- NIC Teaming, Hyper-V switch, QoS and actual performance | part 1 – Theory
- NIC Teaming, Hyper-V switch, QoS and actual performance | part 2 – Preparing the lab
- NIC Teaming, Hyper-V switch, QoS and actual performance | part 3 – Performance
- NIC Teaming, Hyper-V switch, QoS and actual performance | part 4 – Traffic classes
One of the basics of every Hyper-V configuration is networking. Set aside the missing flexibility, the choices for a Hyper-V cluster design in Windows Server 2008 R2 were clear. A dedicated network interface for each type of traffic (management, cluster, live migration). With this configuration in production NICs were underutilized most of the time and when you needed the bandwidth it was capped at the maximum of a single interface. In the (rare) case of a NIC dying on you there was no failover. In Windows Server 2008 R2 there was no NIC Teaming support. For load balancing and failover the only option was resorting to the NIC Teaming software provided by the hardware vendor.
From experience I can say that a lot of customers were having trouble designing their networking in a Windows Server 2008 R2 cluster correctly. Problems with 3rd party NIC Teaming, live migration over VM networks, not enough physical adapters, you name it, we’ve seen the most “creative” configurations.
Most customers are stuck in the Windows 2008 R2 thinking pattern. This is understandable as Microsoft strongly recommended that each network in a Windows 2008 R2 Hyper-V cluster had its own dedicated physical NIC in each host.
In Windows Server 2012, NIC Teaming is delivered by Microsoft out of the box. The official term is NIC Teaming, it is also referenced as Load Balancing and Failover (LBFO). NIC Teaming is an integral part of the Windows 2012 operating system. With NIC Teaming you can team multiple NICs into a single interface. You can mix NICs from different vendors, as long as they are physical Ethernet adapters and meet the Windows Logo requirement. The NICs must operate at the same speed. Teaming NICs operating at different speeds is not supported. But flexibility comes with complexity and many choices.
With Hyper-V in Windows Server 2012 it is even possible to create a Hyper-V switch on top of a NIC team. The Hyper-V switch is a full-fledged software based layer 2 switch with features like QOS, port ACLs, 3rd party extensions, resource metering and so on. You can create virtual adapters and attach them to the Hyper-V switch. These developments provide us with the proper tools to create converged fabrics.
What to expect
Usually the first thing tested after initial configuration is copying a large file between two hosts. With a Hyper-V Switch configured on a NIC team composed of two 10Gb adapters you might expect the file to copy with (2 x 10 Gbits / 8 =) 2.5 GBytes per second. When you copy the file you find that actual throughput is a lot lower (about 400 MB/s to 800 MB/s).
The first reaction : it doesn’t work!!
Let me clarify. It’s a little more complicated than just combining two 10Gb NICs and expecting a 2.5 GB/s file copy. It is possible to get these bandwidth results but you need to understand that there are a lot of factors of influence on the actual throughput.
Before we dive in to testing first we will have to look at the choices provided by Windows Server 2012 and how the inner workings of these choices are of influence on the actual bandwidth.
Theory
Transmission Control Protocol
TCP is one of the main protocols in the TCP/IP suite.
Transmission Control Protocol (TCP) is a transport protocol (layer 4). TCP provides reliable, ordered delivery of a stream of octets. TCP provides the mechanism to recover from missing or out-of-order packets. Reordering packets generates great impact on the throughput of the connection. Microsoft’s NIC Teaming (or any other serious NIC Teaming solution) will try to keep all packets associated with a single TCP stream on the same NIC to minimize out-of-order packets.
Hardware
There are some NIC hardware functionalities you should be aware of.
Receive side scaling
Receive side scaling (RSS) enables the efficient distribution of network receive processing across multiple processors.
It is possible to specify which processors are used for handling RSS requests. You can check if your current NIC hardware has RSS support by running the following PowerShell Get-SmbServerNetworkInterface
Virtual machine queue
Virtual machine queue (VMQ) creates a dedicated queue on the physical network adapter for each virtual network adapter that requests a queue. Packets that match a queue are placed in that queue. Other packets, along with all multicast and broadcast packets, are placed in the default queue for additional processing in the Hyper-V switch. You should enable VMQ for every virtual machine (and it is enabled by default). The new WS2012 feature, D-VMQ, will automatically assign the queues to the right VMs as needed based on their current activity.
Note Hyper-threaded CPUs on the same core processor share the same execution engine. RSS and VMQ will ignore hyper-threading.
Receive Side Coalescing
Receive Side Coalescing (RSC) improves the scalability of the servers by reducing the overhead for processing a large amount of network I/O traffic by offloading some of the work to network adapters.
For advanced configuration of these NIC hardware features Microsoft has released a great document on performance tuning guidelines for Windows Server 2012.
MS Christmas present: SMI-S Storage Provider for iSCSI Target Server
Dec 26th
It is only several days since Microsoft made the final bits of System Center 2012 SP1 available, although for a relatively limited audience. If you have a TechNet or MSDN subscription, you have access to SP1 now. All others have to wait till general availability in the course of January 2013.
One of my fellow MVP’s in the Cloud and Datacenter Management department, Graham Davies, raised my interest by claiming there was something in VMM 2012 SP1 that had gone unnoticed: a brand-new SMI-S storage provider for the onboard iSCSI Target Server role in Windows Server 2012! People using the iSCSI Target Server for both demo/test/development and production will get quite excited if they discover how well this software storage solution is integrated, not only in Windows but now also in System Center 2012 SP1.
Back in March 2011, I wrote several blogs about the new Fabric concept in Virtual Machine Manager 2012 which was still in its early beta. One blog focused on deep storage integration using a SNIA standard called Storage Management Initiative Specification (SMI-S). When I was contributing my Fabric chapters for the Microsoft Private Cloud Computing book, there was only a limited number of usually very expensive storage arrays to really dig into this subject. I had access to some HP and NetApp storage to test SMI-S integration which was still very limited at the time. When we saw how the iSCSI Target Server, which was previously a separate install on Windows Server 2008 R2, developed and became included as a role in Windows Server 2012, we begged the product managers responsible for storage in System Center 2012 to also provide SMI-S support. Well guys … here it is!
Fast forwarding about two years, a lot has happened in the storage space and most storage vendors have introduced SMI-S support for their most important storage products. I now consider storage that does not support SMI-S (and some of the other cool technology such as ODX and Unmap/Trim), as a sign that these products are on that vendor’s death list and will soon be obsolete.
Moving Core Cluster Resources in a Windows Sever 2012 Hyper-V Cluster
Aug 31st
In Windows Server 2008 R2 it was not possible to move the Core Cluster Resources via Cluster Failover Manager (FCM). When performing maintenance on a cluster node, I prefer not only to evacuate all running guests on that node, but also the CSV disks which are owned by that node. If the Cluster Resource Group also resides on the node that is due for maintenance, one must resort to PowerShell to move that group somewhere else.
In a Windows Server 2012 cluster, the PowerShell commands are exactly the same as in Windows Server 2008 R2. If you don’t feel comfortable using PowerShell, you might be pleasantly surprised that this activity can now also be performed from the Failover Cluster Manager.
In FCM, right-click the Clustername, click on More Actions, Move Core Cluster Resources and select either Best Possible Node or select the node of your choice.
Hyper-V Network Virtualization in Windows Server 2012
Aug 24th
Today I have the pleasure to announce a new promising blogger for Hyper-V.nu. I will let him introduce himself:
Hi, my name is Marc van Eijk. Ever since I bumped into my first TCP packet, I have had great interest in and a strong preference for IT infrastructure. I am currently working at Duvak as an IT Consultant and responsible for all IT Pro’s in the organization.
I have focused on Hyper-V since the first beta.
Known to be driven, persistent and a true Microsoft supporter.
I am constantly learning and find helping people with infrastructure challenges very gratifying.
Marc told me he had been following Hyper-V.nu for quite a while now and that it had helped him on several occasions in his own IT consultancy practice. Now he wants to give back his knowledge and experience to the Hyper-V community. When Marc handed in his first blog for review, I was very happy to see the result. Without much further ado … here is Marc’s blog on Microsoft Hyper-V Network Virtualization. Please give him a warm welcome and many RT’s. If you have any questions about this very interesting subject, please use the comment and I will make sure you are answered.
Network Virtualization
In the early days of Virtualization we tried get our heads around the idea of abstracting a server from its hardware, disks presented as a files and even movability of a guest. Today it is just business as usual. At least for most of us I hope.
Looking at the reimagined Windows Server 2012 a new concept called Network Virtualization is emerging. This feature is the basis of a larger trend called software defined networking (SDN).
Is this something like the networks we created in the Windows 2008 R2 virtual switch? On the contrary!
What is Network Virtualization?
Similar to running multiple virtual servers on a single host, with network virtualization it is possible to run multiple isolated virtual networks on a single host. In essence this is not a new feature, because with VLANs this has already become possible in the Windows 2008 hypervisor. But VLANs – for larger organizations – have limitations (4096 to be exact) and in these environments VLAN maintenance can be error-prone and cumbersome. Network virtualization is a great feature for hosters, but many private clouds can take advantage of this in multiple ways as well. It should be noted that when a VM uses network virtualization it cannot use VLANs and vice versa.
Address Space
With network virtualization the virtual networks get abstracted from the physical network topology. The physical network called the Provider Address (PA) Space is maintained by the fabric administrator. The virtual isolated networks called the Customer Address (CA) Space can be maintained by the customer (or a division or unit in your organization).
Routing
Similar to VLANs, you can create multiple subnets. Every virtual subnet is a single broadcast domain and has a unique Virtual Subnet ID (VSID). A virtual subnet routes traffic to other virtual subnets that belong to the same routing domain (called the customer network). If you have a single routing domain with two virtual subnets, the virtual subnets are allowed to automatically route traffic to each other. If you have virtual subnets within different routing domains there is no routing between them.
The routing within the routing domain is enabled by default, you cannot disable it. If you do not want routing between virtual subnets you are advised to configure them in separate routing domains. The default gateway within a virtual subnet is the first usable IP address within the IP range.
Twitter
RSS