Hyper-V Network Virtualization in Windows Server 2012

Today I have the pleasure to announce a new promising blogger for Hyper-V.nu. An introduction:

Marc van Eijk has a great interest in Microsoft virtualization and fabric management. Starting his IT career in 2000 at a Microsoft Partner he become responsible for all consultants after three years. A steep learning curve combined with a determined mind, resulted in great projects and helped the organization to evolve. After 13 years of combining two jobs, he decided to follow his heart and agreed to start as a Consultant at Inovativ, the System Center and Microsoft virtualization specialists in the Netherlands. His job description has a clear focus on Hyper-V, System Center VMM, Service Provider Foundation and Windows Azure Pack. Enjoys sharing knowledge by blogging for www.hyper-v.nu and speaking at events. Known to be persistent and driven by passion.

Marc told me he had been following Hyper-V.nu for quite a while now and that it had helped him on several occasions in his own IT consultancy practice. Now he wants to give back his knowledge and experience to the Hyper-V community. When Marc handed in his first blog for review, I was very happy to see the result. Without much further ado … here is Marc’s blog on Microsoft Hyper-V Network Virtualization. Please give him a warm welcome and many RT’s. If you have any questions about this very interesting subject, please use the comment and I will make sure you are answered.

Network Virtualization

In the early days of Virtualization we tried get our heads around the idea of abstracting a server from its hardware, disks presented as a files and even movability of a guest. Today it is just business as usual. At least for most of us I hope.

Looking at the reimagined Windows Server 2012 a new concept called Network Virtualization is emerging. This feature is the basis of a larger trend called software defined networking (SDN).

Is this something like the networks we created in the Windows 2008 R2 virtual switch? On the contrary!

What is Network Virtualization?

Similar to running multiple virtual servers on a single host, with network virtualization it is possible to run multiple isolated virtual networks on a single host. In essence this is not a new feature, because with VLANs this has already become possible in the Windows 2008 hypervisor. But VLANs – for larger organizations – have limitations (4096 to be exact) and in these environments VLAN maintenance can be error-prone and cumbersome. Network virtualization is a great feature for hosters, but many private clouds can take advantage of this in multiple ways as well. It should be noted that when a VM uses network virtualization it cannot use VLANs and vice versa.

Address Space

With network virtualization the virtual networks get abstracted from the physical network topology. The physical network called the Provider Address (PA) Space is maintained by the fabric administrator. The virtual isolated networks called the Customer Address (CA) Space can be maintained by the customer (or a division or unit in your organization).

Routing

Similar to VLANs, you can create multiple subnets. Every virtual subnet is a single broadcast domain and has a unique Virtual Subnet ID (VSID). A virtual subnet routes traffic to other virtual subnets that belong to the same routing domain (called the customer network). If you have a single routing domain with two virtual subnets, the virtual subnets are allowed to automatically route traffic to each other. If you have virtual subnets within different routing domains there is no routing between them.

Figure 1

The routing within the routing domain is enabled by default, you cannot disable it. If you do not want routing between virtual subnets you are advised to configure them in separate routing domains. The default gateway within a virtual subnet is the first usable IP address within the IP range.

Abstraction Mechanism

The abstraction of the network can be achieved in two ways:

  • The first and probably the most obvious one, is based on the IP packet encapsulation protocol Generic Routing Encapsulation (GRE).

GRE is a tunneling protocol and is used to encapsulate CA IP address in a package with a new header in the Provider Address. Microsoft, Arista Networks, Intel, Dell, HP, Broadcom, Emulex have submitted a draftwith the IETF for Network Virtualization Generic Routing Encapsulation (NVGRE) as a proposed standard.

  • The second mechanism uses IP Rewrite that modifies the CA IP address of the packet on the virtual machine with the PA IP address before it is transferred on the physical network.

 

IP Rewrite is interesting for VMs requiring very high throughput (~10Gbps). Existing network hardware offload technologies, such as Large Send Offload (LSO), Virtual Machine Queue (VMQ) and multipath routing in the switches (for example, ECMP, Equal-Cost, Multi-Path) work as expected. Unfortunately this comes with a huge disadvantage. IP Rewrite requires a unique PA for each VM CA in order to isolate VMs from different customers using overlapping IP addresses. IP Rewrite requires one PA per virtual machine CA IP addresses, whereas NVGRE only needs one PA per host. I will therefore focus on NVGRE in this blog.

NVGRE

Network Virtualization uses GRE to encapsulate the packet going from one VM to another in a tunnel.

During physical network routing, the traffic only sees host to host traffic. The VMs are unaware of the Network Virtualization and only see the VM to VM traffic. Only traffic coming from VMs can optionally be virtualized. For VMs running on the same host that do not use network virtualization and other traffic (management, live migration, cluster shared volumes, storage) the VSID will default to 0. If the VSID is 0, the packet will pass-through the Windows Network Virtualization (WNV) module.

When VM1 (192.168.1.5) running on host 1 (172.16.1.1) wants to send a package to VM3 (192.168.1.7) running on host 2 (172.16.1.2),VM1 will do an ARP for 192.168.1.7. This ARP package enters the Hyper-V extensible switch which in turn checks if there are any other VMs running locally (on the same host) with the same virtual subnet ID (VSID) and sends the ARP package to these VMs. The ARP request with its VSID also gets send out to the network by the extensible switch. This ARP package is intercepted by the network virtualization module and is never broadcasted on to the CA network. The Network virtualization module will look at its PA/CA mapping table to check if there is an entry for the target IP and VSID.

Each Hyper-V host only has a segment of the complete mapping table. The mapping table consists of virtualization policy rules that define

  • The mapping of CA and PA for each VM
  • The corresponding virtualization mechanism (rewrite/encap)
  • Which customer subnet these rules are for
  • Routing topology between customer virtual subnets, WNV virtual subnets and non-WNV networks

If there is no entry in the local mapping table, the host will request System Center Virtual Machine Manager 2012 SP1 (SCVMM 2012 SP1) for the current location. If there is an entry in the local mapping table or the host just learned this through SCVMM 2012 SP1, the Network Virtualization module creates an ARP response with the MAC address of the target VM and sends this into the extensible switch. The source VM can now create a package with the source and the target mac address and the source and target IP address.

When the packet of the source VM enters the extensible switch in host, the VSID of the source VM is added. This enables switch extensions (running in the CA Space) to use the VSID of the different VMs coming in and to understand which Virtual Subnet a VM belongs to. Once the package leaves the extensible switch, it enters the network virtualization module. This module will look at its PA/CA mapping table to check if the source VM’s VSID is allowed to send a package to the target VM’s VSID. If this is not allowed the package is dropped. This check enforces secure isolation at the source. If it is allowed, the source and target MAC address of the VMs packets is encapsulated in a tunnel, the virtual subnet id (VSID) of the source and target VM are added. In the packet header the IP address of the source VM is changed to the IP address of the host this VM is running on and the IP address of the target VM is changed to the IP address of the host that is running the target VM.

Figure 2

Once the packet arrives at the target host, the complete process takes place again but in reverse.

Ok, sounds good. So where is the checkmark to enable it?

Well, with Windows Server 2012 there is no virtual networking checkbox. Since there is no GUI for Network Virtualization within Windows Server 2012 you are invited to enable Network Virtualization with PowerShell, which works great by the way!

Even if you are PowerShell power ranger, Network Virtualization quickly gets the better of you. With a single Hyper-V host you’re good to go, but as you can imagine, with an increasing number of hosts and VMs moving all over the place the maintenance of the policy tables with PowerShell will very quickly become impossible. If I made you curious, you might take a look at this PowerShell command. This is where SCVMM 2012 SP1 comes into play.

System Center Virtual Machine Manager 2012 SP1

At this moment the community technology preview 2 (CTP2) of SP1 is publicly available for download and Microsoft has promised the feature complete beta very soon. This isn’t your Grand Dad’s Service Pack, consisting of hotfixes and some minor updates. The Service Pack 1 for SCVMM 2012 would have been better off called Service Packed 1 as this update is packed with new functionality. Awesome features in Windows Server 2012 are tightly integrated into VMM 2012 with Service Pack 1. When you are considering Network Virtualization, the upcoming SCVMM 2012 SP1 is a must-have.

Logical Networks

If you have worked with SCVMM 2012 RTM which shipped last April, you will be familiar with the term Logical Networks. A logical network is an abstraction of the physical network to a simple understandable name. When a customer, division or unit want to create a VM and is presented with the choice whether to add VLAN 20 or VLAN 21 they probably don’t have a clue. But if you ask them if they want the VM to be in the corporate network or in the DMZ they will understand just fine (at least some of them).

Logical Network Definitions

The logical network consists of one or more Logical Network Definitions. When you have multiple sites where for instance, a DMZ is configured (but in different VLANs with different subnets) and you deploy a VM or a complete service, you can just select the logical network. Based on placement of the VMs, SCVMM uses the logical network definitions to attach the VMs to the correct physical network.

clip_image005

IP and MAC Address Pools

Each Logical Network Definition can have an IP or MAC Address Pool which SCVMM uses to assign a new VM an IP or MAC Address. The IP Address Pool takes away the requirement to manually check if you have already assigned some IP.

The IP Address Pool assigns static IP addresses. Within an IP Address Pool you define an IP range in the CIDR format, reservations (IP addresses that won’t be assigned from the range), one or more default gateways (ordered with a metric), DNS servers, DNS suffix and even, for the unlucky ones, WINS.

Figure 4

Of course you can use a DHCP server assigning dynamic IP addresses on some VLAN. The IP Address Pools are optionally.

NICs on Hyper-V hosts

In Hyper-V hosts you have multiple NICs (if not please read this). Although these can be physical NICs, with Windows Server 2012 you can create an Extensible Hyper-V switch connected to a NIC team. On that switch you can rapidly create multiple virtual NICs (see blog on converged fabric by Hans Vredevoort). From within SCVMM you can subsequently define which logical networks belong to which particular NIC.

Figure 5

This will ensure that VMM selects the host as a possible candidate during deployment and also connects the VM to the correct NIC.

What’s new in SCVMM SP1?

SCVMM 2012 SP1 changes the GUI of the network abstraction by providing 4 types of VM networks.

  1. No Isolation. This VM Network is a direct pass-through to the logical network. You can only have one No Isolation VM Network per Logical Network. When you upgrade your SCVMM 2012 RTM to SP1, your current logical networks will be converted to VM networks type No Isolation. This means you do not have to reconfigure your settings within SCVMM before you can upgrade to SP1.
  2. Hyper-V Network Virtualization. By default, VMM uses IP Rewrite in the CTP2 release. There is no GUI for NVGRE (PowerShell only).
  3. VLAN. You can only have one VLAN VM Network per Logical Network. These must be configured at the physical switch and uses the VLANs specified in the Logical Network Definitions.
  4. External. Isolation is managed by extensions in the Hyper-V extensible switch. SCVMM only gets reports on the changes for information display, but it doesn’t do any active management.

So how can you create a VM Network based on the type Network Virtualization?

The creation is divided in two steps.

1. First you will have to create the PA Space which represents the physical network.

2. On top of the PA Space you create the CA Spaces.

1. Create the Provider Address (PA) Space

In SCVMM 2012 SP1, select the Fabric, click on networking and create a new logical network (just as in RTM).

Figure 6

Create a network site with one subnet (Vlan id must be 0). This network is the PA (Provider Address Space).

2. Create the Customer Address (CA) Space with IP Rewrite

When a new tenant, division or unit needs to be configured, open VMs and Services, click on VM Networks and create a new VM Network. Specify the abstraction name (for example the name of the tenant, division or unit), select the logical network you created in the previous step (PA Space).

Figure 7

Now you can define one or multiple subnets. These subnets will be part of the same routing domain, so routing is enabled between them by design.

Figure 8

Once the VM Network is created you can assign IP Pools as you would in pre SP1.

Figure 9

Note that the first IP address is reserved for subnet routing within the same routing domain. SCVMM SP1 removes the first IP address from the possible IP range.

Create the Customer Address (CA) Space with NVGRE

The default setting in CTP2 is to use IP Rewrite. When you want to create a VM Network that uses NVGRE in CTP2, your only option is the VMM command shell (there is no GUI setting, yet). In the RTM of SP1 it might look something like this.

Figure 10

But don’t worry, even the faint of heart should get this right. (If you are using this as a step by step you should first remove the subnet you created in the previous step).

$SubNetVLan = New-SCSubnetVLan -Subnet “192.168.1.0/24″

$VMNetwork = Get-SCVMNetwork -Name “Customer Address Space”

New-SCVMSubnet -Name “hyper-v.nu” -SubnetVLan $SubnetVLan -VMNetwork $VMNetwork -VMSubNetType “GREWindowsNetworkVirtualization”

Which results in

Name : hyper-v.nu

Description :

VMSubnetID : 15711845

VMSubnetType : GREWindowsNetworkVirtualization

VMNetwork : Customer Address Space

SubnetVLans : {192.168.1.0/24}

ServerConnection : Microsoft.SystemCenter.VirtualMachineManager.Remoting.ServerConnection

ID : 19d6d1a5-74e3-49f1-80fc-6f3316f429e0

IsViewOnly : False

ObjectType : VMSubnet

MarkedForDeletion : False

IsFullyCached : True

Network Virtualization Gateway

At some point traffic will need to leave or enter the Routing Domain. You can think of traffic to and from the Internet or traffic to and from your current on-premise network. This means you will need some kind of gateway. There are two different kinds of Network Virtualization Gateways.

  1. Routing gateway. Routes traffic between the virtualized network and the physical network. This gateway is mainly used in private clouds.
  2. VPN gateway. Routes traffic between the virtualized network through a VPN tunnel to an on-premise physical network. This gateway will be more for hosters.

Hyper-V Network Virtualization gateway support in System Center VMM 2012 SP1 is expected later this year. The gateway is more than a simple Windows 2012 VM although Windows 2012 hosts with Windows 2012 VMs can be the basis for a gateway. Microsoft is looking at their partners for Windows based gateway appliances.

Figure 11

It is possible to use a RRAS S2S VPN VM with 2 virtual NICs. However, you would need to populate the WNV filter with the appropriate policies as well as configuring RRAS in the VM. You can have a look at the PowerShell script here. The gateway must have all policies for all the virtual subnets. This means that every time a VM moves (live migration) and gets a new PA address the gateway must be updated. This update must happen quickly for existing TCP connections otherwise the connection will be dropped.

Conclusion

Whether you are a hoster or own a private cloud, network virtualization can really scale and bring true multitenancy. There is still a lot of ground to cover but this really is built for the future, ready now!

Additional Resources

Two great videos can be found here :

Hyper-V Network Virtualization for Scalable Multi-Tenancy in Windows

Network Management in Microsoft System Center 2012 SP1 – Virtual Machine Manager

Read more in the TechNet library :

Network Virtualization technical details

Hyper-V Network Virtualization Gateway Architectural Guide

4 Comments

  1. August 27, 2013    

    Is it possible to share a logical switch between two or more Hyper-V hosts? I’m trying to create a lab environment where I have two Hyper-V hosts and three tenants (HOSTER, ACME, and BOSCO). I want the tenants to be host independent (failover) and I want ACME and BOSCO to be isolated from one another but able to access HOSTER resources. Is there a way to achieve that?

    • August 28, 2013    

      Hi Brian, if you are using network virtualization you are forced to route to HOSTER outside the NVGRE environment. When you add virtual subnets to the same routing domain they a routable by design, so you should use separate routing domains. If you are using VLANs you can use Private VLANs to achieve your design. For some info on PVLANs take a look here http://technet.microsoft.com/en-us/library/jj679878.aspx
      Hope this helps,
      Marc

  2. August 29, 2013    

    Thanks Marc. Here is my current configuration:

    - 2 Hyper-V Hosts: HV01 & HV02
    - ACME resides on HV01
    - BOSCO resides on HV02
    - HOSTER resides on HV02

    I’m using a internal virtual switch for each tenant:
    - vSwitch_ACME
    - vSwitch_BOSCO
    - vSwitch_HOSTER

    I also have an external vSwitch for each Hyper-V host: vSwitch_External.

    I create a Vyatta router for each tenant and assign two NICs.

    - ACMEROUTER
    – vSwitch_External
    – vSwitch_ACME
    - BOSCOROUTER
    – vSwitch_External
    – vSwitch_BOSCO
    - HOSTERROUTER
    – vSwitch_External
    – vSwitch_HOSTER

    When I build my VMs, I use the internal vSwitch for that VM and configure Vyatta to route the traffic. This gives me the control I’m looking for but I cannot figure out how to provide complete VM and host failover functionality with this design.

    Any ideas on that?

  3. January 30, 2014    

    >>>If the VSID is 0, the packet will pass-through the Windows Network Virtualization (WNV) module.

    Your assumptions are incorrect! If a CA is not assigned with a VSID that means it is not participating in the NVGRE netwokring – as simple as that!

    So you can run VMs which are “not” participating in the HNV and can be connected to the same NVGRE switch!

    /Mark

  1. Windows Azure Services for Windows Server on November 9, 2012 at 12:16
  2. Windows Azure Services for Windows Server | Marc van Eijk on January 20, 2013 at 23:40
  3. Installing and configuring Service Management Portal and API – Part 2 on March 24, 2013 at 23:44
  4. Windows Server 2012 R2 and System Center 2012 R2 – Hyper-V Network Virtualization Whitepaper on November 22, 2013 at 17:59
  5. Windows Server 2012 R2 and System Center 2012 R2 – Hyper-V Network Virtualization Whitepaper | Marc van Eijk on November 22, 2013 at 18:12

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code class="" title="" data-url=""> <del datetime=""> <em> <i> <q cite=""> <strike> <strong> <pre class="" title="" data-url=""> <span class="" title="" data-url="">