Powered by System Center
At Inovativ, the company I work for, we more or less use Windows Azure Pack in production now. We have begun to operate as a hoster and each consultant is a tenant in our Inovativ Cloud. They can go to the Windows Azure Pack tenant portal and consume the resources we have available in our cloud. If you don’t set any rules your tenants are likely to grab all the resources that are available. Currently we have not yet set any quotas, but I’m afraid we will have to very soon.
One problem that we encountered was that VMs which were created outside of the portal did not have the correct Owner and User Role. This would mean that those VMs were not visible as owned by a certain tenant in the Windows Azure Pack tenant portal. Also when you set quota on for instance the number of VMs that could be created or the amount of memory that could be consumed, only the VMs were counted that were actually owned by the tenant.
This would be a serious problem if you are a hoster an wanted to move existing VMs to a tenant’s subscription or service plan as it is called in Windows Azure Pack. Another problem that we think is linked to this is that Cloud Cruiser which collects usage information for billing or showback is not able to process the usage records.
This is what it looks like from the admin portal when you go to VM Clouds and find all VMs for a certain tenant and subscription to a tenant plan. As you can see there not all of my VMs are listed.
It was quite easy to give access to the manually added VMs via the VM properties in Virtual Machine Manager 2012 R2. The first requirement is assign the VM to the same VMM Cloud that is used in Windows Azure Pack. If the VM does not have an Self-Service owner, you can select one. The problem however is that the only accounts you can choose are Active Directory accounts. This is exactly what we don’t want.
See for the latest updates the end of this post.
In this post Marc van Eijk points out connectivity issues with VMs and vNICs. At random virtual machine or vNIC would loose connectivity completely. After a simple live migration the virtual machine would resume connectivity.
Marc has already logged a support case at Microsoft and HP and they are investigating this issue. Last week I also discovered this issue, here is my configuration:
Currently we experience network connectivity issues with one of our cluster networks in a Windows Server 2012 R2 Hyper-V cluster environment.
Our environment is as follows:
- Two HP BL460 G7 servers (name of the servers: Host01 and Host02)
- 6x HP NC553i Dualport Flexfabric 10GB Converged Networkadapters (only 2 active)
- Installed with Windows Server 2012 R2 Hyper-V (full edition)
- Configured in a Windows Failover Cluster
The NICs are installed with the following driver
Driver: Emulex, Driver date: 5-6-2013, Driver version: 220.127.116.11
We have configured a switch-independent NIC team with dynamic loadbalancing with 2 NIC team members. Upon this NIC team we have configured a vswitch.
In this vswitch we have created three vNICs of type Management OS:
- Live Migration
- Cluster CSV
Every NIC is configured in a separate VLAN. Only the Live Migration network may be used for Live Migration traffic (Configured in Windows Failover Clustering).
The initial installation and configuration of Hyper-V and the Windows Failover Cluster was OK. Over all the networks, communication between the hosts in the cluster was possible.
The Cluster Validation Wizard runs successfully without any warning or error.
After the installation of the Hyper-V cluster we start creating and installing the virtual machines. No problems at all, till we build a specific VM called VM06. This VM was created on the host Host01.
When the VM resides on this host everything is OK. As soon as we move this virtual machine (via Live Migration) to the host Host02 the cluster network called Live Migration went down and communication on this network between the two Hyper-V hosts is not possible anymore. When we move the virtual machine back to Host01 the cluster network called Live Migration comes back online. Also when we shut down the virtual machine when it resides on node Host02 the cluster network called Live Migration comes back online.
When we change the NIC teaming configuration to a Active/ Standby configuration, as Mark described in his blog, this network issue does not appear.
Microsoft requested us to disable Large Send Offloading: “Get-NetadapterLSO | Disabel-NetadapterLSO” (with NIC teaming in active/ active). However the issue is still there.
Update 11-26-2013 14:45: After disabling RSS and RSC (which does not change te situation) Hans suggest to disabling VMQ. We used PowerShell to disable VMQ on all interfaces: “Get-NetAdapterVmq | Disable-NetAdapterVmq” …. and yes disabling VMQ does the trick. Off course this is not a solution but only a workaround. These findings are logged in the case @ Microsoft and they will investigate this futher.
Update 12-02-2013 10:45: After applying update KB2887595-v2 to both of our Hyper-V nodes the network problems with our Live Migration network are gone. Even with VMQ enabled the network keeps up and running. However this update does fix the problem for our situation but not for the situation that Marc describes. So it seems that we’ve two different issues here.
We, Hans, Mark en me, will continue to investigate this issue and will update you on www.hyper-v.nu!
My first post on hyper-v.nu was on Network Virtualization. The technology was unknown to most at the time and the information was scarce. Nevertheless the possibilities of the solution looked very promising. Now, more than a year later, Network Virtualization has evolved. Windows Server 2012 R2 provides all the building blocks to run Network Virtualization in a production environment. With a lot of moving components Network Virtualization might seem daunting.
To reduce the complexity, a couple of community members have put a lot of effort in collecting the proper information, testing implementations and with the input from the relevant product teams they have written a whitepaper to get you started with Network Virtualization. The whitepaper is created by Kristian Nese (MVP CDM) and Flemming Riis (MVP CDM) and was reviewed by Daniel Neumann (MVP CDM) and Stanislav Zhelyazkov (MVP CDM). I had the pleasure of exchanging ideas and experiences with them.
If you want to get started with Network Virtualization this whitepaper is a great way. It provides an excellent step by step to configure all the required components and also points out the potential pitfalls. No more time to waste. Download the whitepaper Hybrid Cloud with NVGRE (WSSC 2012 R2) and start exploring the world of Network Virtualization today!
A couple of week ago Windows Server 2012 R2 and System Center 2012 R2 reached the GA milestone. We started with a LAB environment for validation our designs. During the deployment we were experiencing connectivity issues with VMs and vNICs. At random virtual machine or vNIC would loose connectivity completely. After a simple live migration the virtual machine would resume connectivity. After verifying our VLAN configuration a couple of times things even got more weird. After live migrating the virtual machine back to the host were it lost connectivity, it still was accessible. Most virtual machines were functioning properly and there was no clear pattern in the what and when a virtual machine was having the issue. And without a way to reproduce the issue on demand it was complex to troubleshoot.
A week later I did an implementation at a customer site. The design was based on a two node Windows Server 2012 R2 Hyper-V cluster with System Center workloads and a five node Windows Server 2012 R2 Hyper-V cluster for production workloads. The nodes of the production cluster were deployed using the bare metal deployment process in System Center VMM 2012 R2. All the hosts were deployed successfully, but we were having issues creating a cluster from these nodes. The cluster validation wizard showed connectivity issues between the nodes. As you might know from my previous blog on bare metal deployment, System Center VMM 2012 R2 can only create a NIC Team with the Logical Switch if a vSwitch is created on top of the NIC team. This required vNICs in the ManagementOS for host connectivity. After validating the VLAN configuration we rebooted the host. Connectivity resumed when a host was rebooted , but at random different hosts lost connectivity again. We were experiencing a similar situation as in our lab environment.
The was another similarity in the two environments. The customer site and our lab consisted of an HP BladeSystem c7000 with BL460c Gen8 blades that contained HP FlexFabric 10Gb 2-port 554FLB Adapters. These BladeSystems use Virtual Connect technology for converged networking. We upgraded our Virtual Connect to the latest version 4.10 before implementing Windows Server 2012 R2, but the customer was still running version 3.75. The HP FlexFabric 10Gb 2-port 554FLB Adapter is based on Emulex hardware and an inbox driver was provided by Microsoft with version number 10.0.430.570. After contacting my friend Patrick Lownds at HP he provided me with a link to the Microsoft Windows Server 2012 R2 Supplement for HP Service Pack. Running this did not provide any update to drivers. The details of the HP FlexFabric 10Gb 2-port 554FLB Adapter showed that this is Emulex hardware. A search on the Emulex site provided an newer version of the driver. After installing the new driver with version 10.0.430.1003 the issue occurred again.
We submitted a case with Microsoft and I have been debugging this issue with a Software Development Engineer from Microsoft (who has verified my blog series on NIC Teaming about a year ago) for the last week. I must say Kudos to Silviu for his assistance every evening this week and Don Stanwyck for communicating with HP. I also reached out to a couple of community members to know if the issue sounded familiar. Rob Scheepens (Sr. Support Escalation Engineer at Microsoft Netherlands) was aware of another customer with the same issue on exactly the same hardware and yesterday evening I was contacted by another one. Same issue, same hardware. This morning I was pinged by Kristian Nese who has a repro of the issue with 2x IBM OCe11102-N2-X Emulex 10GbE in a team (created from VMM) with Emulex driver version 10.0.430.570.
The issue is not solved yet but I though that a quick post would prevent a lot of people from wasting valuable time on troubleshooting. Please submit a case at the hardware vendor as this would create more priority at their site. I’ll update the blog with any progress or relevant information.
A possible temporary workaround seems to configure the NIC Team members in Active/Passive. I have not been able to test and confirm this.
Have you ever wondered why you DO see performance data in Windows 8/8.1 under the performance tab in Task Manager, but DON’T see this same information in Windows Server 2012/2012 R2?
Well I kind of missed seeing that information in Server but never really bothered to really investigate.
Windows Server 2012/2012 R2
I know the solution is too simple for words, but just in case you also forgot about this good old Diskperf command which you can run from an administrative command or Powershell prompt:
Diskperf –Y to switch on the Disk Performance counters. You don’t need an academic degree to figure out how to disable them again.
So much for an easy tip with great results.
A hotfix has been released today for Windows Server 2012 Hyper-V servers which are unable to access LUNs over a Synthetic Fibre Channel after a VM is live migrated to another host in the cluster.
This problem can occur if the following conditions are met:
- You have two Windows Server 2012-based computers that have the Hyper-V role installed
- You install a virtual machine on one of the Windows Server 2012 Hyper-V hosts
- You set up a guest failover cluster, and then you make the virtual machine a cluster node
- The virtual machine is configured to access LUNs over a Synthetic Fibre Channel
- You try to perform live migration to move the virtual machine to another host.
This issue is caused by the inability of the Hyper-V host to restore the Synthetic Fibre Channel LUN on behalf of the virtual machine during live migration.
The problem applies to Windows Server 2012 Standard and Datacenter.
For further details look up the Microsoft Support Article: http://support.microsoft.com/kb/2894032
5nine Software announces complimentary 5nine Security for Microsoft Hyper-V with built-in Security and Compliance Scanner
5nine Software, the only vendor delivering management and agentless/host-based security and compliance products for Windows Server and Microsoft Hyper-V, has released a complimentary 5nine Security for Hyper-V with built-in 5nine Security and Compliance Scanner.
When I started Hyper-V.nu back in 2008 I could never have imagined that it would be this successful. A lot of people have contributed to this success in the past. The current team consisting of Hans Vredevoort (who joined me in 2008), Marc van Eijk en Peter Noorderijk, the rising stars of Hyper-V.nu, but also the people at the local Microsoft Netherlands office that offered their advice on a regular basis.
The last couple of months I have been thinking about my own future within the Hyper-V.nu team, especially since I’m also active in the UC community, after all I am an Exchange Server MVP.
Looking at the plans we have with Hyper-V.nu the upcoming months I think it’s a good time for me to step back and make room for Marc, Peter and Hans and give them all the option they need to further extend the success of Hyper-V.nu.
I will continue my work in the UC Community, you can follow me on my own site www.jaapwesselius.com or on Twitter via @jaapwess. And don’t forget, you can still find me at a lot of the major Microsoft events of course.
I want to thank my Hyper-V.nu colleagues, friends and everybody that contributed to the success of Hyper-V.nu and hope to see you again sometime in the (near) future.
If you are a regular reader of this blog you have noticed that a lot of content that is written is related to Windows Azure Pack (WAP for short), formerly known as Windows Azure Services for Windows Server. There are also numerous good other blogs as well as videos about WAP written by fellow MVPs and the different Microsoft Product Teams.
I was already collecting my own list of WAP content but decided to put it in a Wiki on Microsoft TechNet. The WAP Wiki contains the following headings:
- Windows Azure Services for Windows Server (older content but still useful)
- Windows Azure Pack
- Service Provider Foundation
- Usage and Billing
- VM Role
- Service Management Automation
- Hyper-V Network Virtualization Gateway
- Hyper-V Network Virtualization
- Disaster Recovery
Of course if you login to the Wiki with your Microsoft Account, you can add content to it. Please mark the new content with (New!) so it is easy to see what has been added.
With the General Availability of Windows Azure Pack more organizations are interested in or are already implementing the complete CloudOS. Compared to the previous release Microsoft has put more effort in documentation for the product. After you have implementing your first lab environment (and you should, before trying anything like this in production) you will see that the default URLs for accessing the Admin Site is configured on port 30091 and the Tenant Site is configured on port 30081. In the previous release of Windows Azure Pack (named Windows Azure Services for Windows Server) you could just change the port to 443 in IIS, assign a public certificate to the website and you were done.
In Windows Azure Pack Microsoft introduced the possibility to use Active Directory Federations Services (ADFS) for authentication. This functionality enables a single sign on experience for end users. ADFS (besides many other features) makes Windows Azure Pack also interesting for enterprise organizations wanting to provide “Service Provider like” offerings to their internal customers.
The integration of ADFS required some changes to the validation procedure within Windows Azure Pack. Even without configuring ADFS the Admin Site and the Tenant Site now have their own dedicated authentication sites. In this blog I’ll describe the required steps to change the default URLs (name and port number) to public URLs.
Before we start it is good to understand that it is not possible to have multiple sites on a single server listening on the same default SSL port 443. I have even tried to add additional IP addresses to the same server NIC and bind each Website to a different IP Address (all on port 443). This will work within the same subnet, but will not function when accessing through NAT externally. For this configuration we are looking at four websites.
- Admin Site
- Admin Authentication Site
- Tenant Site
- Tenant Authentication Site
If you want to access these sites externally over a default SSL connection on port 443 you would need a virtual machine for each website (four in total) and four public IP addresses. For a production environment you would not provide external access to the Admin Site. The Admin Site and the Admin Authentication Site can be installed on a single machine. The Tenant Site and the Tenant Authentication site are probably going to be accessed from the internet. You can use a wildcard certificate provided by a public Certificate Authority for all websites. For this blog I will reference three virtual machines.
- wap01 (Admin Site & Admin Authentication Site)
- wap02 (Tenant Site)
- wap03 (Tenant Authentication Site)
In a more robust production environment you would double these servers and make their services high available with a Load Balancer (NLB or Hardware Load Balancer). I have verified that these configuration steps are the same for a scale out scenario using Load Balancing.
The Admin Site
In a default configuration the Admin Site is configured on port 30091 and the Admin Authentication Site is configured on port 30072. Accessing the the admin site through a browser can be divided in to the following seven steps.
- You enter the NetBIOS name of the server with the port configured in IIS for the Admin Site
- Windows Azure Pack gets the Admin Site Fully Qualified Domain Name (FQDN) and port number from the database and notifies the browser
- The browser is redirected to the FQDN and port number of the Admin Site configured in the Windows Azure Pack database
- The Admin Site detects you do not have the correct token to validate and notifies the browser with the FQDN and port number of the Admin Authentication Site configured in the database
- The browser is redirected to the FQDN and port number configured for the Admin Authentication Site in the Windows Azure Pack database
- After validation a token is provided and the Admin Authentication Site gets the FQDN and port number from the database for the Admin Site and notifies the browser
- The browser is redirected to the FQDN and port number configured for the Admin Site
For this example I want access the Admin Site on https://admin.hyper-v.nu and the Admin Authentication Site on https://admin.hyper-v.nu:30072. I will describe each configuration step referencing the numbers in the picture.