Posts by Hans Vredevoort

Another Update on the VMQ Issue with Emulex and HP

I wrote my last blog on this topic 6 months ago. Meanwhile I have seen several firmware and driver updates from Emulex, usually followed by HP several weeks later. I’m still talking about the ongoing VMQ problems with  HP/Emulex 554FLB CNAs in HP BL460c Gen 8 blade servers in c7000 blade enclosures. Meanwhile I have tested several incarnations of this firmware/driver combination in our own Azure Pack cloud environment. I already found out in previous attempts, that a switch to new Emulex firmware and HP/Emulex drivers, including the switch from VMQ disabled to enabled, can be disruptive and hosts and VMs need to be restarted if things turn bad.

And it did turn bad on all four occasions I tried until now. In July 2014 our hopes were raised by Emulex when Mark Jones from Emulex posted the first update on July 23 2014. I tried them out but not until HP released their OEM specific driver for the CNA. It didn’t take me very long to find out that my test guest cluster quickly got disconnected during a live migration with one node being evicted from the cluster during the black-out. First opportunity: failed

On August 4 a special release was issued for non-OEM Emulex branded cards. I decided not to try this out on our HP/Emulex CNAs. I later found out that some people who tried had to call in HP to replace their servers. Second opportunity: Ignored

Just over a month later, another update appeared on the Emulex blog letting us know that HP versions of the firmware/drivers had been made publicly available. Previous versions had been considered but we found a mystery insider in the comments of our blog stating:

Without revealing NDA information, the HP driver released in September is not really a fix, it is also a workaround, with some caveats that don’t appear in the published notes. My understanding is that the fix delivered by Emulex was considered unstable by HP, and a truly “fixed” driver won’t be released until sometime in Q1 2015.

I didn’t have to think long. Third opportunity: Ignored

image

On October 21st 2014, another update appeared on the Emulex blog. I had become very suspicious of again another update and lacked the time or drive to once more spend many hours testing this version. What changed my mind was when fellow Hyper-V MVP Patrick Lownds sent a message to one of the MVP distribution lists, letting us know that HP had released their OEM version of the HP/Emulex 554FLB firmware and driver. READ MORE »

New blood for Hyper-V.nu

I’m very happy to announce that three very talented young men have agreed to officially start blogging for Hyper-V.nu. This time not as guest bloggers, but as official Hyper-V.nu bloggers.

If you check Hyper-V.nu on a regular basis, you will have noticed that in recent weeks very few blogs have appeared on the site. This is largely due to the enormous success of Windows Azure Pack which is more or less keeping us fully occupied.

Apart from the many CloudOS related projects that Peter Noorderijk, Marc van Eijk and I run on a daily basis, we also maintain the Azure Pack Wiki, some of the Hyper-V hotfix lists, do presentations at IT events, write books, blog for the MS Building Clouds blog, evangelizing hybrid cloud with Azure Pack, and as MVPs have very regular meetings with Microsoft product teams. In other words, there are not enough hours in the day to make this all work.

So that’s enough explanation why we need more bloggers to help fill the pages of Hyper-V.nu. In the previous year, you may have already seen several guest blogs by Darryl van der Peijl, Ben Gelens and Mark Scholman. All three also happen to be colleagues at INOVATIV, but that is not why they join Hyper-V.nu. It is their real world experience with Azure and Azure Pack technology that makes them special and why we let them join as bloggers.

Let me quickly introduce Darryl, Ben and Mark.

Darryl van der Peijl

image

Darryl was working for a service provider where I met him during the deployment of what was then called Windows Azure Services for Windows Server. Darryl is a very clever young man and quickly came up to speed with the Microsoft System Center and Cloud offering. He is also very proficient in PowerShell which is of course a must-have knowledge these days. Darryl has been implementing Azure Pack ever since, often sharing scripts he developed such as the Azure Pack Tool , the Windows Azure Pack Update Script on the TechNet Gallery. After several guest blogs, he just submitted his first blog on Scale-Out File Servers.

Darryl tweets at @DarrylvdPeijl and has his own blog at http://www.darrylvanderpeijl.nl/

Ben Gelens

image

I met Ben virtually via Twitter and was amazed at the quality of this blogs on VMM, storage, bare metal deployment. I praised his blogs a couple of times and got to know more of what Ben was doing. He happens to be also very versed at PowerShell and PowerShell Workflow, which is as you might know the center of focus in Service Management Automation (SMA), which was first exposed via the Windows Azure Pack admin portal. We then talked about several guest blogs about Bare Metal Post-Deployment using SMA and VM Role.

Ben tweets at https://twitter.com/bgelens and blogs at http://mssecbyben.wordpress.com/

Mark Scholman

image

Mark also quickly made fame while promoting his blogs via Twitter on networking, Azure Pack, NVGRE and Network Virtualization. These are all qualities which are highly desirable if you start implementing Windows Azure Pack in the real world. Mark recently starting investigating Azure Pack Websites for one of the projects we currently engage in. Learning and writing always ends up in a great blog for Mark and the Installing and Configuring HA Azure Pack Websites series is just one example.

Mark tweets at https://twitter.com/markscholman and blogs at http://sysctr.nl/

Let me finish by saying that these three guys are worth following and hopefully they’ll share many blogs on Hyper-V.nu.

Hans Vredevoort
@hvredevoort

Additional Background on the VMQ Issue with Emulex and HP

Today I had a conference call with the people from Emulex responsible for the network adapters, firmware and drivers. As many of you know we’ve had a long lasting issue with HP/Emulex 554FLB CNAs in HP BL460c Gen 8 blade servers in c7000 blade enclosures. After we had replaced Windows Server 2012 with Hyper-V by Windows Server 2012 R2 with Hyper-V on the same hardware, we started to notice virtual machines losing connection. We have multiple guest clusters on top of the Hyper-V clusters and sometimes during Live Migration of one of the cluster nodes, we would see that connectivity was lost, even to the point that a cluster node would be forced to leave the cluster and come back later when the network connection was re-established. In fact that was our single best test to reproduce the problem.

Marc van Eijk en Peter Noorderijk wrote blogs about it and together they got over 200 comments from customers all over the world seeing the same problem, the majority of them having Emulex through an OEM like HP, Dell, IBM and Cisco. In fact we also read similar cases with HP rack servers, other NICs and different hardware combinations which led me to believe there could also be an issue in the networking/teaming stack in Windows Server 2012 R2.

During the months of November and December 2013 we collected a list of 10 registered support cases with either HP, Microsoft or both. At that time my primary contact was a senior escalation engineer at Microsoft who was able to collect several customer cases mostly from European customers. Unfortunately we could not register a support case with HP ourselves because we did not have a support contract with them. But the amount of customers that discovered they had the same problem, grew and grew. There must have been hundreds of open support calls with both HP and Microsoft and the storm of discontent was growing.

According to HP they were dependent on Emulex and HP did not get any feedback either. Also Microsoft was left in the dark for a long time.

READ MORE »

System Center, Hyper-V, Azure and Meat

If you like System Center, Hyper-V, Azure as well as meat, don’t forget to register for System Center Summer Night “The MasterChef edition” which is rapidly approaching. In 10 days from today, nine experts, seven of them are MVPs, will present five interesting presentations.

Because we have plenty of space left we invite those who have registered to bring a friend without additional cost. If you haven’t registered yet, please do and have a great afternoon which is completed with a nice barbecue.

Date of event

June 26th

Program

15:00 – 16:15 How many System Center fits on one grill by Ronny de Jong / James van den Berg [MVP] / Helmer Zandbergen / Marc van Eijk [MVP] / Dieter Wijckmans [MVP]

16:15 – 17:30 How Service Manager can do everything you need – a best-of-the-best Swiss cheese selection by Marcel Zehner [MVP]

17:30 – 17:45 Break

17:45 – 19:00 Light up the fire on your Hyper-V by Hans Vredevoort [MVP] & Peter Noorderijk on Hyper-V Architecture

19:00 – 20:15 Become a Masterchef on Microsoft Azure Automation by Maarten Goet [MVP]

20.15 BBQ time

This event is organized by SCUG.nl and Hyper-V.nu

Registration

Please register at https://www.eventbrite.nl/e/tickets-system-center-summer-night-2014-9265847399

Location

Unieplaza
Multatulilaan
4103 NM Culemborg

Hyper-V Amigos Back in Quartet Formation

Back in 2011, four Hyper-V MVPs decided to take on a server virtualization master class series focusing on Hyper-V for which they adopted the Spanish sounding name “The Hyper-V Amigos”. In the previous months German Hyper-V MVP Carsten Rachfahl, well known for his Hyper-V podcasts and videos, had already made three episodes with Belgian Hyper-V MVP, Didier van Hoye aka @WorkingHardInIT which gave us a lot of background of both Carsten and Didier in Episode 1, as well as some great explanatory showcasts on Unmap and Live Migration in Windows Server 2012 R2 in Episodes 2 and 3.

This time Carsten also invited Aidan Finn, Hyper-V MVP out of Ireland and myself to what seemed like a reunion. Three of the MVPs had visited TechEd North America and in Episode 4 they look back on how they came to know as The Hyper-V Amigos and what their thoughts were on the TechEd 2014 event.

I invite you to watch the next episode of The Hyper-V Amigos:
http://www.youtube.com/watch?v=PFURtvxbFaU&feature=youtu.be

Update Rollup for June 2014

The update rollup for June 2014 fixes the issues that are documented in the following Microsoft Knowledge Base (KB) articles:

  • (http://support.microsoft.com/kb/2959146/ )

    Update for data deduplication to improve scalability in Windows Server 2012 R2

  • (http://support.microsoft.com/kb/2960387/ )

    You are prompted for BitLocker recovery key when Windows enters the automatic repair process

  • (http://support.microsoft.com/kb/2961977/ )

    “Hyper-V Replica Cluster Broker is not installed” error when you replicate private clouds to Windows Azure

  • (http://support.microsoft.com/kb/2963523/ )

    DNS server crashes after you install update 2919355 for Windows Server 2012 R2.

  • (http://support.microsoft.com/kb/2964723/ )

    Connectivity lost between two nodes when a node reconnects to a Windows Server 2012 R2-based cluster

  • (http://support.microsoft.com/kb/2964724/ )

    CPrepSrv.exe process crashes or Failover Cluster Manager freezes when you validate storage in Windows Server 2012 R2

  • (http://support.microsoft.com/kb/2964725/ )

    Removed nodes can access shared disk resources unexpectedly in Windows Server 2012 R2

  • (http://support.microsoft.com/kb/2964729/ )

    You cannot stop the cluster service on a Windows Server 2012 R2-based failover cluster

  • (http://support.microsoft.com/kb/2964730/ )

    Storage spaces take a long time to move to another node after a node fails on a Windows Server 2012 R2 failover cluster

  • (http://support.microsoft.com/kb/2964732/ )

    STS passive sign-in fails when a sign-in request is sent to a Windows Server 2012 R2-based STS server through STS proxy

  • (http://support.microsoft.com/kb/2964733/ )

    AD FS device authentication is slow or fails in Windows Server 2012 R2

  • (http://support.microsoft.com/kb/2964735/ )

    Authentication failures and event 422 when AD FS STS servers and AD FS proxy servers are in Windows Server 2012 R2

  • (http://support.microsoft.com/kb/2964804/ )

    Long wait when you first open File Explorer in Windows RT 8.1 or Windows 8.1

  • (http://support.microsoft.com/kb/2964814/ )

    Virtual machine network fails when you start the second VM on a Windows Server 2012 R2-based Hyper-V server

  • (http://support.microsoft.com/kb/2964951/ )

    Windows Update does not download drivers for shared printers in Windows 8.1 or Windows Server 2012 R2

  • (http://support.microsoft.com/kb/2965074/ )

    Error occurs when you run Get-VirtualDisk|Get-ClusterResource cmdlet in Windows 8.1 or Windows Server 2012 R2

  • (http://support.microsoft.com/kb/2965174/ )

    OneDrive improvement update for Windows RT 8.1 and Windows 8.1: June 2014

  • (http://support.microsoft.com/kb/2965492/ )

    “0x80041013″ error on a WMI provider in Windows RT 8.1, Windows 8.1, or Windows Server 2012 R2

  • (http://support.microsoft.com/kb/2965699/ )

    “There was a problem” error when you redeem a promotional CSV token in Windows 8.1 or Windows Server 2012 R2

  • (http://support.microsoft.com/kb/2965770/ )

    Cannot select Chinese suggestion words from on-screen keyboard in Windows 8.1 or Windows Server 2012 R2

  • (http://support.microsoft.com/kb/2966039/ )

    Settings are migrated incorrectly after you refresh the system by using PBR in Windows RT 8.1 or Windows 8.1

  • (http://support.microsoft.com/kb/2966055/ )

    Logon UI crashes when you connect to a remote server that is running a Windows Server 2012 R2 Core installation

  • (http://support.microsoft.com/kb/2968633/ )

    Update to improve the OneDrive experience in Windows RT 8.1 and Windows 8.1

  • (http://support.microsoft.com/kb/2960837/ )

    Excel freezes when you convert Japanese characters in Windows

  • (http://support.microsoft.com/kb/2956014/ )

    Audit event ID 4661 triggers an invalid XML error in a Windows Server 2012 R2 or Windows Server 2008 environment

  • (http://support.microsoft.com/kb/2950080/ )

    “The CA certificate could not be retrieved, element not found” error occurs when the CA server host name is longer than 52 characters

  • 2936341(http://support.microsoft.com/kb/2936341/ ) The WebClient service does not send cookies in Windows

Workaround for Updating a Hyper-V Generation 2 VM with KB2920189

During my last updating round, I noticed that a number of VMs in my Windows Azure Pack lab, had problems with security update KB2920189. Reading the Microsoft Security Advisory, it states that Microsoft is revoking the digital signature for four private, third-party UEFI (Unified Extensible Firmware Interface) modules that could be loaded during UEFI Secure Boot.

These UEFI (Unified Extensible Firmware Interface) modules are partner modules distributed in backup and recovery software. When the update is applied, the affected UEFI modules will no longer be trusted and will no longer load on systems where UEFI Secure Boot is enabled. The affected UEFI modules consist of specific Microsoft-signed modules that are not in compliance with our certification program and are being revoked at the request of the author.

Microsoft is not aware of any misuse of the affected UEFI modules. Microsoft is proactively revoking these non-compliant modules in coordination with their author as part of ongoing efforts to protect customers. This action only affects systems running Windows 8, Windows 8.1, Windows Server 2012, and Windows Server 2012 R2 that are capable of UEFI Secure Boot where the system is configured to boot via UEFI and Secure Boot is enabled. There is no action on systems that do not support UEFI Secure Boot or where it is disabled.

I concluded that this update only targeted Hyper-V Generation 2 VMs with Secure Boot enabled, which was in fact the case for all VMs involved.

No matter how many times I tried, each update ended up failed. READ MORE »

Out-of-Band Management in a Windows Server 2012 R2 Hyper-V Environment

Let me introduce Darryl van der Peijl who wrote this guest blog on Out-of-Band Management in Windows Server 2012 R2 Hyper-V. Darryl is working for Inovativ since April of this year and is working in the same team as Marc van Eijk and myself focusing on CloudOS, System Center Virtual Machine Manager and Windows Azure Pack.

_____________________________________________________________________________________

In this post we will discuss Out-of-Band Management of virtual machines running on Hyper-V, and a glimpse into the future. Hyper-V in Windows Server 2012 R2 has a large number of significant improvements, but there’s a new feature called Guest Services that hasn’t been much in the spotlight yet. I’ll provide a quick overview of Hyper-V Guest Services and also an example of how it can be used to perform out-of-band operations.

Guest Services

Guest Services are disabled by default on VMs. In order to use Guest Services, it needs to be enabled on each VM which will make use of it. To enable Guest Services on each VM, you can configure the checkbox setting shown in the screenshot below.

And of course you can enable Guest Services with PowerShell, using the Enable-VMIntegrationService command.

Get-VM –Name “Test VM” | Enable-VMIntegrationService -Name “Guest Service Interface”

Guest Services leverages the Hyper-V Virtual Machine Bus (VMBus) to which each VM is connected. The VMbus is a communication mechanism used for inter-partition communication and device enumeration on systems with multiple active virtualized partitions. To bring it the easy way: The hypervisor and virtual machines are communicating with each other through the VMBus. For now, the only Out-of-Band operation Guest Services is supporting is the Copy-VMfile cmdlet.

With this cmdlet you can copy a file from a host into the VM.

$VM = Get-VM -Name “Test VM”
Copy-VMFile -VM $VM -SourcePath “C:\scripts\powershellscript.ps1″ -DestinationPath “C:\scripts\powershellscript.ps1″ -CreateFullPath -FileSource Host -Force

Out-of-Band Management

Out-of-band management (OOB), sometimes called lights-out management, involves the use of a dedicated management channel for device maintenance. The term “Out-of-Band” in this blog is actually referring to a way to manage machines without using network. Now you might think “Manage without network? why?”

With routed VLANs you can easily manage your VMs using remote PowerShell, but with network virtualization, although a very cool technology, it also introduces some big challenges. Network virtualization isolates the network meaning nobody can access the network without being in it. You may encounter a similar scenario with isolated development environments with VMs running in a DMZ, behind a Firewall or using ACLs. So, how are you going to manage VMs which you can’t reach through network connectivity?

Right! Out-of-Band Management

So the goal is to manage VMs running on Hyper-V without having to worry about what network the VM is on, or if it’s even connected to any network.

A Glimpse into the future

My prediction is that the Virtual Machine Manager and Hyper-V product teams are working hard to get this type of management arranged. Virtual Machine Manager will be the center of Out-of-Band management, shooting commands through the VMM agent to the Hyper-V host. The Hyper-V server will pass these commands (PowerShell of course) through the VMBus to the VM using Integration Services.

This way you can manage any VM, connected to any network.

Other System Center components could benefit from Out-Of-Band management also: monitor a VM while the network is down with Operation Manager, change a VM IP with Configuration Manager etc. Since much is focused on Microsoft Azure, I am assuming this kind of functionality will also be available on Microsoft Azure and Windows Azure Pack through the API’s.

How cool would it be to pass PowerShell commands from your laptop on a public airport Wifi to your VMs on Azure?

Is there a catch?

The question is if this functionality poses a potential risk for your VMs or even your whole environment. What would happen if the security of the APIs is breached, would it be possible to execute commands from within your VMs from anywhere. The Hyper-V Integration Services are running as a services with ‘Local System’ privileges, so in theory there will be no security restrictions within the VM.

The above paragraph is of course speculative, but if a file copy through the VMBus is possible… I suspect anything is.

Let me know what your thoughts are on this subject, leave a comment!

Offline Update of Windows Azure Pack and 3rd Party Components

Recently I needed to update a Windows Azure Pack installation from RTM to Update 2. Because the servers had no Internet access, I needed an offline approach.

One way is to download the Windows Azure Pack 2013 Update 2 components, extract the files and install them one by one, thereby carefully checking that only the installed components are updated. It is very easy to make a mistake and add unwanted ones on a WAP portal server.

Components

Because this approach only works for the WAP 2013 Update 2 components and doesn’t give you the partner components such as Gridpro and Cloud Cruiser, the offline Web Platform Installer approach is the best choice. In fact WebPI has been upgraded from version 4.6 to 5.0 containing Update 2 plus the 3rd party components.

  READ MORE »

Hyper-V.nu Speakers at TechEd 2014 NA and HASMUG

Fellow blogger at Hyper-V.nu Marc van Eijk did an amazing job. At his first TechEd he delivered no less than three presentations. I went to every single one of them and was really proud to see him be so successful. Great job Marc!

Lessons Learned: Designing and Deploying the Windows Azure Pack in the Real World

Effortless Migration from VMware to Windows Server 2012 R2 Hyper-V

Transforming Bare Metal into Logical Switches Using Microsoft System Center 2012 R2 Virtual Machine Manager

Cameron Fuller and Maarten Goet realized that there was still a full day available and many MVPs were still hanging around in Houston on Friday. In cooperation with the Houston Area System Management User Group (HASMUG) they found enough sponsors to organize #TE14D5 or an informal community event offering two tracks: Enterprise Client Management (ECM) and Cloud and Datacenter Management (CDM).

Both Marc van Eijk and Hans Vredevoort delivered a presentation:

Windows Azure Pack Usage Service and the System Center Components that it depends on
by Hans Vredevoort

WAP showdown – VM Template vs. VM Role
by Marc van Eijk