Posts tagged Hyper-V

Veeam Rapidly Fixes Host Level VSS Backup for Hyper-V after KB 2919355 (WS2012 R2 Update)

Update (16 April 2014): see bottom of blog

I recently blogged about Windows Server 2012 R2 Update. As usual any update and certainly an update as large as this, has some risks. Therefore we usually advise to postpone Windows Updates, Update Rollups an Hotfixes and leave a couple of weeks before deploying updates in production. Always test in a lab if you can and if you can’t, keep an eye on the forums and the blogs from MVP’s specializing in the related technology.

Testing Host Level Hyper-V Backup

This weekend I came across a tweet from Richard Skinner who reported an issue related to Veeam Backup & Replication and Hyper-V Backup after applying the Windows Server 2012 R2 (Spring) Update. Meanwhile, Veeam had already confirmed the problem and was working frantically into the weekend to fix this nasty problem.

I decided to report a support case with Veeam as well, even though I’m only running it in a Windows Azure Pack lab. I found that the problem was easily reproducible, but only if VSS was enabled in the backup job.

The Problem: Using Hyper-V Checkpoints

When this selection is made as an alternative to Veeam’s default changed block tracking (CBT), the backup fails because it cannot deal with the file path of the checkpointed VHDX files. When you zoom in on the directory of a VM that is being protected, as soon as VSS kicks in, a checkpoint is made of the active disks. This causes the writes to be redirected from the VHDX to a corresponding AVHDX file which makes it possible for the backup software to take a clean and ‘frozen’ copy of the virtual hard disk. When the backup is ready, the written data to the AVHDX file is merged back into the VHDX file. Only briefly you’ll see n AutoRecovery.avhdx file created which is deleted when it is ready with the merge operation.

Important: Microsoft started to coin the term checkpoint in VMM. After Hyper-V had used the term Snapshot for a long time, this changed with Windows Server 2012 R2. We can now better distinguish between VSS snapshots and Hyper-V checkpoints. Backup software now uses Hyper-V checkpoints just as in Hyper-V Replica.

If you want to read more about the changed method of Hyper-V backup in Windows Server 2012 R2, please take a look at fellow MVP Aidan Finn’s post:


Makeover Hyper-V Update List for Windows Server 2012 and R2

There are several update lists available on TechNet. Some are curated by the product team and some are kept up-to-date by MVP’s and other people in the community. For easy reference we decided to place a shortcut to these lists in the header of

The following hotfix and update lists are available:

  • Hyper-V: Update List for Windows Server 2012
  • Recommended hotfixes and updates for Windows Server 2012-based failover clusters (updated by Cluster product team)
  • Hyper-V: Update List for Windows Server 2012 R2
  • Recommended hotfixes and updates for Windows Server 2012 R2-based failover clusters clusters (updated by Cluster product team)

Because notably the Windows Server 2012 list had become a bit of a mess, I have rearranged the list, removed outdated or replaced hotfixes and added a sorted date column.

I have also updated the XML file for both Windows Server 2012 and R2 so that you can use a PowerShell cmdlet to quickly scan your Hyper-V hosts if a hotfix or update is installed or not. These files can be downloaded from my OneDrive.

Run the update checker like this:
.\HyperV2012R2UpdatesCheck.ps1 [name host1]



Recommended hotfixes and updates for Windows Server 2012-based failover clusters

Definitive Guide to Hyper-V 2012 R2 Network Architectures

Back in May 2011, I posted a blog called Definitive Guide to Hyper-V Networking Optimizations, which in fact was a tribute and pointer to the blog series that a respected Premier Field Engineer called Cristian Edwards Sabathe had written. The blog was one of our most read blogs in 2011 and Cristian’s blog received a lot of hits via

Yesterday I spent some time on the phone with Cristian who is now EMEA Virtualization Lead responsible for the PFEs in EMEA. Cristian reached out to me because of a long thread on the Cluster MVP distribution list which is a closed channel for MVPs dealing with server, storage, network, cluster, etc. I had been discussing the ongoing problems we see with network device drivers for Windows Server 2012 R2 which keep on nagging us. Cristian had been testing at a customer with similar hardware but a different network configuration using the Emulex driver and firmware made available in the HP ProLiant Service Pack 2014.02, two weeks ago.

Two of the blogs (blog1, blog2) that Marc van Eijk and Peter Noorderijk posted received an incredible amount of comments around either the specific HP/Emulex network problem with HP Virtual Connect Flex-10 configurations, but many other similar cases from other network adapter vendors. Most people who have responded to these blogs were able to circumvent their problems by turning off Virtual Machine Queuing (VMQ) or some other hardware offload.

Meanwhile Cristian has written a new and updated blog series, which I will again list as a must read if you want to learn about Hyper-V network architecture in Windows Server 2012 R2:

Hyper-V 2012 R2 Network Architectures Series (Part 1 of 7) – Introduction

Hyper-V 2012 R2 Network Architectures Series (Part 2 of 7) – Non-Converged Networks, the classical but robust approach

Hyper-V 2012 R2 Network Architectures Series (Part 3 of 7) – Converged Networks Managed by SCVMM and PowerShell

Hyper-V 2012 R2 Network Architectures Series (Part 4 of 7) – Converged Networks using Static Backend QoS

Hyper-V 2012 R2 Network Architectures Series (Part 5 of 7) – Converged Networks using Dynamic QoS

Hyper-V 2012 R2 Network Architectures Series (Part 6 of 7) – Converged Network using CNAs

Hyper-V 2012 R2 Network Architectures Series (Part 7 of 7) – Conclusions and Summary

And don’t forget to read the bonus blog which Cristian wrote yesterday based on our specific setup with HP c7000, HP BL460c Gen8, HP/Emulex 554FLC NICs and Windows Server 2012 R2 Hyper-V:

Hyper-V 2012 R2 Network Architectures (Part 8 of 7) – Bonus

In the meantime, I’ve been having some good discussions with one of my contacts in HP Labs in California. We now know for certain that the HP/Emulex 554FLB firmware and driver that became available with HP ProLiant Service Pack 2014.02 does not yet fix the vNIC disconnect problems we are seeing when VMQ is enabled. The latest available firmware for the Emulex 10Gb CNA is v4.9.311.20 and the driver version is 10.0.430.1109 (18 Feb 2014) which can be download as part of the Service Pack or as a separate executable cp022157.exe.

The ETA of the Emulex 554FLB driver that finally supports VMQ properly is four to six weeks from now (so around end of March/early April). By then Windows Server 2012 R2 will have been RTM for over half a year. If this is representative of the quality control of network adapter vendors, we must fear for the worst with ever faster release schedules by Microsoft for Windows Server.

Unless the new leadership at Microsoft decides this is now enough and dramatically scales up testing capacity and qualification procedures.

Delivery of a networking driver that basically works is a no go!








Mindset of the community

2013 is coming to an end. It’s an understatement to say that it was turbulent year. I left the organization that I worked at for 13 years and agreed to work for Inovativ. A decision I do not regret for a second. Windows Server 2012 R2 and System Center 2012 R2 were released. Together with my personal favorite Windows Azure Pack. We have done some really great projects with these products. I’ve started working in two times zones, GMT+1 (Netherlands) and PST (Redmond) and Lync has changed my world of communication.

But all these things pale in comparison with the people I have met. For a long time I lived with the mindset of keeping the knowledge for myself and did not see any reason of sharing it. It took me days or weeks to get some new feature figured out and I was not about to put in a lot of effort in documenting it, let alone putting it out on the web for everybody to grab. But at the same time, I searched the internet, very happy to find a solution someone posted.

In June 2012, I was at a SharePoint readiness session for partners at Microsoft. Don’t ask!! I was clearly at the right spot, but at the wrong time. The room filled with suits and ties. After a couple of sessions Robert Bakker (SSP Datacenter at Microsoft NL) walked in. Never saw the guy before. He started his session on Virtualization, so I resumed from hibernation. After a couple of slides he asked an open question. “Is there a question you have on the private cloud offering. You can ask me anything.” And for those who know me, yeah I had a couple of questions. I asked about multi-tenancy and the Dynamic Datacenter Suite (a really early version of Windows Azure Pack).

Nico van Veen (Partner Sales Manager Hosting & Cloud at Microsoft NL) that I work with closely nowadays, was my Partner Account Manager at the time. He jumped in, saying that no one in the room was understanding a word we were saying and asked us to continue that discussion at the end of the day. And so we did. Robert introduced me to Edie van den Berge (responsible for Hosting in NL at Microsoft at the time). I spoke with Edie and he advised me to contact Hans Vredevoort.

Hans Vredevoort

Hans was my first contact with the community. After some emails he agreed to meet me. We spoke for an hour or so and after 15 minutes we we already sitting behind a laptop looking at some designs. I proposed to write a blog on network virtualization for He would review it and if it was good enough it was published. To be honest, I put in a lot of effort on that blog. I was not in a writing mode and had to figure out a way to not only document my findings but also make it readable for someone else. I have a background on networking, but this network virtualization was something else. There was almost no information out there at all. That only motivated me more to get a well tested and documented blog. Hans reviewed it and approved it. With the second blog on Windows Azure Services for Windows Server I earned my login credentials for Since that moment I have had more and more contact with Hans. He shared a lot of information with me. He introduced me to his peers in the community and to members from the product teams. When I changed jobs and started at Inovativ we even became direct colleagues. We did some great project together and we complement each other in the knowledge we have and share. If I have to describe Hans in a couple of words, I’d say he is the embodiment of the community.

Peter Noorderijk

The second member of the community is Peter Noorderijk. This man is an organizer with every fiber in his body. And don’t mistake him for his knowledge on the Windows Server and System Center platform. Unfortunately most of the work that Peter does for the community goes unseen. Every event organized or was part of, Peter was a driving force behind the scenes. And not only behind the scenes. He’s a gifted speaker too. Always critical on the parts that did not go according to his high standards. We are very honored to have him as a part of and it is the combination of these talents that make it a great community.

Maarten Goet

In my opinion this man is the face of the community in the Netherlands. And for a reason. His relentless effort to give community members a platform is paying of more and more. Take the last Experts Live event. 550 seats sold out. Working until 2am the night before to get the last things organized and ready to rock the stage early the next morning. Maarten knows how to play a crowd and does this like breathing at bigger event like TechEd, MMS (sadly no more) and SCU. He knows how to get the best out of you, challenging you. Most of all he is an enabler. He provides the stage, you have to take it. He should start his own company with that talent, oh wait he did… I work there.

Didier van Hoye

I met Didier in April where he spoke on Advanced Networking features at a event. I did a session on NIC Teaming, the Hyper-V switch and QoS. There was no overlap but we could have swapped sessions easily. The mutual interest in the same subject (networking) formed an initial base for a more frequent contact. We talked about issues we were encountering, we exchanged ideas and solution and often we would just ramble for an hour or more. We are both not only very enthusiastic about the  things we do, but we also tend to talk that way. This enthusiasm is also present in his presentations and he does know how to throw some fun in there.

Aidan Finn

Aidan is the walking Hyper-V encyclopedia. And has also written it all down on his blog. There were times where I wondered if he had oil for lunch. Not only the sheer amount of blogs, but also the actual content and research that gone in to them, is mind blowing. But after I met him in Dublin I know for sure he is human. Nothing but respect for this man. I was close to a presentation once, but never actually had the privilege yet. 2014 will bring change to that. I’m sure of it. If you know some Hyper-V, you know Aidan, or at least visited his site a thousand times.

Carsten Rachfahl

Aahh… Unsere deutschen freund! I have spoken with Carsten a couple of times and laughed my head of. What a great guy. But between the jokes he asks the most difficult questions and has answers to even more. Carsten is also putting in a lot of effort in the community. Besides his blogging he is a talented interviewer. I have seen numerous interviews recorded and performed by Carsten with MVPs, program managers and other community members. Think it is easy, I dare you. Try it. And not to forget the podcasts Carsten is doing. Every month a new podcast. And those things are an hour avarage. Can you imagine the time it takes to create them.

Patrick Lownds

Patrick and I were, how should I put it, ….assigned… to the same event in Dublin. Because the hotel, where all the event attendees were staying, was fully booked I was located at another hotel. Well, at least I had time to catch up on some reading. In my 2 by 1 hotel room I noticed a tweet from Patrick saying he arrived at Dublin airport. Asked him if he was in Dublin for the same event and luckily he was. The next morning I went for breakfast. With sixty empty tables I found myself a table with some power supply next to it. Some morning reading on the NVGRE Gateway with a coffee and some toast. About 15 minutes later a guy positioned himself a couple of tables away, also for breakfast. He did have a familiar face. Laptop was still open so checked twitter again. Mmm.. small picture. Might be him. Only one way to find out. “Patrick?” “Yeah! Marc, I thought it was you”. We hanged out for two days. I had a lot of interesting conversations with Patrick and been in contact with him ever since.

Kristian Nese

Since the preview of Windows Azure Services for Windows Server I have been blogging about, speaking on and working with this great product. In the early days of the product there was almost no information available. One of the community members that was also publishing blogs about the Windows Azure Services for Windows Server was Kristian. Recently Kristian and me have more contact and it turns out that the type of projects we do are very similar. Together with Flemming Riis they have released a whitepaper on Network Virtualization, that is now turning in a series of whitepapers on the CloudOS we are doing in a joint effort.

Flemming Riis

I have had a couple of calls with Flemming about the whitepaper series. He actually build the environment for the NVGRE whitepaper. Flemming is enthusiastic but also very realistic. We first started talking about a book about the CloudOS, but with his insight into the effort it takes and the kind of work we are doing that would quickly become a very difficult adventure. Together with Kristian Nese we settled on a series of whitepapers.

Damian Flynn

I know Damian from all the work he is doing on getting the NVGRE story out there. Speaking at event, the blog series with Nigel Cain, webinars, you name it. Over time Damian is also putting more emphasis on Windows Azure Pack. I’m still aiming at a joined Windows Azure Pack session with him on TechEd this year.

I can also mention, Stanislav Zhelyazkov, Daniel Neumann, James van den Berg, Kevin Greene, Thomas Maurer, Gordon McKenna, Ronny de Jong, Darryl van der Peyl, and then some. All these guys have day jobs and usually work more than 40 hours a week. Besides their normal working hours they put in a lot of effort to learn and test new functionalities. They take screenshots, create diagrams and document it in an easy readable format and post in on the internet for you to read at no cost. They create PowerPoint presentations, give presentations, webinars, interviews, podcast and organize events. And they do all of this with a single purpose. Sharing knowledge. Behind every hyperlink in this blog post there is tons of valuable information and its free.

I learned that sharing knowledge is not a bottomless pit, but it forms the basis of gaining knowledge. I’m honored to be part of this community and I want to thank everyone that I was privileged to interact with this year.

Merry Christmas and a happy new sharing year.

Marc van Eijk

Update List for Windows Server 2012 R2 Hyper-V

Taylor Browne has very recently created a new Wiki for the latest Hyper-V service packs, updates, and hotfixes for Windows Server 2012 R2 Hyper-V:

Currently only one KB article KB2883200 is on the list but it will not take long before this list will grow. KB2773200 was the General Availability Rollup which was delivered at Windows Server 2012 R2 GA release in October 2013.  This KB article also recommends to install KB2884846 which is the Windows Server 2012 R2 update rollup for October 2013 and includes “performance and reliability improvements”.

Many of the monthly update rollups for Windows Server 2012 R2 which come via Windows Update hide the exact details of what exactly has been changed or solved. It is very difficult to determine if it is specific to Hyper-V, clustering, networking, storage or all of them.

The November 2013 update rollup KB2887595 contains a fix for Hyper-V guest OS does not shut down when you restart the host computer that is running Windows Server 2012 R2 or Windows 8.1 which I encountered quite a lot in our environment. What seriously worries me however is that this same November 2013 update rollup also fixes things that are not mentioned in the list.  For instance during a support case with Microsoft which dealt with VMs losing network connections on Windows Server 2012 R2 Hyper-V hosts, we learnt that a particular update part of the November rollup was also supposed to fix a problem with LBFO teaming. It turned out that it didn’t fix the problem for everyone as we still have to disable VMQ to keep our VM networking operational.

The December 2013 update rollup KB2903939 is more explicit on what updates have been made but I have a hard time figuring out what critical updates are made for enterprise servers. I have to filter through things like issues with client computers, Lumia 2520 camera app crashes, errors in Windows Live Messenger, mouse pointer stutters, input devices that fail and something that never enters a DRIPS, whatever that may be!


Here is another update that is not yet on the list and applies to Windows Server 2012 R2 and Hyper-V: Known issues after you enable data deduplication on CSV.

My advice to Microsoft:

  • be much more specific about what has been fixed
  • avoid stealth fixes (as we call them among Hyper-V MVP’s) which solve problems but are not described at all
  • separate the server updates from client and mobile.
  • be open and honest if updates turn out to be bad or broken (as we have seen several times in the past)
  • formalize the wiki by offering a hotfix/update list authorized by the product teams like the cluster team does with: Recommended hotfixes and updates for Windows Server 2012-based Failover clusters.

This last support page is a very good initiative but it can take a while before updates are posted. A vital hotfix KB2908415 was announced only yesterday: Clustered shared volumes go offline or the Cluster service stops during VM backup on a Windows Server 2012 Hyper-V host server but is not yet on either the Cluster Update Wiki or the Official list for Windows Server 2012.

In other words: work to be done!

Windows Azure Pack Is Clearly The Place To Be

You may or may not have noticed it, but the daily stream of updates around Windows Azure Pack is becoming more and more amazing. The WAP Wiki initiative I took with Marc van Eijk several weeks ago, makes us very aware of all the blogs and articles that are published around Windows Azure Pack, Service Provider Framework, VM Role, Service Management Automation, Usage and Billing and to a lesser extent Database as a Service, Service Bus and Web Sites.

It is more than obvious that a huge amount of Microsoft resources have been dedicated to making the Windows Azure Pack a success. The Building Clouds section on the Microsoft Server & Tools blog which was kicked off by Brad Anderson himself is very active indeed.

Sometimes I see several posts a day coming from the different team members. Guys like Anders Ravnholt, Tiander Turpijn, Jim Britt, Charles Joy, Mark Stanfill, Shiram Natarajan, Thomas Roettinger, Michael Greene, Eamon O Reilly, Justin Incarnato and many more write really interesting blogs that light up the for many still undiscovered possibilities with the Windows Azure Pack. In the community we see good blogs from Marc van Eijk, Walter Eikenboom, Daniel Neumann, Nathan Lasnoski, Michael Rueefli, Joe Thompson, Damian Flynn, Stanislav Zhelyazkov, Lai Yoong Seng and others.


Of course this is only the beginning and we have tons of feature requests for additional functionality. But then, there is so much to learn in this corner of the cloud that many of us can hardly keep up. As one of the initiators of the blog, I’m fully aware that we can build on a superb platform using Windows Server 2012 R2, Hyper-V, System Center 2012 and now with the Windows Azure Pack, Microsoft has just begun to realize its full potential.

If you are looking for an organized index to Windows Azure Pack related subjects, just navigate to the WAP Wiki! There is also @WAzureP tweeting on Windows Azure Pack and related stuff. You can follow @hvredevoort and @_marcvaneijk for regular updates on these topics.

Please help us promote this with a tweet, blog or face… (no nothing on FB please)
We use the #WAPack hashtag and suggest you do too!


Hotfix: VM Virtual Fibre Channel Loses Access to LUN after Live Migration

A hotfix has been released today for Windows Server 2012 Hyper-V servers which are unable to access LUNs over a Synthetic Fibre Channel after a VM is live migrated to another host in the cluster.

This problem can occur if the following conditions are met:

  • You have two Windows Server 2012-based computers that have the Hyper-V role installed
  • You install a virtual machine on one of the Windows Server 2012 Hyper-V hosts
  • You set up a guest failover cluster, and then you make the virtual machine a cluster node
  • The virtual machine is configured to access LUNs over a Synthetic Fibre Channel
  • You try to perform live migration to move the virtual machine to another host.

This issue is caused by the inability of the Hyper-V host to restore the Synthetic Fibre Channel LUN on behalf of the virtual machine during live migration.

The problem applies to Windows Server 2012 Standard and Datacenter.

For further details look up the Microsoft Support Article:



Windows Azure Pack Wiki

If you are a regular reader of this blog you have noticed that a lot of content that is written is related to Windows Azure Pack (WAP for short), formerly known as Windows Azure Services for Windows Server. There are also numerous good other blogs as well as videos about WAP written by fellow MVPs and the different Microsoft Product Teams.


I was already collecting my own list of WAP content but decided to put it in a Wiki on Microsoft TechNet. The WAP Wiki contains the following headings:

  • Windows Azure Services for Windows Server (older content but still useful)
  • Windows Azure Pack
  • Service Provider Foundation
  • Usage and Billing
  • VM Role
  • Service Management Automation
  • Hyper-V Network Virtualization Gateway
  • Hyper-V Network Virtualization
  • Disaster Recovery

Of course if you login to the Wiki with your Microsoft Account, you can add content to it. Please mark the new content with (New!) so it is easy to see what has been added.

You can find the WAP Wiki here:


Some More Background on Windows Update KB2855336

Not very often do I remember a Windows Update KB article off the top of my head, but this time I have talked and written about KB2855336 so often that it was probably written into my short term read-cache..

If you have been careful and missed the first version of this update because you rather wait and see what others run into after Patch Tuesday, I must compliment your smartness!

This time I have an update on the update for you. First of all the rewritten version of the KB article and secondly a link to a blog of Microsoft’s Escalation Engineer, Rob Scheepens

The part of the July update rollup KB2855536 that corrects the AD corruption risk was KB2853952. The updated KB article, which is now titled “Loss of consistency with IDE-attached virtual hard disks when a Windows Server 2012-based Hyper-V host  server  experiences an unplanned restart”, can be found here: 

The blog by Rob Scheepens details a debug for the 0xD1 bug check after installing the first release of KB2855336: This blog refers to the source of the bug check while live migrating virtual machines:


In the past week we have not seen any problems after updating all clusters with the latest Windows Updates and hotfixes. So make sure you reserve some extra testing time when next Patch Tuesday arrives. Tell your boss you definitely need a test cluster to try these updates on first. I know of several customers who found about the 0xD1 STOP by first deploying the updates to their test clusters.


Keeping your Virtual Active Directory Domain Controllers Safe

In my last blog I sent out a red alert on a killer Windows Update that had not been sufficiently tested. The net result was a full crash of a two-node System Center fabric management cluster. The fabric was still in the making and backups were only provisionally taken in the form of Virtual Machine exports of the most important virtual machines.  As fellow Hyper-V MVP Aidan Finn wrote unambiguously: “Something Has Gone Very Wrong With Microsoft Patch Testing

Where did it go wrong?

I was actually demonstrating the fantastic Cluster Aware Updating functionality in Windows Server 2012 clusters, which would automatically move all VMs off a host, update it, reboot it, live migrate the VMs back to the updated cluster node and move on to the next.

The problematic July Update Rollup KB2855336 – which was one of the updates to be processed – is actually a collection of originally 20 issues that solves problems in several areas. A still unidentified part of that rollup caused a 0x000000D1 Stop error while live migrating a VM on a Windows Server 2012-based server. So Cluster Aware Updating using the Live Migration mechanism to place a host in maintenance mode, combined with the mentioned update, sent shockwaves through the cluster. In this case both cluster nodes crashed within minutes.

Catch 22

Ironically enough this same July Update Rollup also contained an important fix for a problem that has been around for some time: Active Directory database becomes corrupted when a Windows Server 2012-based Hyper-V host server crashes (KB2853952).


Assume that you have a Windows Server 2012-based virtualized domain controller on a Windows Server 2012-based Hyper-V host server. When the Hyper-V host server crashes or encounters a power outage, the Active Directory database may become corrupted.


This issue occurs because the guest system requests the Hyper-V server to turn off disk caching on a disk. However, the Hyper-V server misinterprets the request and keeps disk caching enabled.

If you try to disable the write caching manually you will see this error: “Windows could not change the write-caching setting for the device. Your device might not support this feature or changing the setting.”  On a physical domain controller this has never been a problem.



It takes very little imagination to guess what happened to Active Directory if you combine the full STOP of the fabric management cluster and the AD domain controllers that were virtualized on that same Hyper-V cluster without the required updates and hotfixes.


Our Sponsors

Powered by