Powered by System Center
W2K12 Hotfix related to Host Level Backup of VMs
[UPDATE 1/15/2013]: See problems reported after installing this hotfix at end of this forum post. I have opened a line with the product team to find out what is going on
[UPDATE 2/15/2013]: We are now exactly 1 month after I warned you to be very careful with installing http://support.microsoft.com/kb/2799728/en-us as it caused a severe memory leak. If you have followed the forum post mentioned above, you will have gathered that Microsoft has worked feverishly to get on top of the problem. Mike Jacquet who has been very communicative about this issue, has today confirmed that a fix is now code complete, has been fully tested and is only waiting for the KB article to be written.
The memory leak was caused by a fault found in the CSV filter driver (CSVFLT.sys). When the fix arrives (any time now), you can simple install it whether you have applied the hotfix mentioned in this blog or not. The original kb article will be superseded by this one.
[UPDATE 2/17/2013]: The hotfix is available from http://support.microsoft.com/kb/2813630/en-us
[UPDATE 2/18/2013]: If you are still seeing a problem, take a look at this (which is an excerpt from the forum post mentioned above:
“Cluster Shared Volume ‘Volume2′ (‘ClusterStorage Volume 2′) is no longer available on this node because of ‘STATUS_CLUSTER_CSV_AUTO_PAUSE_ERROR(c0130021)’. All I/O will temporarily be queued until a path to the volume is reestablished.
STATUS_CLUSTER_CSV_AUTO_PAUSE_ERROR is generated when csvfs filter attempts to retrieve the Copy On Write bitmap for a snapshot volume that has been cleaned up. This error is most likely occurring on large scale hyper-v deployments and is one of the issues we discover after fixing other scale out problems addressed in the V2 fix. Due to ongoing long haul testing required to be done, we did not want to hold up V2 of the fix that we just released, so the Windows group will release a more compressive V3 patch a little later to address that and other issues found during large scale testing.
For any customers still experiencing the same symptoms as outlined in KB2813630 after installing the fix, please check binary versions on all nodes.
File name File version File size Date
====== ========= ====== ====
Csvflt.sys 6.2.9200.20626 205,824 06-Feb-2013
Clussvc.exe 6.2.9200.20623 7,217,152 07-Feb-2013
Ntfs.sys 6.2.9200.20623 1,933,544 07-Feb-2013
If Binaries are correct on all nodes, please open a support case so we can investigate the issue further.”
If you are running host level backups of Windows Server 2012 Hyper-V VMs which are located on Cluster Shared Volumes using Data Protection Manager 2012 SP1, then the following hotfix is applicable to you.
Virtual machine enters a paused state or a CSV volume goes offline when you try to create a backup of the virtual machine on a Windows Server 2012-based failover cluster
Consider the following scenario:
- You enable the Cluster Shared Volumes (CSV) feature on a Windows Server 2012-based failover cluster.
- You create a virtual machine on a CSV volume on a cluster node.
- You start the virtual machine.
- You try to create a backup of the virtual machine on the CSV volume by using Microsoft System Center Data Protection Manager (DPM).
In this scenario, one of the following issues occurs:
- The backup is created, and the virtual machine enters a paused state.
- The CSV volume goes offline. Therefore, the virtual machine goes offline, and the backup is not created.
Additionally, the following events are logged in the Cluster log and System log respectively:
Software snapshot creation on Cluster Shared Volume(s) (‘volume location‘) with snapshot set id ‘snapshot id‘ failed with error ‘HrError(0×80042308)(2147754760)’. Please check the state of the CSV resources and the system events of the resource owner nodes.
Note This issue also occurs when you try to create a snapshot of the virtual machine by using a backup software that uses the Microsoft Software Shadow Copy Provider.
The virtual machine enters a paused state because the Ntfs.sys driver incorrectly reports the available space on the CSV volume when the backup software tries to create a snapshot of the CSV volume. Additionally, the CSV volume goes offline because the CSV volume does not resume from a paused state after an I/O delay issue or an I/O error occurs.
Note The CSV volume is resilient.
See support article for further details.
|Print article||This entry was posted by Hans Vredevoort on February 15, 2013 at 21:31, and is filed under Hans Vredevoort, Hyper-v. Follow any responses to this post through RSS 2.0. You can leave a response or trackback from your own site.|
- Microsoft Most Valuable Professional (MVP) – Best Posts of the Week around Windows Server, Exchange, SystemCenter and more – #12 – TechCenter – Blog – TechCenter – Dell Community
- Microsoft Most Valuable Professional (MVP) – Best Posts of the Week around Windows Server, Exchange, SystemCenter and more – #12 – Dell TechCenter – TechCenter – Dell Community
- Microsoft releases bug fix to fix issues on CSV backup of Windows Server 2012 Hyper-V « UP2V
- February Round-Up: Best of the Web and Observations
about 3 weeks ago - 5 comments
Have you ever wondered why you DO see performance data in Windows 8/8.1 under the performance tab in Task Manager, but DON’T see this same information in Windows Server 2012/2012 R2? Well I kind of missed seeing that information in Server but never really bothered to really investigate. Windows 8/8.1 Windows Server 2012/2012 R2 I…
about 3 weeks ago - 3 comments
A hotfix has been released today for Windows Server 2012 Hyper-V servers which are unable to access LUNs over a Synthetic Fibre Channel after a VM is live migrated to another host in the cluster. This problem can occur if the following conditions are met: You have two Windows Server 2012-based computers that have the…
about 4 months ago - 4 comments
Several months ago I had a very short encounter with a Dell Compellent storage array which had just been installed for a customer in their New Jersey office. In a short blog I showed that creation of a 250GB VHDX could be done in just a few seconds. This was the result of Compellent’s support…
about 4 months ago - 4 comments
Not very often do I remember a Windows Update KB article off the top of my head, but this time I have talked and written about KB2855336 so often that it was probably written into my short term read-cache.. If you have been careful and missed the first version of this update because you rather…
about 4 months ago - 11 comments
In my last blog I sent out a red alert on a killer Windows Update that had not been sufficiently tested. The net result was a full crash of a two-node System Center fabric management cluster. The fabric was still in the making and backups were only provisionally taken in the form of Virtual Machine…
Disable: NetFTFlt driver (Microsoft Failover Cluster Virtual Adapter Performance Filter (NetFT-LWF) ) when using Windows Server 2012
about 4 months ago - 1 comment
A while ago I wrote a blog about problems with virtual guest clusters and NIC teaming. See this link. I ended this blog with a workaround: disable checksum offloading. Today I received a message from Microsoft Premier Support that they found the root cause for this problem: The NetFTflt (Microsoft Failover Cluster Virtual Adapter Performance…
about 5 months ago - 2 comments
[Update July 13, 2013 - I was able to deploy the newly issued KB2855336 to all of my physical and guest cluster nodes. There have been no issues so far. The same KB will also show up in most of your VMs as it is a collection of 21 updates touching all kinds of bugs…
about 5 months ago - 7 comments
Update June 26, 2013: Cristian Edwards notified me that he has updated the script to now support using a cluster name. That will save you some typing if you cluster counts 64 nodes. See end of blog Update July 1, 2013: Great to see that Niklas Akerlund and Trond Hindenes made great extensions on the…
Virtual Guest cluster and NIC teaming in the host results in an evicted cluster Node #Broadcom #Emulex
about 5 months ago - 3 comments
Recently I was involved in an implementation of a private cloud based on Hyper-V Server 2012 and System Center 2012 SP1. We’ve build a two node Hyper-V cluster (HP DL 980 servers) dedicated for Fabric Management. Both nodes in the cluster have a total of four 10Gbit interfaces (Emulex). Two of them are combined in…
about 6 months ago - No comments
We already know that we wouldn’t have to wait four years to get significant new features in Windows Server & Hyper-V. Looking at the list of builds since the first version of Hyper-V, we can observe there were considerable intervals between the releases of Windows Server 2008 (R2) and Windows Server 2012. We’ve seen three…