W2K12 Hotfix related to Host Level Backup of VMs

[UPDATE 1/15/2013]: See problems reported after installing this hotfix at end of this forum post. I have opened a line with the product team to find out what is going on

[UPDATE 2/15/2013]: We are now exactly 1 month after I warned you to be very careful with installing http://support.microsoft.com/kb/2799728/en-us as it caused a severe memory leak. If you have followed the forum post mentioned above, you will have gathered that Microsoft has worked feverishly to get on top of the problem. Mike Jacquet who has been very communicative about this issue, has today confirmed that a fix is now code complete, has been fully tested and is only waiting for the KB article to be written.
The memory leak was caused by a fault found in the CSV filter driver (CSVFLT.sys). When the fix arrives (any time now), you can simple install it whether you have applied the hotfix mentioned in this blog or not. The original kb article will be superseded by this one.

[UPDATE 2/17/2013]: The hotfix is available from http://support.microsoft.com/kb/2813630/en-us

[UPDATE 2/18/2013]: If you are still seeing a problem, take a look at this (which is an excerpt from the forum post mentioned above:

“Cluster Shared Volume ‘Volume2′ (‘ClusterStorage Volume 2′) is no longer available on this node because of ‘STATUS_CLUSTER_CSV_AUTO_PAUSE_ERROR(c0130021)’. All I/O will temporarily be queued until a path to the volume is reestablished.

STATUS_CLUSTER_CSV_AUTO_PAUSE_ERROR is generated when csvfs filter attempts to retrieve the Copy On Write bitmap for a snapshot volume that has been cleaned up.  This error is most likely occurring on large scale hyper-v deployments and is one of the issues we discover after fixing other scale out problems addressed in the V2 fix. Due to ongoing long haul testing required to be done, we did not want to hold up V2 of the fix that we just released, so the Windows group will release a more compressive V3 patch a little later to address that and other issues found during large scale testing.

For any customers still experiencing the same symptoms as outlined in KB2813630 after installing the fix, please check binary versions on all nodes.

File name       File version       File size        Date        
======      =========   ======    ====
Csvflt.sys     6.2.9200.20626   205,824      06-Feb-2013
Clussvc.exe  6.2.9200.20623   7,217,152   07-Feb-2013
Ntfs.sys       6.2.9200.20623   1,933,544   07-Feb-2013

If Binaries are correct on all nodes, please open a support case so we can investigate the issue further.”

If you are running host level backups of Windows Server 2012 Hyper-V VMs which are located on Cluster Shared Volumes using Data Protection Manager 2012 SP1, then the following hotfix is applicable to you.

Virtual machine enters a paused state or a CSV volume goes offline when you try to create a backup of the virtual machine on a Windows Server 2012-based failover cluster

Symptoms

Consider the following scenario:

• You enable the Cluster Shared Volumes (CSV) feature on a Windows Server 2012-based failover cluster.
• You create a virtual machine on a CSV volume on a cluster node.
• You start the virtual machine.
• You try to create a backup of the virtual machine on the CSV volume by using Microsoft System Center Data Protection Manager (DPM).

In this scenario, one of the following issues occurs:

• The backup is created, and the virtual machine enters a paused state.
• The CSV volume goes offline. Therefore, the virtual machine goes offline, and the backup is not created.

Additionally, the following events are logged in the Cluster log and System log respectively:

Software snapshot creation on Cluster Shared Volume(s) (‘volume location‘) with snapshot set id ‘snapshot id‘ failed with error ‘HrError(0×80042308)(2147754760)’. Please check the state of the CSV resources and the system events of the resource owner nodes.

Note This issue also occurs when you try to create a snapshot of the virtual machine by using a backup software that uses the Microsoft Software Shadow Copy Provider.

Cause

The virtual machine enters a paused state because the Ntfs.sys driver incorrectly reports the available space on the CSV volume when the backup software tries to create a snapshot of the CSV volume. Additionally, the CSV volume goes offline because the CSV volume does not resume from a paused state after an I/O delay issue or an I/O error occurs.
Note The CSV volume is resilient.

See support article for further details.