Powered by System Center
Posts tagged Cluster Shared Volumes
[UPDATE 1/15/2013]: See problems reported after installing this hotfix at end of this forum post. I have opened a line with the product team to find out what is going on
[UPDATE 2/15/2013]: We are now exactly 1 month after I warned you to be very careful with installing http://support.microsoft.com/kb/2799728/en-us as it caused a severe memory leak. If you have followed the forum post mentioned above, you will have gathered that Microsoft has worked feverishly to get on top of the problem. Mike Jacquet who has been very communicative about this issue, has today confirmed that a fix is now code complete, has been fully tested and is only waiting for the KB article to be written.
The memory leak was caused by a fault found in the CSV filter driver (CSVFLT.sys). When the fix arrives (any time now), you can simple install it whether you have applied the hotfix mentioned in this blog or not. The original kb article will be superseded by this one.
[UPDATE 2/17/2013]: The hotfix is available from http://support.microsoft.com/kb/2813630/en-us
[UPDATE 2/18/2013]: If you are still seeing a problem, take a look at this (which is an excerpt from the forum post mentioned above:
“Cluster Shared Volume ‘Volume2′ (‘ClusterStorage Volume 2′) is no longer available on this node because of ‘STATUS_CLUSTER_CSV_AUTO_PAUSE_ERROR(c0130021)’. All I/O will temporarily be queued until a path to the volume is reestablished.
STATUS_CLUSTER_CSV_AUTO_PAUSE_ERROR is generated when csvfs filter attempts to retrieve the Copy On Write bitmap for a snapshot volume that has been cleaned up. This error is most likely occurring on large scale hyper-v deployments and is one of the issues we discover after fixing other scale out problems addressed in the V2 fix. Due to ongoing long haul testing required to be done, we did not want to hold up V2 of the fix that we just released, so the Windows group will release a more compressive V3 patch a little later to address that and other issues found during large scale testing.
For any customers still experiencing the same symptoms as outlined in KB2813630 after installing the fix, please check binary versions on all nodes.
File name File version File size Date
====== ========= ====== ====
Csvflt.sys 6.2.9200.20626 205,824 06-Feb-2013
Clussvc.exe 6.2.9200.20623 7,217,152 07-Feb-2013
Ntfs.sys 6.2.9200.20623 1,933,544 07-Feb-2013
If Binaries are correct on all nodes, please open a support case so we can investigate the issue further.”
Update October 21st, 2011 (see bottom)
Let’s assume the following scenario:
- You have a Hyper-V R2 SP1 cluster that has three or more nodes
- You are using Cluster Shared Volumes
- You learnt about the Cluster Validation problem that was solved by KB2531907 back in May 2011, which prevented the “Failed to get SCSI page 83h VPD descriptors for cluster disk <number> from node <node name> status 2” error after running the Validate SCSI Vital Product Data. It also solved the “Disk with identifier <value> has a Persistent Reservation on it. The disk might be part of some other cluster. Removing the disk from validation set” error when running the List Potential Cluster Disks test. The hotfix resolves an issue in which the storage test incorrectly runs on disks that are online and not in the Available Storage group. The problem can also be caused by other issues such as storage problems or an incorrect configuration which means you have to check your storage configuration and check related events on your Hyper-V hosts.
- You want to add more capacity and add a new Hyper-V host.
Back in May, Femi Adegoke reviewed an early beta version of Altaro Hyper-V Backup which was geared to easily backup & recover Hyper-V guests using the Hyper-V VSS Writer. Now Femi submitted his second guest blog focusing on how to protect guests in a clustered Hyper-V environment.
By Femi Adegoke
This is a follow up to our previous blog post from May 2011:
One interesting feature in this Altaro Hyper-V Backup release is support for CSV:
The current revision of the software is 184.108.40.206 BETA, download size is approximately 12.1 MB.
Our test was performed on a Dell 4 node Hyper-V cluster running 2008 R2 SP1 with a few virtual machines.
Installing the software was fast & easy and involved only a few mouse clicks. Can you believe that?
This software installs in less than 3 minutes per node.
No agent is required inside each guest
Dashboard view (this so schweeet…)
I proposed a presentation for TechNetLive 2011. This presentation will be in Dutch and held at World Forum, The Hague, Netherlands (26-20 april 2011). All you have to do is vote for this session or else somebody else will take the stage. I’d be honored!
Ever since the appearance of Hyper-V in the market, I have been stressing and promoting the blessings of the Microsoft Virtualization platform. I started writing articles and blogs about Hyper-V and joined Jaap Wesselius in the Dutch Hyper-V User Group which assembles quarterly in different parts of the country. We have been presenting Hyper-V and related technologies many times and have grown a small community in the Netherlands. My endeavors have even earned me an MVP, albeit not for Hyper-V but for Cluster. If you meet me on Twitter, you know that whatever I communicate on this social network deals with Hyper-V and System Center related topics. I use it to promote the further growth of Hyper-V.
We have seen Hyper-V version 1 still beta in Windows Server 2008, Hyper-V RTM as an update, the major update Hyper-V R2 and Hyper-V Server 2008 R2. Now we are looking forward to Hyper-V vNext, “the next version” of Hyper-V, Hyper-V in Windows 8, Hyper-V Next Generation, Hyper-V Cloud, Hyper-V 2012 or whatever the marketing people at Microsoft think will capture our minds. I personally believe not even the word Hyper-V is sacred and could be replaced if it serves a purpose.
This hotfix is only for the described symptoms:
If you see that the Cluster services stops responding in a Windows Server 2008 R2 cluster while some backup application that use VSS in parallel, you might solve this with hotfix KB2277439.
This issue occurs because of a race condition between the calls of the VSS writer on the cluster. If an OnBackupShutdown method call occurs between an OnFreeze method call and an OnThaw method call, the lock on the cluster hive is not released. Therefore, a deadlock occurs, and the Cluster service stops responding.
With respect to parallel backups of Hyper-V Virtual Machines with DPM 2010, I can refer to a very helpful blog by Matthew Hodgkins: Enabling Serialized Hyper-V Virtual Machine Backups in DPM 2010 RC
Ahmed Bisht, Senior Program Manager from the Microsoft Clustering & High Availability team published a nice PowerShell script for exposing the free disk space in Cluster Shared Volumes.
Check out his blog at:
Last week a number of great presentations were given at TechEd 2010 North America in New Orleans .
Here is a selection of sessions on Hyper-V R2 and Clustering that I consider to be my field. Some of them are presented by my Cluster MVP Lead Symon Perriman.
Disaster Recovery by Stretching Hyper-V Clusters Across Sites:
Failover Clustering Deployment Success:
Windows Server 2008 R2 and Microsoft SQL Server 2008 – Failover Clustering Implementations (by fellow cluster MVP Allan Hirt): http://www.msteched.com/2010/NorthAmerica/DAT407
Advanced Storage Infrastructure Best Practices to Enable Ultimate Hyper-V Scalability:
Hyper-V and Dynamic Memory in Depth (by Ben Armstrong):
Virtualization FAQ, Tips and Tricks (by Virtualization MVP Janssen Jones):
Networking and Windows and Windows Server 2008 R2 Hyper-V: Deployment Considerations:
Failover Clustering Pro Troubleshooting with Windows Server 2008 R2: http://www.msteched.com/2010/NorthAmerica/WSV314
HP Insight Control for Microsoft System Center (by HP Product Manager Doug de Werd): http://www.msteched.com/2010/NorthAmerica/MGT206
Windows Server 2008 R2 SP1: http://www.msteched.com/2010/NorthAmerica/WSV307
Hyper-V Backup: A look under the hood (Soumya Das Bhaumik)
I will be presenting at ExpertsLive which is held in Nijkerk, the Netherlands on June 16th 2010.
Experts Live is a knowledge event around Microsoft Infra & Security, Unified Communication, Virtualization and Management with Microsoft System Center.
My talk will be called Hyper-V R2 Hotspots zooming in on Hyper V R2 Clusters, Cluster Shared Volumes, Live Migration, Direct I/O, Dynamic Storage, Direct Memory and VHD Tooling.
Proud to say that my company Nobel is one of the main sponsors of the event, along with Citrix and PQR.
The event has free admission but requires registration via http://expertslive.nl/
The release candidate of DPM2010 has just been released.
The download location is:
The RC features support backup of machines within your intranet, which includes:
- Workgroup machines
- Machines in untrusted domains within your intranet
Please note that in this release, this feature has been built for backing up machines within environment and DPM2010 RC does not support backup of machines outside your intranet, which includes:
- Machines in DMZ network
- Machines in untrusted domains on customers’ sites connected through VPN
In a subsequent release (not sure if this is RTM), support for extranet scenario’s might be added.
Support for Hyper-V R2 clusters and cluster shared volumes (CSV) was already available in the beta.
Important to know is you can upgrade from beta to RC to RTM, or directly from DPM2007 to RC or RTM or any other combination.
In our practice we have deployed quite a number of beta versions of DPM2010 and we are quite happy with the result. I am glad we are getting closer to RTM which is expected to be on schedule for mid April 2010.
DPM2010 RC support can be found here: http://www.microsoft.com/communities/newsgroups/list/en-us/default.aspx?dg=microsoft.public.dataprotectionmanager