Update October 21st, 2011 (see bottom)

Let’s assume the following scenario:

  • You have a Hyper-V R2 SP1 cluster that has three or more nodes
  • You are using Cluster Shared Volumes
  • You learnt about the Cluster Validation problem that was solved by KB2531907 back in May 2011, which prevented the “Failed to get SCSI page 83h VPD descriptors for cluster disk <number> from node <node name> status 2” error after running the Validate SCSI Vital Product Data. It also solved the “Disk with identifier <value> has a Persistent Reservation on it. The disk might be part of some other cluster. Removing the disk from validation set” error when running the List Potential Cluster Disks test. The hotfix resolves an issue in which the storage test incorrectly runs on disks that are online and not in the Available Storage group. The problem can also be caused by other issues such as storage problems or an incorrect configuration which means you have to check your storage configuration and check related events on your Hyper-V hosts.
  • You want to add more capacity and add a new Hyper-V host.

  • You install Windows Server 2008 R2 including SP1, enable the Hyper-V role and configure network & storage
  • You are well aware that before changing the cluster you have to run Cluster Validation, so that’s what you do.
  • The Validation Report comes up the some hefty error messages. If your organization only has one cluster, it is very disconcerting that your disks might be part of some other cluster. It couldn’t be your neighbor’s cluster could it!?
  • You check on the Cluster Shared Volumes and to your utter dismay, the new cluster node shows all but one disk. That disk is now empty or at least seems empty.
  • When you call Microsoft you are reassured this is not a CSV issue. Boot sector or MBR corruption has been a problem of all time and can be attributed to bad hardware, drivers or plain misconfiguration. After all Hyper-V clusters and storage require a certain skill level.
  • Nevertheless the guests have disappeared, you have no backup and you are considering to resign before your boss gets the opportunity to fire you (I’d advise you to wait and just let your boss fire you. At least in countries like the Netherlands the social benefit system can still protect you)
  • If you are lucky and you have read this blog, you can perform a troubleshooting step that should ALWAYS be performed first!!!
  • Check which cluster node is the owner of the ‘lost disk’ and run the following command from this same node:
    Cluster.exe node /ClearPR:<disknumber> which will clear the Persistent Reservation from the LUN
  • If that doesn’t solve the issue, your only remedy is trying to get your NTFS partition boot sector repaired (which is not easy and might involve hiring a expert)

Moral of the story:

  • Don’t panic and search the Hyper-V blogs for tips and tricks (you might well find this lifesaver)
  • Install the exact same Windows Updates and (Cluster) hotfixes on the new node
  • Don’t start repairing the disk before you have tried this first troubleshooting tip
  • Make sure you have backups (System Center Data Protection Manager 2010/2012 will make your life much easier!)
  • Start using Hyper-V Replica in Windows Server 2012/Windows Server ‘8’ with Hyper-V version 3 as soon as you can get your hands on

Update October 21st, 2011

One of the Hyper-V.nu bloggers Maarten Wijsman wrote me a message saying one of hist colleagues had experienced an “unpartitioned” CSV disk and used this to solve it:

Use the tool “GPT fdisk” to recover the CSV. Please be very careful with this tool and ask for help if you find it too risky.

- GPT fdisk Tutorial >> http://www.rodsbooks.com/gdisk/

- GPT fdisk sourceforge  >> http://sourceforge.net/projects/gptfdisk/