Powered by System Center
// Main banner ad on homepage
Posts tagged CSV
W2K12 Hotfix related to Host Level Backup of VMs
Feb 15th
[UPDATE 1/15/2013]: See problems reported after installing this hotfix at end of this forum post. I have opened a line with the product team to find out what is going on
[UPDATE 2/15/2013]: We are now exactly 1 month after I warned you to be very careful with installing http://support.microsoft.com/kb/2799728/en-us as it caused a severe memory leak. If you have followed the forum post mentioned above, you will have gathered that Microsoft has worked feverishly to get on top of the problem. Mike Jacquet who has been very communicative about this issue, has today confirmed that a fix is now code complete, has been fully tested and is only waiting for the KB article to be written.
The memory leak was caused by a fault found in the CSV filter driver (CSVFLT.sys). When the fix arrives (any time now), you can simple install it whether you have applied the hotfix mentioned in this blog or not. The original kb article will be superseded by this one.
[UPDATE 2/17/2013]: The hotfix is available from http://support.microsoft.com/kb/2813630/en-us
[UPDATE 2/18/2013]: If you are still seeing a problem, take a look at this (which is an excerpt from the forum post mentioned above:
“Cluster Shared Volume ‘Volume2′ (‘ClusterStorage Volume 2′) is no longer available on this node because of ‘STATUS_CLUSTER_CSV_AUTO_PAUSE_ERROR(c0130021)’. All I/O will temporarily be queued until a path to the volume is reestablished.
STATUS_CLUSTER_CSV_AUTO_PAUSE_ERROR is generated when csvfs filter attempts to retrieve the Copy On Write bitmap for a snapshot volume that has been cleaned up. This error is most likely occurring on large scale hyper-v deployments and is one of the issues we discover after fixing other scale out problems addressed in the V2 fix. Due to ongoing long haul testing required to be done, we did not want to hold up V2 of the fix that we just released, so the Windows group will release a more compressive V3 patch a little later to address that and other issues found during large scale testing.
For any customers still experiencing the same symptoms as outlined in KB2813630 after installing the fix, please check binary versions on all nodes.
File name File version File size Date
====== ========= ====== ====
Csvflt.sys 6.2.9200.20626 205,824 06-Feb-2013
Clussvc.exe 6.2.9200.20623 7,217,152 07-Feb-2013
Ntfs.sys 6.2.9200.20623 1,933,544 07-Feb-2013
If Binaries are correct on all nodes, please open a support case so we can investigate the issue further.”
Altaro Hyper-V Backup getting ready for Windows Server 2012
Aug 7th
Review of Altaro Hyper-V Backup 3.5
notice: Altaro are giving away two Nexus 7’s to testers of their Hyper-V backup for Windows Server 2012 beta. Check out all the details at the bottom of this blog
Background
The crew at Hyper-V.nu were offered a first glance of Altaro Hyper-V Backup back in May 2011. We agreed to do a review of the beta version of the product. That particular version of the beta did not have support for Cluster Shared Volumes (CSV) and we provided feedback to Altaro about this shortcoming. As our Great Big Hyper-V Survey of 2011 showed, over 94% of users with Hyper-V clusters use CSV. We were pleasantly surprised how quickly Altaro responded by adding support for CSV in one of their next beta’s. So we did a second review testing the backup and recovery of guests living on cluster shared volumes on a Hyper-V R2 SP1 cluster.
Since then Altaro released their RTM of version 2, version 3 and v3.1 which have continued to add functionality, ease and performance. You can find a change log of Altaro Hyper-V Backup here: http://wiki.altaro.com/releases/change-log-for-altaro-hyper-v-backup
One noticeable improvement in version 3 was installation of the product on a cluster. When on a cluster the application could detect each node and configure them during the installation. You only needed to install Altaro Hyper-V Backup on one node and it would then automatically deploy modules on all other cluster nodes. Additionally, the administrator was able to manage all guests across all nodes from a single console. This meant that all backups, restores and configuration could be done from one centralized console. In a cluster environment all guests could be backed up to a single backup target such as a USB drive, a NAS or a disk on a SAN. Even when guests moved around in the cluster, Altaro Hyper-V Backup would take care of that.
By using a technique called ReverseDelta, incremental backups could be made at tremendous speed. The last version introduced ReverseDelta v2 which made incremental backups at least 300% faster.
Version 3 of Altaro Hyper-V backup also dealt with one of CSV’s shortcomings: redirected access during backup operations (backup node claiming ownership of disk and redirecting I/O across the CSV network for all other nodes for that CSV disk). A new scheduling feature called ‘Scheduling Groups’ was introduced which allowed a quick and easy drag & drop of Hyper-V guests to one or more defined Scheduling Groups. It would group guests on the same CSV together in order to decrease time required in redirected access mode. If you are a Data Protection Manager user, you will appreciate this feature as DPM does not do this for you automatically. It is also quite time consuming to even select multiple Hyper-V guests for backup and add them to their Protection Group.
Click on More for the rest of the post
Windows 8 Storage and Hyper-V – Part 1: Introduction
Jan 21st
If you are a server, storage or network vendor, please also read the last section.
Now that the Windows Server 8 beta can be expected any time now – but promised before the last week of February 2012 – it might be a good idea to start looking at several of the groundbreaking storage related technologies that could turn up in Windows 8. I stress the word could because we must always be careful since functionality shown in Pre-RTM builds is never guaranteed to be in the GA release.
I am planning to write a couple of blogs about Windows 8 Storage which in many cases is related to the new version of Hyper-V.
If you are running a SAN but also if your company simply can’t afford a SAN, chances are that you will see significant performance increases when reading, writing, copying, moving data with Windows Server 8. In-box storage manageability with PowerShell will strongly contribute to making Windows 8 Storage one of the major pillars of the Microsoft Private Cloud Fabric. As I have looked at it so far, storage is handled extremely well in Windows Server 8. Mind you this is only what I have determined based on what I have heard and seen on \\build and have personally tested since September 2011 with the Windows 8 Developer Preview.
This edition is not at all intended to be stable and testing with de Developer Preview is a true challenge. As a preparation for the Hyper-V.nu event with several sessions on Windows Server 8, I wanted to try out and show some unique new functionality for moving living Hyper-V guests between different types of storage. I had time for building up the pre-requisites and was able to successfully show a Live Storage Migration of a guest between two SMB2 shares on a ScaleOut File Cluster. But there were several other Live Storage moves that I tested but also want to explore further:
-
Live Storage Migrate a guest from USB disk to a local disk
-
Live Storage Migrate a guest from a local disk to a new Windows 8 Pool and Spaces virtual disk
-
Live Storage Migrate a guest from local disk to shared storage on a single host
-
Live Storage Migrate a guest from a shared disk on a single host to a shared disk on a Windows 8 Hyper-V cluster
-
Live Storage Migrate a guest from a shared disk on a cluster to a CSV version 2 volume on the same cluster
-
Live Storage Migrate a guest from any location to an SMB2 file share on a ScaleOut Fileserver with Continuously Available Shares
-
Live Storage Migrate a guest between two SMB2 file shares on the same Windows 8 Hyper-V cluster while the ScaleOut Fileserver cluster is moved between nodes.
-
Live Storage Migrate a guest between an SMB2 file share used by one Windows 8 Hyper-V cluster to another Hyper-V cluster or even to another SMB2 file share on another server.
Am I now running out of options? On the contrary, I have only just started!
Hyper-V interview video’s in cooperation with NGN
Nov 23rd
A couple of weeks ago the Hyper-V.nu team visited the Netwerk Gebruikersgroep Nederland (NGN) to record some interviews about Hyper-V. The NGN record and edit the video’s. Jaap Wesselius interviewed Hans Vredevoort, Maarten Wijsman and me. The first of the three video’s is just published online at the site of NGN.
You can watch the video here
First troubleshooting step for a lost (CSV) disk in Hyper-V cluster
Aug 18th
Update October 21st, 2011 (see bottom)
Let’s assume the following scenario:
- You have a Hyper-V R2 SP1 cluster that has three or more nodes
- You are using Cluster Shared Volumes
- You learnt about the Cluster Validation problem that was solved by KB2531907 back in May 2011, which prevented the “Failed to get SCSI page 83h VPD descriptors for cluster disk <number> from node <node name> status 2” error after running the Validate SCSI Vital Product Data. It also solved the “Disk with identifier <value> has a Persistent Reservation on it. The disk might be part of some other cluster. Removing the disk from validation set” error when running the List Potential Cluster Disks test. The hotfix resolves an issue in which the storage test incorrectly runs on disks that are online and not in the Available Storage group. The problem can also be caused by other issues such as storage problems or an incorrect configuration which means you have to check your storage configuration and check related events on your Hyper-V hosts.
- You want to add more capacity and add a new Hyper-V host.
The teams that contribute to Hyper-V version 3
Aug 18th
When Sinofsky detailed the different teams that are building Windows 8, I noticed that Storage and File Systems are one and the same group. Beneath you see a list of the most important teams in my opinion that will leverage the Microsoft Private Cloud which is built around Hyper-V and System Center 2012. The release of Hyper-V version 3 in Windows Server 2012 or Server “8” as we still have to call it, will be the cream on the cake and will boost the Microsoft Private Cloud even further. What we have seen so far is much higher virtual CPU’s in guests and awesome Hyper-V Replication technology. Add to that a decent file system and new storage innovations, both in the host and in the guest, we are ready for primetime. What will happen to CSV? What will happen to the scalability of the Hyper-V cluster?
We will soon find out! The Build Windows 2011 developers conference will unveil what all these product teams have invented. I really can’t wait! Because the conference is sold out, I plan to reserve time for the live streams instead.
See: http://blogs.msdn.com/b/b8/ and http://www.buildwindows.com/
Reviewing Altaro Hyper-V Backup including CSV support
Aug 2nd
Back in May, Femi Adegoke reviewed an early beta version of Altaro Hyper-V Backup which was geared to easily backup & recover Hyper-V guests using the Hyper-V VSS Writer. Now Femi submitted his second guest blog focusing on how to protect guests in a clustered Hyper-V environment.
By Femi Adegoke
This is a follow up to our previous blog post from May 2011:
http://www.hyper-v.nu/archives/hvredevoort/2011/05/altaro-hyper-v-backup-review/
One interesting feature in this Altaro Hyper-V Backup release is support for CSV:
http://www.altaro.com/hyper-v-backup/
The current revision of the software is 2.0.8.0 BETA, download size is approximately 12.1 MB.
Our test was performed on a Dell 4 node Hyper-V cluster running 2008 R2 SP1 with a few virtual machines.
Installation
Installing the software was fast & easy and involved only a few mouse clicks. Can you believe that?
-
This software installs in less than 3 minutes per node.
-
No agent is required inside each guest
Dashboard
Dashboard view (this so schweeet…)
How to defrag a Hyper-V R2 Cluster Shared Volume
May 31st
Recently I was asked to describe the correct procedure for defragmenting Cluster Shared Volumes on a Hyper-V R2 cluster. This is not really a very complicated task but if you have never had the opportunity to give it a try, this blog post will offer you the exact steps using PowerShell.
Case
Let’s start with a case description: the System Center Operations Manager Windows Management Pack is reporting “Logical Disk Fragmentation Level is high” for your Hyper-V R2 servers.
A Cluster Shared Volume (CSV) contains the configuration, virtual hard disk and snapshot files of multiple Hyper-V guests. Notably fragmentation of the large VHD files deserve your attention.
Fragmentation of these files can become a problem because the disk head needs to use an increasing number of seeks, lowering the throughput and thus the perceived performance of the guest as a whole.
On the other hand, NTFS has become more and more efficient in recent OS versions and fragmentation need not always have a severe impact on performance.
Analysis
CSV is a distributed orchestration layer on top of NTFS (implemented as a file system filter driver) and for fragmentation it takes advantage of all the NTFS techniques. The advantage of this design is that all disk management tools which have been written for NTFS continue to work, including a variety of defrag tools.
February 2011 Hotfix Rollup for Virtual Machine Manager 2008 R2
Feb 10th
With KB249280 Microsoft issued a new hotfix rollup for Virtual Machine Manager 2008 R2 that deals with five issues around Performance and Resource Optimization (PRO), mapping of virtual switches to teamed network adapters, V2V to CSV, P2V with Windows Server 2003 and migration of highly available guests between clusters: http://support.microsoft.com/kb/2492980
Very likely this is the last rollup before we receive SP1 for SCVMM 2008 R2 which integrates the new Dynamic Memory and RemoteFX functionality found in Hyper-V R2 SP1.
If you’ve configured Windows Update for Windows and other products, you’ll see that the update is ready for installation.
Effects of offline System Partition on Hyper-V cluster node (updated)
Oct 4th
One of my colleagues noticed rather strange behavior with one of the Windows Server 2008 R2 servers which is part of a 7-node Hyper-V cluster built with HP BL460c G6 blade servers. This behavior showed up during host level backups of Hyper-V virtual machines.
What problems were observed?
-
The System Reserved Partition did not have a label and was offline after a reboot
-
A CSV could be offline if it was owned by this particular cluster node
These problems could be temporarily solved by assigning a drive letter. Oddly enough Disk Manager and Diskpart did not agree with each other about the disk being online or offline. In Diskpart the problematic partition would show offline and in Disk Manager it was online. It would show offline if the disk had no label:
Without a disk label: 
With a disk label: ![clip_image002[5]](http://hyper-v.nu/blogs/hans/wp-content/uploads/2010/10/clip_image0025.jpg "clip_image002[5]")
Other symptoms:
-
Host level (external) backups of Hyper-V child partitions failed (while internal backups of VM’s with a backup agents would succeed)
-
During this external backup multiple VHD’s are created but apparently nothing is written to it.
Ultimately the Hyper-V VSS writer faults with a non-tryable error. However, if we move the VM to another host, the backup completes successfully.
Multiple attempts to solve this problem were made:
-
Evict node from cluster and rejoin cluster
-
Remove backup agent (DPM2010) and add it again
-
Remove Server Backup feature and add it again
-
Remove DPM2010 Protection Group and recreate it
-
Backup with and without label on reserved partition
So far the only option left was to reinstall the server.
Just by coincidence I found a recent KB article called “System Partition goes offline on Windows Server 2008 and Windows Server 2008 R2 after installing some 3rd Party Disk or Storage Management Software” dated September 29, 2010: http://support.microsoft.com/kb/2419286/
This article names three issues:
Issue 1:
Hyper-V Role Cannot be Installed “Failure configuring Windows features”
Issue 2:
“Failed to prepare storage for testing on node 1 status 87” during Cluster Validation
Issue 3
“The boot configuration data store could not be opened. The system cannot find the file specified.
In our case it was issue 2 that troubled us.
The resolution was to online the System Reserved Partition:
Diskpart
List volume
Select volume n (n= the 100MB system partition)
Online volume
Exit
Or from Disk Management:
Diskmgmt.msc
Select the 100MB volume and Right-Click it
Change drive letter & path
Assign a drive letter
This is something we had already found out. The bad thing is that it returns after a reboot.
We checked the following:
-
No 3rd party disk or storage management software is installed
-
No anti-virus software is installed on the cluster node
______________________________________________________________
Update: October 6, 2010
Talking to several people in my network we’ve come up with the following (partial and possibly full) solutions:
-
Assign a drive letter, and leave it during a reboot: result was that system partition kept online, but the host level backup of guest partitions with DPM 2010 failed (this idea was presented by several people, a.o. Kurt Roggen and an engineer at Microsoft)
-
Run a chkdsk /r on the system drive (part of solution) and run the system readiness tool which replaces the corrupt mum files from a fully functional Hyper-V host to the one that was having issues:
http://support.microsoft.com/kb/947821 (this tip was presented by Annur Sumar)
Unfortunately we were pressured to get the host running again so we just reinstalled it. That is solution #3 and although not very efficient, one that works in almost all situations 
So thanks for the great feedback to all that contributed!
Twitter
RSS