Powered by System Center
Posts tagged VSS
Hyper-V Protection with Data Protection Manager 2012 SP1
Dec 27th
In the quiet days between Christmas and New Year, I had some time to research how DPM2012 SP1 performed with protecting guests on a Windows Server 2012 Hyper-V cluster using CSV v2.0.
According to the SP1 release notes we can expect improved backup performance of Windows Server 2012 Hyper-V over CSV deployments with the following benefits:
-
900% improvement in Express Full backups
-
Parallel backups
-
No performance difference between backups from CSV owners and non-owners
Let me first point out that my setup is based on the following configuration:
Storage Server
-
HP ProLiant BL460c Gen8
-
Windows Server 2012 Datacenter with iSCSI Target Server Role enabled
-
Dual-port 10Gb HP FlexFabric 554FLB (Emulex) Adapter
-
Converged Fabric network configuration
Hyper-V Cluster Nodes
-
HP ProLiant BL460c Gen8
-
Windows Server 2012 Datacenter with Hyper-V role and Failover Cluster feature enabled
-
10Gb HP FlexFabric 554FLB (Emulex) Adapter
-
Converged Fabric network configuration
DPM 2012 SP1 Server
-
Windows Server 2012 Hyper-V Virtual Machine
-
Windows Server 2012 Datacenter
-
1 x 10Gb virtual network adapter (synthetic)
Network Configuration
Both on the iSCSI Target Server and Hyper-V cluster nodes, the two 10Gb network adapters have been teamed using a switch independent teaming mode with Hyper-V Port as the load balancing algorithm. A Hyper-V Extensible Switch is connected to the NIC Team and several virtual networks have been configured using the Converged Fabric method of Windows Server 2012. Each network has a minimum bandwidth Quality of Service configured on the virtual switch level. On the backend, the servers use a Virtual Connect Flex-10 interconnect.
VSS Crash-Consistent vs Application-Consistent VSS Backups (2)
May 8th
This second blog by Altaro dealing with Hyper-V backup discusses when Application-Consistent Backup is vital for proper recovery or not. What if a VSS Writer is not available for a particular application? What are the general requirements for Application-Consistent Backups in Hyper-V? What are the Special Considerations for Cluster Shared Volumes?
When is Application-Consistent Backup Vital? Not all situations require an application-consistent backup. Things such as file and print servers will be fine with crash-consistent and possibly inconsistent backups. If your application doesn’t provide a VSS writer, there might not even be a way to get an application-consistent backup of it while its containing machine is live. The most common need for application-consistent backups is the usage of database-backed applications.
The full blog can be found here:
http://www.altaro.com/blog/vss-crash-consistent-vs-application-consistent-vss-backups-post-2-of-2/
VSS Crash-Consistent vs Application-Consistent VSS Backups (1)
May 1st
I can recommend an interesting 2-part series of blogs which helps you get a better understanding of the theory behind successful backups. In the first blog on Altaro’s Hyper-V blog, VSS Crash-Consistent Backups are compared to Application-Consistent VSS Backups.
When designing any IT solution, many administrators often consider “Backup” to be little more than another box on a long list of items to check off. They verify that the software and hardware they’re using will handle the load, configure it to back up on a reasonable schedule, and forget about it. Some will take the extra step of restoring some data to an alternate location as a test. Hardly any go through the full exercise of simulating an actual catastrophe. Most of the time, this practice is completely harmless. Unfortunately, if disaster does strike, there are often more questions than answers. Planning ahead is critical, and that involves knowing what sort of backup you need and if your backup application can provide it.
You can continue the article here:
http://www.altaro.com/blog/vss-crash-consistent-vs-application-consistent-vss-backups-post-1-of-2/
Look out for the second part of this blog!
Reviewing Altaro Hyper-V Backup including CSV support
Aug 2nd
Back in May, Femi Adegoke reviewed an early beta version of Altaro Hyper-V Backup which was geared to easily backup & recover Hyper-V guests using the Hyper-V VSS Writer. Now Femi submitted his second guest blog focusing on how to protect guests in a clustered Hyper-V environment.
By Femi Adegoke
This is a follow up to our previous blog post from May 2011:
http://www.hyper-v.nu/archives/hvredevoort/2011/05/altaro-hyper-v-backup-review/
One interesting feature in this Altaro Hyper-V Backup release is support for CSV:
http://www.altaro.com/hyper-v-backup/
The current revision of the software is 2.0.8.0 BETA, download size is approximately 12.1 MB.
Our test was performed on a Dell 4 node Hyper-V cluster running 2008 R2 SP1 with a few virtual machines.
Installation
Installing the software was fast & easy and involved only a few mouse clicks. Can you believe that?
-
This software installs in less than 3 minutes per node.
-
No agent is required inside each guest
Dashboard
Dashboard view (this so schweeet…)
Sanbolic releases Melio version 3.5
Jun 20th
Today Sanbolic announced the availability of its latest version of its distributed clustered file system Melio version 3.5, which is currently the best available alternative for Hyper-V Cluster Shared Volumes. I have helped Sanbolic test and improve some of its earlier versions and was very impressed.
Co-founder and CEO of Sanbolic, Momchil Michailov just informed me that both DPM 2010 and Symantec BackupExec are supported, that the product will ship before the end of the week. Sanbolic has much more to offer that simply to replace CSV. If you haven’t looked at this product and are thinking of building large Hyper-V clusters, VDI or SQL deployments where enterprise ready storage is expected without the compromises, I strongly advise you to start evaluating Sanbolic’s file system Melio version 3.5
If you need any assistance with testing this product or writing a design, don’t hesitate to contact me and make me part of your project.
PS
I have no shares in Sanbolic (although maybe I should)
Hotfix for a specific problem with backup applications on Windows Server 2008 R2 clusters
Oct 1st
This hotfix is only for the described symptoms:
If you see that the Cluster services stops responding in a Windows Server 2008 R2 cluster while some backup application that use VSS in parallel, you might solve this with hotfix KB2277439.
Cause:
This issue occurs because of a race condition between the calls of the VSS writer on the cluster. If an OnBackupShutdown method call occurs between an OnFreeze method call and an OnThaw method call, the lock on the cluster hive is not released. Therefore, a deadlock occurs, and the Cluster service stops responding.
If you are using Hyper-V R2 cluster with CSV, install this hotfix on the coordinator node.![]()
With respect to parallel backups of Hyper-V Virtual Machines with DPM 2010, I can refer to a very helpful blog by Matthew Hodgkins: Enabling Serialized Hyper-V Virtual Machine Backups in DPM 2010 RC
Conditions when DPM cannot backup a Hyper-V VM
Jul 2nd
We recently saw a situation where a running VM could not be backed up by Data Protection Manager.
Here is why:
If at least one of the following conditions is true
-
The Hyper-V integration components for volume snapshot copy disabled for the VM
-
Dynamic disks configured inside of the VM
-
Non-NTFS based partitions inside the VM
-
Changed shadow storage assignments within the VM
-
VM is not in running state.
In such a case, the VM is put into saved state before the host volumes are snapshotted for back up. For such VMs, Hyper-V writer reports the VM datasouce name in the format "Backup Using Saved State<VMName>". If online backup is possible for a VM, the format will be "Backup Using Child Partition Snapshot<VMName>".
Hyper-V R2 hotfixes for DPM 2010
Feb 9th
Install at least these two hotfixes on Hyper-V R2 servers:
-
KB975354 A Hyper-V rollup package dealing with parallel backups on same cluster shared volumes and VM’s hosted by different servers; Avoiding data truncation as a result of simultaneous backup of VM from host and from within virtual machine; properly restoring a VM with snapshots to another location
-
KB975921 You may be unable to perform certain disk-related operations after an exception when a hardware provider tries to create a snapshot in Windows Server 2008 R2 or Win 7.
Although not specifically related to Hyper-V and DPM2010, I also install hotfix KB974909 which solves loss of the network connection within a virtual machine with heavy outgoing network traffic and many concurrent network connections. This could easily be the case when a VM level backup is being conducted.
There is also a list of tested hardware VSS providers:
http://blogs.technet.com/dpm/archive/2010/02/05/tested-hardware-vss-provider-table.aspx
I may have a chance to test the HP LeftHand VSS hardware provider, which is not yet on this list. The HP EVA 4×00, 6×00 and 8×00 is on the list by the way.
A comprehensive list of Windows updates and hotfixes for Hyper-V and Hyper-V R2 can be found here:
http://technet.microsoft.com/en-us/library/dd430893(WS.10).aspx
If you need to update your Hyper-V servers, you might just as well include this recent security update:
Microsoft Security Bulletin MS10-010 – Important
Vulnerability in Windows Server 2008 Hyper-V Could Allow Denial of Service (977894)
http://www.microsoft.com/technet/security/bulletin/ms10-010.mspx
When you try to protect VM’s on a Hyper-V R2 clusters with DPM2010 RC you now get a warning if the required hotfixes are not installed.
This is very helpful!
HP promises tighter integration of HP LeftHand and Volume Shadow Copy (VSS)
Feb 3rd
HPStorageGuy tweeted a very interesting reference to an article by Mike Moroze from HP LeftHand about tighter integration with VSS.
A new option in HP LeftHand P4000 iSCSI storage systems, which we use a lot at Nobel in Hyper-V R2 clusters, now fully embraces the Volume Shadowcopy Services framework.
Currently the preferred method for backup up fair quantities of Hyper-V virtual machines is using the VDS hardware provider of the SAN vendor. This would in turn trigger the storage regular hardware snapshot method. Being a 100% VSS solution, I expect DPM2010 to benefit a great deal from this new LeftHand option to off VSS for creating hardware based snapshots. It would alleviate the Hyper-V host to a great extent because IO Redirection can be kept very short for the CSV LUNs on which the VM’s reside which are being snapshotted.
Because there is no similar VMware integration (yet, if at all via VSS), the VSS option for LeftHand is not yet the default. It will be default for most of our customers (grin).
For a single snapshot:
For a schedule to snapshot a volume:
How to Protect your MS Virtualized Environment with DPM2010 (Part 3)
Nov 16th
Next in this series is a TechEd Europe 2009 presentation by Asim Mitra, a senior program manager from Microsoft, on data protection and recovery of Hyper-V and Hyper-V R2 workloads with Data Protection Manager 2010.
DPM2010 beta has been publicly available for a few weeks. In my own lab DPM2010 is spinning happily, protecting a Hyper-V R2 cluster with Exchange Server 2010 in a VM on cluster shared volumes (CSV) and several Windows 7 laptops and PC’s. It is quite nice to be in control as a user and determine what to protect from my laptop disks and to be able to restore something whenever I want to. But protecting my production mail server on CSV was really urgent, so I upgraded my DPM2007 SP1 server in-place to DPM2010. It even took care of upgrading the SQL Server 2005 database to SQL Server 2008 without hassle.
Before I went to this session, I had already decided that DPM2010 beta is already more convenient and complete that its predecessor.
The number of supported data sources has grown with DPM2010 and several of them were co-authored with the application developer for a smooth protection and recovery solution. SAP, MS Dynamics re examples of these joint efforts. New data sources are Exchange Server 2010, SharePoint 2010, Hyper-V R2. I probably have missed a few.
In DPM2007 SP1 host-level backup of Hyper-V on Windows Server 2008 and Hyper-V clusters was introduced including Quick Migration.
In DPM2010 this support is extended to Hyper-V R2 with Cluster Shared Volumes (CSV) and is now able to protect virtual machines that are live migrating between hosts. Not many users realized that DPM2007 SP1 was only able to restore VM’s to its original Hyper-V host. Microsoft provided a script to work around that. Now DPM2010 can do Alternate Host Recovery and even Item Level Recovery. This last feature is really unique to DPM!
The question has always been: Should I protect from host or guest?
-
I want to selectively backup individual data objects like databases & files
-
I want to backup each virtual machine as a single object for protection
![]()
The following guidelines can be given to answer those questions:
Host level
-
Protect or recover the whole virtual machine
-
Protect non-Windows servers
-
No granularity of backup
-
“Bare Metal Recovery” and “Item Level Recovery” of every VM
-
Requires single DPM license on host (protecting all guests)
Guest level
-
Protect or recover data specifically
-
-
SQL Server
-
Exchange
-
SharePoint
-
Files
-
-
-
No different than protecting physical server
-
Requires a DPM license per guest (VM)
Whole Node Protection
- This includes the host OS and all VM’s with host level protection
- Requires same single DPM licence on host as in host level protection
Protecting both the parent and the guest is a very cleverly designed cooperation between Volume Shadow Copy writers and requestors. In the example of a Hyper-V R2 server with multiple guests this looks like this:
The Hyper-V VSS writer interacts with a requestor service which is actually the Hyper-V VSS integration component in each VM.
Because there is some time difference between the VSS snapshot of the host and in the guest, there is a potential for data corruption if this wouldn’t be handled correctly. So DPM2010 takes a post-snapshot step to fix the data.
In this post-snapshot step the VHD in the guest is mounted and the amount of changed bytes or blocks are synchronized between the host VSS snapshot and the client VSS snapshot. It sounds easy and it is easy!
Protecting a Live Migrating VM
A more challenging task is the proper protection of a guest which is in the process of migrating between Hyper-V R2 hosts. DPM2010 is able to handle this as well. It works like this:
-
DPM2010 performs an incremental backup of VM from cluster node A
-
The VM then migrates for instance to cluster node C
-
DPM2010 automatically performs the next incremental backup of the Live Migrated VM from cluster node C
DPM has to be aware of the new shared storage architecture of Hyper-V R2. CSV is implemented as a filter driver and sits directly on top of NTFS. DPM can only make a new incremental backup if it knows how to handle the underlying disk architecture. In a Hyper-V R2 cluster all nodes in the cluster can read from a disk in the CSV pool. They can also write to the disk with the VHD on it. Only the so-called coordinator node has full access to the metadata of the underlying physical disk. This coordinator node is not static and can move between cluster nodes, or else it would be a single point of failure. DPM is clever enough to failover the coordinator role to the node in the cluster that needs to take an incremental backup. So if node A owns the disk, and the VM is moved to node B, the DPM agent moves the CSV disk also to node B. This effectively switches the node from Indirect I/O to Direct I/O which makes it possible to make a local incremental backup of the VHD’s from the correct cluster node. Solved that!
There are several Hyper-V recovery options with DPM2010:
-
Restore VM back to original host or cluster
-
Restore VM to a different host or cluster
-
Item Level Recovery (ILR) to file share
This ILR functionality requires that the Hyper-V R2 role is enabled on the DPM2010 server. Hyper-V will not have to do anything else but attach the VHD. This sounds odd since R2 can do this natively without the Hyper-V role enabled.
Planning the deployment
Normally when I plan for DPM storage I roughly calculate 300% the amount of used production disk to be used for the DPM Storage Pool. This is often enough for a retention time of about 14 days on disk. The applied schedule is not relevant for this calculation, because it doesn’t matter whether you synch the data once a day or once an hour. The amount changes per day remain the same. It becomes a different matter if you also want to protect complete guests. Some data would have to be protected multiple times. With ILR this problem is largely eliminated. But to help planning the deployment, the DPM Team offers a Storage Requirements Calculator For Hyper-V Workloads (which is currently in development). We could download a pre-release from the TechEd website.
Why is DPM a suitable data protection product for Hyper-V?
-
Incremental backups only – full once (first replica)
-
No more backup window – online backups
-
Application consistency via VSS
-
Protect Live Migration VM’s in CSV clusters
-
Protects whole VM and recovers individual items
-
Auto protects new VM’s
-
Routine backups (nightly or more frequently)
As part of Microsoft System Center, Microsoft claims that:
-
It is the best product for protecting Windows file and application servers
-
It is built for Microsoft Virtualization environments
-
It is designed for Windows Clients
-
It has Enterprise-Ready scalability and reliablity












Twitter
RSS