Powered by System Center
Posts tagged DPM2010
Hyper-V, DPM and MS Clustering win 2010 Redmond Readers Choice Award
Nov 2nd
Several prizes were won by the products we work with every single day. The subtitle says “Independent Voice of the Microsoft IT Community”. Well I don’t know the magazine very well so I will not begin to doubt that it is. It’s always nice winning prizes, especially if VMware comes in second.
Best Virtual Server Product
6 products in category
Microsoft Windows Server Hyper-V, 30.7 percent, Winner
VMware vSphere, 29.2 percent, Preferred Product, ISV Winner, Five-Star Award
An upset! Finally, Hyper-V, which had been slowly closing the gap, takes this category over 2009 Grand Slam winner VMware. However, vSphere, now ISV Winner five years running, still merits a Five-Star Award honor. For what it’s worth, though, Microsoft is reaching the virtualization market, at least among Redmond readers.
Best Storage Management Product
38 products in category
Microsoft System Center Data Protection Manager, 15.1 percent, Winner
Acronis Backup and Recovery Advanced Server, 12.9 percent, Preferred Product, ISV Winner
EMC ControlCenter SRM Software, 7.9 percent, Preferred Product
Symantec Storage Central, 7.9 percent, Preferred Product
Best Clustering and Failover Solution
11 products in category
Microsoft Windows Server Clustering Services, 29.5 percent, Winner, Five-Star Award
VMware Fault Tolerance, 24.2 percent, Preferred Product, ISV Winner
Barracuda Load Balancer, 9.6 percent, Preferred Product
The margin closes every year, but Microsoft Windows Server Clustering Services held off VMware Fault Tolerance this year to earn a Five-Star Award. The VMware product, having won consecutive ISV awards. Barracuda Load Balancer jumped into Preferred Product status after being shut out in 2009.
http://redmondmag.com/Articles/2010/11/01/2010-Redmond-Readers-Choice-Awards.aspx?Page=6
Hotfix for a specific problem with backup applications on Windows Server 2008 R2 clusters
Oct 1st
This hotfix is only for the described symptoms:
If you see that the Cluster services stops responding in a Windows Server 2008 R2 cluster while some backup application that use VSS in parallel, you might solve this with hotfix KB2277439.
Cause:
This issue occurs because of a race condition between the calls of the VSS writer on the cluster. If an OnBackupShutdown method call occurs between an OnFreeze method call and an OnThaw method call, the lock on the cluster hive is not released. Therefore, a deadlock occurs, and the Cluster service stops responding.
If you are using Hyper-V R2 cluster with CSV, install this hotfix on the coordinator node.![]()
With respect to parallel backups of Hyper-V Virtual Machines with DPM 2010, I can refer to a very helpful blog by Matthew Hodgkins: Enabling Serialized Hyper-V Virtual Machine Backups in DPM 2010 RC
TweetCloud showing my pre-occupations
Sep 28th
Somebody was twittering his TweetCloud the other day. This inspired me to do the same. Just register at http://tweetcloud.com/ and see what you have been up to in Twitterland. Since I use Twitter primarily for business use there are not many words in it that I have to be sorry for. Ok, maybe a few
Conditions when DPM cannot backup a Hyper-V VM
Jul 2nd
We recently saw a situation where a running VM could not be backed up by Data Protection Manager.
Here is why:
If at least one of the following conditions is true
-
The Hyper-V integration components for volume snapshot copy disabled for the VM
-
Dynamic disks configured inside of the VM
-
Non-NTFS based partitions inside the VM
-
Changed shadow storage assignments within the VM
-
VM is not in running state.
In such a case, the VM is put into saved state before the host volumes are snapshotted for back up. For such VMs, Hyper-V writer reports the VM datasouce name in the format "Backup Using Saved State<VMName>". If online backup is possible for a VM, the format will be "Backup Using Child Partition Snapshot<VMName>".
Migrating disks in Data Protection Manager
Jul 1st
Ruud Baars, consultant and DPM expert at Microsoft NL explains how a disk in Data Protection Manager can be migrated to another disk by freeing up space, even if there are volumes spanning multiple disks.
See his blog at Ctrl P – The Data Protection Manager Blog:
http://blogs.technet.com/b/dpm/archive/2010/06/05/space-needed-to-migrate-a-disk.aspx
Hyper-V R2 hotfixes for DPM 2010
Feb 9th
Install at least these two hotfixes on Hyper-V R2 servers:
-
KB975354 A Hyper-V rollup package dealing with parallel backups on same cluster shared volumes and VM’s hosted by different servers; Avoiding data truncation as a result of simultaneous backup of VM from host and from within virtual machine; properly restoring a VM with snapshots to another location
-
KB975921 You may be unable to perform certain disk-related operations after an exception when a hardware provider tries to create a snapshot in Windows Server 2008 R2 or Win 7.
Although not specifically related to Hyper-V and DPM2010, I also install hotfix KB974909 which solves loss of the network connection within a virtual machine with heavy outgoing network traffic and many concurrent network connections. This could easily be the case when a VM level backup is being conducted.
There is also a list of tested hardware VSS providers:
http://blogs.technet.com/dpm/archive/2010/02/05/tested-hardware-vss-provider-table.aspx
I may have a chance to test the HP LeftHand VSS hardware provider, which is not yet on this list. The HP EVA 4×00, 6×00 and 8×00 is on the list by the way.
A comprehensive list of Windows updates and hotfixes for Hyper-V and Hyper-V R2 can be found here:
http://technet.microsoft.com/en-us/library/dd430893(WS.10).aspx
If you need to update your Hyper-V servers, you might just as well include this recent security update:
Microsoft Security Bulletin MS10-010 – Important
Vulnerability in Windows Server 2008 Hyper-V Could Allow Denial of Service (977894)
http://www.microsoft.com/technet/security/bulletin/ms10-010.mspx
When you try to protect VM’s on a Hyper-V R2 clusters with DPM2010 RC you now get a warning if the required hotfixes are not installed.
This is very helpful!
Data Protection Manager 2010 Release Candidate has arrived
Feb 9th
The release candidate of DPM2010 has just been released.
The download location is:
http://connect.microsoft.com/site840/Downloads/DownloadDetails.aspx?DownloadID=26452
The RC features support backup of machines within your intranet, which includes:
- Workgroup machines
- Machines in untrusted domains within your intranet
Please note that in this release, this feature has been built for backing up machines within environment and DPM2010 RC does not support backup of machines outside your intranet, which includes:
- Machines in DMZ network
- Machines in untrusted domains on customers’ sites connected through VPN
In a subsequent release (not sure if this is RTM), support for extranet scenario’s might be added.
Support for Hyper-V R2 clusters and cluster shared volumes (CSV) was already available in the beta.
Important to know is you can upgrade from beta to RC to RTM, or directly from DPM2007 to RC or RTM or any other combination.
In our practice we have deployed quite a number of beta versions of DPM2010 and we are quite happy with the result. I am glad we are getting closer to RTM which is expected to be on schedule for mid April 2010.
DPM2010 RC support can be found here: http://www.microsoft.com/communities/newsgroups/list/en-us/default.aspx?dg=microsoft.public.dataprotectionmanager
A Hyper-V update rollup package is available for Windows Server 2008 R2
Nov 25th
Several issues around VSS based protection & recovery are being addressed with this Hyper-V update rollup package. The package can be downloaded from:
http://support.microsoft.com/hotfix/KBHotfix.aspx?kbnum=975354&kbln=en-us
The last three issues might sound familiar if you already perform host level protection of virtual machines on Cluster Shared Volumes (CSV) with Microsoft System Center Data Protection Manager 2010.
Issues that are fixed in this update rollup package
Issue 1
Consider the following scenario:
- Some Internet SCSI (iSCSI) connections are created in a virtual machine that is running Windows Server 2003.
- You back up this virtual machine on the virtual machine host server.
In this scenario, the error code 0x800423f4 occurs when you back up the virtual machine. Additionally, the following event is logged into the Hyper-V Virtual Machine Management Service event log:
The number of reverted volumes does not match the number of volumes in the snapshot set for virtual machine "’virtual machine name’ (Virtual machine ID <GUID>)".
Cause of Issue 1
When a virtual machine is being backed up, the VSS writer of the server that is running Hyper-V makes a call to the guest virtual machine to check whether any iSCSI connections exists. This call has a default time-out of 60 seconds. If this call does not return within the time limitation, the VSS writer of the server that is running Hyper-V incorrectly assumes that there is no iSCSI connection. Therefore, the backup operation fails.
Issue 2
Consider the following scenario:
- Cluster shared volumes are enabled on a failover cluster for Hyper-V.
- Some virtual machines are saved on the same volume. But they are running on different nodes.
- These virtual machines are backed up in parallel.
In this scenario, the virtual machine backup operation fails.
Cause of Issue 2
When the virtual machines on different nodes are backed up in parallel, every node waits to become the cluster shared volume owner to create the snapshots. However, the Cluster service moves the volume owner from one node to another node immediately after a snapshot is created without waiting for post-snapshot tasks to be completed. If another node requests the same shared volume for a backup operation before the post-snapshot tasks are completed, the Cluster service changes the volume to another node. Therefore, the VSS writer that is in the previous node cannot find the cluster shared volume locally when it performs post-snapshot tasks. This behavior causes the virtual machine backup operation to fail.
Issue 3
Consider the following scenario:
- A virtual machine is being backed up on a server that is running Hyper-V.
- At the same time, an application backup operation is being performed in the same virtual machine.
In this scenario, some data is truncated from the application backup in the virtual machine. Therefore, this behavior causes data loss.
Cause of Issue 3
The application backup operation in the virtual machine is incorrectly affected by the virtual machine backup operation on the server that is running Hyper-V.
Issue 4
Consider the following scenario:
- A virtual machine that has some snapshots is backed up on a server that is running Hyper-V.
- Then, this virtual machine is restored to another location.
In this scenario, the restore operation fails and the virtual machine may be corrupted.
Cause of Issue 4
The snapshot files are not restored successfully when you restore the virtual machine.
Full details: http://support.microsoft.com/kb/975354
How to Protect your MS Virtualized Environment with DPM2010 (Part 3)
Nov 16th
Next in this series is a TechEd Europe 2009 presentation by Asim Mitra, a senior program manager from Microsoft, on data protection and recovery of Hyper-V and Hyper-V R2 workloads with Data Protection Manager 2010.
DPM2010 beta has been publicly available for a few weeks. In my own lab DPM2010 is spinning happily, protecting a Hyper-V R2 cluster with Exchange Server 2010 in a VM on cluster shared volumes (CSV) and several Windows 7 laptops and PC’s. It is quite nice to be in control as a user and determine what to protect from my laptop disks and to be able to restore something whenever I want to. But protecting my production mail server on CSV was really urgent, so I upgraded my DPM2007 SP1 server in-place to DPM2010. It even took care of upgrading the SQL Server 2005 database to SQL Server 2008 without hassle.
Before I went to this session, I had already decided that DPM2010 beta is already more convenient and complete that its predecessor.
The number of supported data sources has grown with DPM2010 and several of them were co-authored with the application developer for a smooth protection and recovery solution. SAP, MS Dynamics re examples of these joint efforts. New data sources are Exchange Server 2010, SharePoint 2010, Hyper-V R2. I probably have missed a few.
In DPM2007 SP1 host-level backup of Hyper-V on Windows Server 2008 and Hyper-V clusters was introduced including Quick Migration.
In DPM2010 this support is extended to Hyper-V R2 with Cluster Shared Volumes (CSV) and is now able to protect virtual machines that are live migrating between hosts. Not many users realized that DPM2007 SP1 was only able to restore VM’s to its original Hyper-V host. Microsoft provided a script to work around that. Now DPM2010 can do Alternate Host Recovery and even Item Level Recovery. This last feature is really unique to DPM!
The question has always been: Should I protect from host or guest?
-
I want to selectively backup individual data objects like databases & files
-
I want to backup each virtual machine as a single object for protection
![]()
The following guidelines can be given to answer those questions:
Host level
-
Protect or recover the whole virtual machine
-
Protect non-Windows servers
-
No granularity of backup
-
“Bare Metal Recovery” and “Item Level Recovery” of every VM
-
Requires single DPM license on host (protecting all guests)
Guest level
-
Protect or recover data specifically
-
-
SQL Server
-
Exchange
-
SharePoint
-
Files
-
-
-
No different than protecting physical server
-
Requires a DPM license per guest (VM)
Whole Node Protection
- This includes the host OS and all VM’s with host level protection
- Requires same single DPM licence on host as in host level protection
Protecting both the parent and the guest is a very cleverly designed cooperation between Volume Shadow Copy writers and requestors. In the example of a Hyper-V R2 server with multiple guests this looks like this:
The Hyper-V VSS writer interacts with a requestor service which is actually the Hyper-V VSS integration component in each VM.
Because there is some time difference between the VSS snapshot of the host and in the guest, there is a potential for data corruption if this wouldn’t be handled correctly. So DPM2010 takes a post-snapshot step to fix the data.
In this post-snapshot step the VHD in the guest is mounted and the amount of changed bytes or blocks are synchronized between the host VSS snapshot and the client VSS snapshot. It sounds easy and it is easy!
Protecting a Live Migrating VM
A more challenging task is the proper protection of a guest which is in the process of migrating between Hyper-V R2 hosts. DPM2010 is able to handle this as well. It works like this:
-
DPM2010 performs an incremental backup of VM from cluster node A
-
The VM then migrates for instance to cluster node C
-
DPM2010 automatically performs the next incremental backup of the Live Migrated VM from cluster node C
DPM has to be aware of the new shared storage architecture of Hyper-V R2. CSV is implemented as a filter driver and sits directly on top of NTFS. DPM can only make a new incremental backup if it knows how to handle the underlying disk architecture. In a Hyper-V R2 cluster all nodes in the cluster can read from a disk in the CSV pool. They can also write to the disk with the VHD on it. Only the so-called coordinator node has full access to the metadata of the underlying physical disk. This coordinator node is not static and can move between cluster nodes, or else it would be a single point of failure. DPM is clever enough to failover the coordinator role to the node in the cluster that needs to take an incremental backup. So if node A owns the disk, and the VM is moved to node B, the DPM agent moves the CSV disk also to node B. This effectively switches the node from Indirect I/O to Direct I/O which makes it possible to make a local incremental backup of the VHD’s from the correct cluster node. Solved that!
There are several Hyper-V recovery options with DPM2010:
-
Restore VM back to original host or cluster
-
Restore VM to a different host or cluster
-
Item Level Recovery (ILR) to file share
This ILR functionality requires that the Hyper-V R2 role is enabled on the DPM2010 server. Hyper-V will not have to do anything else but attach the VHD. This sounds odd since R2 can do this natively without the Hyper-V role enabled.
Planning the deployment
Normally when I plan for DPM storage I roughly calculate 300% the amount of used production disk to be used for the DPM Storage Pool. This is often enough for a retention time of about 14 days on disk. The applied schedule is not relevant for this calculation, because it doesn’t matter whether you synch the data once a day or once an hour. The amount changes per day remain the same. It becomes a different matter if you also want to protect complete guests. Some data would have to be protected multiple times. With ILR this problem is largely eliminated. But to help planning the deployment, the DPM Team offers a Storage Requirements Calculator For Hyper-V Workloads (which is currently in development). We could download a pre-release from the TechEd website.
Why is DPM a suitable data protection product for Hyper-V?
-
Incremental backups only – full once (first replica)
-
No more backup window – online backups
-
Application consistency via VSS
-
Protect Live Migration VM’s in CSV clusters
-
Protects whole VM and recovers individual items
-
Auto protects new VM’s
-
Routine backups (nightly or more frequently)
As part of Microsoft System Center, Microsoft claims that:
-
It is the best product for protecting Windows file and application servers
-
It is built for Microsoft Virtualization environments
-
It is designed for Windows Clients
-
It has Enterprise-Ready scalability and reliablity
Data Protection Manager 2010 beta now available for download
Nov 4th
Today the Data Protection Manager development team released the public beta of Microsoft System Center Data Protection Manager (DPM2010), the third generation of Microsoft backup software. Actually backup is not the right word since it is data protection and guaranteed data recovery what makes DPM really stand out. It was really designed for backup to disk.
The first generation (DPM2006) was a first but limited attempt to offer protection and easy recovery for file servers. Much like the volume shadowcopy for shared folders, it was easier to protect fileservers from a central location, creating replica’s and recovery points to a DPM storage pool. With DPMv1 there was no direct tape support and a second backup product was required for long term protection.
Towards the end of 2007 the second generation arrived with DPM2007. It added other data source protection such as SQL Server, Exchange, Sharepoint Portal Server, Virtual Server and SystemState. An important missing data source was Hyper-V. Even though Hyper-V R1, as we can call it now, arrived in February 2009 as a beta with the RTM of Windows Server 2008 and the RTM of Hyper-V R1 in June 2008, we had to wait until January 2009 before Hyper-V data sources could be protected at the host level with SP1 of DPM2007. Microsoft Office SharePoint System (MOSS) and SQL Server 2008 were also added.
However, having to wait for seven months before being able to protect a very strategic product like Hyper-V is unacceptable to say the least. There was also no clear message when Hyper-V protection would arrive. Compared to the 60 days after Hyper-V R2 message from the Virtual Machine Manager team, this leaves room for improvement.
Hopefully the gap between Hyper-V R2 RTM and DPM2010 RTM will be smaller, although I have reason to doubt this. Expect the final version of the third generation DPM somewhere in the first quarter of 2010. Again much too late for customers who want to protect virtual machines which are placed on Cluster Shared Volumes.
This leaves us with adding a DPM Agent to the virtual machines only and forgetting about the host until April 2010 or so. But that’s enough for the criticism.
We can finally lay our hands on the beta of DPM2010 after about 50 customers have been able to test the new data protection product in a Technical Adoption Program. Apparently DPM2010 is ready to show to the world and welcome it is. The majority of our customers have abandoned their traditional Symantec Backup Exec, NetBackup and HP Data Protector backup programs in favor of DPM and I must say the product has become notably better in the last few months with SP1 and a couple of hotfixes. Nevertheless there were plenty of product improvements I have been able to share with the development team and hopefully most of these have made it in the product.
In the light of this Hyper-V blog we are of course most focused on protection of Microsoft virtualized environments and specifically Hyper-V R2.
Here is a quote from the DPM blog:
Virtualization
“This has been one of the biggest investments that we made in DPM 2010, and we hope that you will absolutely love the features. First and foremost, DPM 2010 Beta protects highly available virtual machines (VM) deployed on Windows Server 2008 R2 using Cluster Shared Volumes (CSV) clusters — in addition to standalone Hyper-V servers and Windows Server 2008 Hyper-V clusters. For all above mentioned server configurations, DPM 2010 Beta supports:
Seamless protection of Live Migrating VMs (For Windows Server 2008 R2): DPM 2010 is Live Migration aware and seamlessly protects a VM after it migrates to another node of the Hyper-V R2 cluster to another without manual intervention.
Item Level Recovery from host level backup: DPM 2010 Beta supports item level recovery (ILR) which allows you to do granular recovery of files and folders, volumes and virtual hard disks (VHD) from a host level backup of Hyper-V VMs to a network share or a volume on a DPM protected server.
Original Location Recovery: DPM 2010 Beta supports online recovery of the protected VM to the original location.
Alternate Host Recovery: DPM 2010 Beta supports alternate location recovery (ALR) which allows you to recover a Hyper-V VM to an alternate stand-alone or clustered Hyper-V host.”
Notable the Item Level Recovery from host level backup is a really fantastic feature which for certain can not be matched by any product in the VMware world. We have Bare Metal Recovery of virtual machines including the ability to perform a specific restore of selected items within the host level backup of the virtual hard disk (VHD).
Of course there is a shipload of improvements to the product which can be found at the following locations:
http://blogs.technet.com/DPM/
http://www.microsoft.com/systemcenter/dataprotectionmanager/en/us/2010beta-overview.aspx









Twitter
RSS