Veeam Rapidly Fixes Host Level VSS Backup for Hyper-V after KB 2919355 (WS2012 R2 Update)

Update (16 April 2014): see bottom of blog

I recently blogged about Windows Server 2012 R2 Update. As usual any update and certainly an update as large as this, has some risks. Therefore we usually advise to postpone Windows Updates, Update Rollups an Hotfixes and leave a couple of weeks before deploying updates in production. Always test in a lab if you can and if you can’t, keep an eye on the forums and the blogs from MVP’s specializing in the related technology.

Testing Host Level Hyper-V Backup

This weekend I came across a tweet from Richard Skinner who reported an issue related to Veeam Backup & Replication and Hyper-V Backup after applying the Windows Server 2012 R2 (Spring) Update. Meanwhile, Veeam had already confirmed the problem and was working frantically into the weekend to fix this nasty problem.

I decided to report a support case with Veeam as well, even though I’m only running it in a Windows Azure Pack lab. I found that the problem was easily reproducible, but only if VSS was enabled in the backup job.

The Problem: Using Hyper-V Checkpoints

When this selection is made as an alternative to Veeam’s default changed block tracking (CBT), the backup fails because it cannot deal with the file path of the checkpointed VHDX files. When you zoom in on the directory of a VM that is being protected, as soon as VSS kicks in, a checkpoint is made of the active disks. This causes the writes to be redirected from the VHDX to a corresponding AVHDX file which makes it possible for the backup software to take a clean and ‘frozen’ copy of the virtual hard disk. When the backup is ready, the written data to the AVHDX file is merged back into the VHDX file. Only briefly you’ll see n AutoRecovery.avhdx file created which is deleted when it is ready with the merge operation.

Important: Microsoft started to coin the term checkpoint in VMM. After Hyper-V had used the term Snapshot for a long time, this changed with Windows Server 2012 R2. We can now better distinguish between VSS snapshots and Hyper-V checkpoints. Backup software now uses Hyper-V checkpoints just as in Hyper-V Replica.

If you want to read more about the changed method of Hyper-V backup in Windows Server 2012 R2, please take a look at fellow MVP Aidan Finn’s post:


Boot from VHDX for Windows 8.1 Update

You may have seen my previous blogs on how to very quickly make your computer multi-boot into another operating system.

Now that Windows 8.1 Update and Windows Server 2012 R2 Update has been released, I’ll briefly repeat the steps.

  1. Download Convert-WindowsImage.ps1 and copy it to a temporary directory
  2. Start Windows PowerShell ISE in Administrator mode
  3. Run it with .\Convert-WindowsImage.ps1 -ShowUI
  4. Choose the required ISO file
  5. Choose the SKU
  6. Choose the VHD/VHDX Format, Type and Size
  7. Modify the Working Directory if necessary
  8. Type a name for the VHDX
  9. Optionally add an existing Unattend.xml file for further customization
  10. Hit Make the VHD!

You will see

My Windows 8.1 Enterprise VHDX was only 7.6GB small which came as a pleasant surprise.

The VHDX should still be mounted under a drive letter. If not, right-click the VHDX and mount it.
In my case I had ejected the mounted disks and manually re-mounted under drive F:

The following steps are needed to make your computer boot from the VHDX file:

  1. Open an administrative command prompt via WIN+X Command Prompt (Admin)
  2. Type bcdboot F:\Windows
  3. Type bcdedit /v to see the result in the Windows Boot Loader section

Taking the identifier you can change the description in your bootlist by typing:

bcdedit /set {545a3023-1918-11e2-bed1-bd8926e5c774} description “Windows 8.1 Enterprise with Update”

If you had configured Hyper-V on your Windows 8.1 computer, don’t forget to enable the hypervisor launchtype:

bcdedit /set hypervisorlaunchtype auto

Memory Leak on HP ProLiant Servers with NIC Teaming

HP recently published a customer advisory explaining that HP ProLiant servers running Microsoft Windows 2012 as well as Windows Server 2012 R2 and using Microsoft Windows NIC Agent 9.40 may report a memory leak up to 5Mb/hour.

The memory leak is caused by HP’s Microsoft Windows NIC Agent 9.40.

The problem can be easily observed in Task Manager under Processing as you can see in below diagram.

Windows Server 2012 R2 Update is Important

Today we received a note from Microsoft about the importance of Windows Server 2012 R2 Update which is coming to Windows Update on April 8th, 2014. For MSDN and TechNet subscribers this update is already available in the form of an ISO of Windows Server 2012 R2 with the Update included and a smaller collection of 6 updates for Windows 8.1 and Windows Server 2012 R2

In a blog post published on April 2nd, the Microsoft Windows Server Team explains the importance of this update as it provides an easy way to get up to date with the patches, bug-fixes, and improvements that Microsoft has provided since the release of Windows Server 2012 R2. For failover clustering, this update is certainly important as it contains many of the “hoster patches” that Microsoft has worked on in the past six months. The Update consists of six files:

Microsoft recommends to install the Update in this order:

  1. KB2919442
  2. KB2919355
  3. KB2932046
  4. KB2937592
  5. KB2938439
  6. KB2949621

I tried the update on my Windows 8.1 tablet. Because of the large size of KB2919355 (707MB) this update can take a while, so a little patience is required. Of course for multiple clustered Hyper-V hosts, the manual update method is not to be advised. If you have not yet tried Cluster Aware Updating (CAU), an automated cluster aware update tool included with Failover Clustering and available since Windows Server 2012, I strongly advise you to try this out. It not only allows you to install Windows Updates, but also hotfixes, drivers and firmware for your server hardware. Take a look at fellow MVP Didier van Hoye’s blog and video Hands on with Hyper-V Clustering Maintenance Mode & Cluster Aware Updating TechNet Screencast. Another good explanation can be found in chapter 8 of Windows Server 2012 Hyper-V Installation and Configuration Guide by Aidan Finn, Patrick Lownds, Michel Luescher and Damian Flynn.



Makeover Hyper-V Update List for Windows Server 2012 and R2

There are several update lists available on TechNet. Some are curated by the product team and some are kept up-to-date by MVP’s and other people in the community. For easy reference we decided to place a shortcut to these lists in the header of

The following hotfix and update lists are available:

  • Hyper-V: Update List for Windows Server 2012
  • Recommended hotfixes and updates for Windows Server 2012-based failover clusters (updated by Cluster product team)
  • Hyper-V: Update List for Windows Server 2012 R2
  • Recommended hotfixes and updates for Windows Server 2012 R2-based failover clusters clusters (updated by Cluster product team)

Because notably the Windows Server 2012 list had become a bit of a mess, I have rearranged the list, removed outdated or replaced hotfixes and added a sorted date column.

I have also updated the XML file for both Windows Server 2012 and R2 so that you can use a PowerShell cmdlet to quickly scan your Hyper-V hosts if a hotfix or update is installed or not. These files can be downloaded from my OneDrive.

Run the update checker like this:
.\HyperV2012R2UpdatesCheck.ps1 [name host1]



Recommended hotfixes and updates for Windows Server 2012-based failover clusters

VConnect – A Windows Azure Pack Extension for VMware Hosts

While searching for new content for the Windows Azure Pack Wiki, I stumbled on a blog by RaviCK called Ravi’s Cloud 360o which pointed me to a video on how to integrate VMware hosts with Windows Azure Pack. In a recent project we integrated VMware hosts with Windows Azure Pack by means of Virtual Machine Manager and adding vCenter Servers and indirectly adding VMware hosts to a Microsoft Cloud. This approach has a few disadvantages because only standalone Virtual Machines can be deployed and Console Connect does not work for VMs deployed to VMware hosts. All the wealth of VMRole Gallery Items are lost in this solution.

So I was surprised to find that someone has actually written a custom extension for Windows Azure Pack called VConnect from Cloud Assert which brings VMware hypervisors to the platform. Administrators of Windows Azure Pack can now setup plans that provides Virtual Machine services based on VMware hosts.

VConnect is still in beta and only supports a few basic operations such as:

  • Adding a VSphere endpoint of a VMWare hypervisor server
  • Lists the Virtual Machines from all the added servers
  • Basic operations such as Power On, Power Off, Suspend and Reset VM
  • Connect to the VM via Remote Desktop (VMWare tools has to be installed on the VM)
  • Take a screenshot of the Virtual Machine screen
  • ShutDown, Standby and Reboot of Guest OS (VMWare tools has to be installed on the VM)

Take a look at the demo at

HP 3PAR and support for ODX

Over the weekend my fellow MVP Flemming Riis from Denmark contacted me about a new customer bulletin (c04205854) from HP about HP 3PAR StoreServ Storage with HP 3PAR OS 3.1.2 which warns for a serious issue when used with Windows Server 2012 and Windows Server 2012 R2. In the past year we saw and heard of several cases with Live Storage Migration causing corruption of VHDX caused by a bad implementation of ODX in HP 3PAR’s firmware. Previously we advised users to disable ODX at the operating system level as a temporary workaround. Now it seems, HP has found the issue and offers a patch for HP 3PAR OS.


An issue has been discovered with the HP 3PAR OS and the use of Windows Server 2012 Off-loaded Data Transfer (ODX) commands, which may result in a number of blocks incorrectly zeroed beyond the requested range under certain conditions. This issue is not observed with ODX disabled.

Detailed analysis

When the Write Using Token requests using Block Device Zero Token is greater than 16 MBs in size, and the size is not a multiple of 16 MBs, a number of blocks may be zeroed beyond the end of the requested range. Disabling ODX eliminates the use of Write Using Token requests using Block Device Zero Token.


Windows Server 2012 or Windows Server 2012 R2 hosts with ODX in use with HP 3PAR StoreServ Storage running HP 3PAR OS version 3.1.2 GA, 3.1.2 MU1, 3.1.2 MU2, 3.1.2 EMU2, or 3.1.2 MU3.


Upgrade the HP 3PAR OS on the HP 3PAR StoreServ Storage to 3.1.2 MU2 or later if running a lower HP 3PAR OS version. Next apply the patch as follows:

  • For 3.1.2 MU2 and 3.1.2 EMU2, apply Patch 11 followed by Patch 36.
  • For 3.1.2 MU3, apply Patch 30.


Disable ODX on the Windows 2012 or Windows 2012 R2 hosts. To disable ODX:

Set-ItemProperty HKLM:\SYSTEM\CurrentControlSet\Control\FileSystem -Name “FilterSupportedFeaturesMode” -Value 1

A server reboot is required for all servers in which the registry value is modified.

If you have an opportunity to test this patched version of HP 3PAR in your own environment, I would strongly advise you to do this first!

You can find the customer advisory here

Windows Azure Pack – You must first register Service Management Automation on Resource Provider VM Clouds

During a recent Windows Azure Pack deployment at a customer site I encountered an issue with the registration of Service Management Automation. I have done the installation and configuration numerous times without issues before. I performed the same steps at this site and the registration of SMA completed successfully. But when I wanted to link a runbook to an action in the VM Clouds resource provider I was treated with the following surprise. As you can see in the screenshot, the resource provider Automation shows the 27 sample runbooks.

You must first register SMA

We seen some other interesting issues in this environment so I tied this inconsistency to the same list.

My fellow MVP Kristian Nese recently published a blogpost explaining how to re-register SPF in Windows Azure Pack. You can actually use the same cmdlets to unregister SMA (or other resource providers) as well. I unregistered the SMA endpoint on the Windows Azure Pack server with the following cmdlets.

$Credential = Get-Credential

$Token = Get-MgmtSvcToken -Type Windows –AuthenticationSite https://yourauthenticationsite:30072 – ClientRealm http://azureservices/AdminSite -User $Credential -DisableCertificateValidation

Get-MgmtSvcResourceProvider -AdminUri “https://localhost:30004″ -Token $Token -DisableCertificateValidation -name “Automation”

Remove-MgmtSvcResourceProvider -AdminUri “https://localhost:30004″ -Token $Token -DisableCertificateValidation -Name “Automation” -InstanceId “the instance ID you got from Get-MgmtSvcResourceProvider

When you verify the registration status in the admin portal after running the cmdlets you should be able to perform the registration again. I successfully registered the SMA endpoint again in the admin portal.

Register SMA

But the Automation tab in the VM Clouds presented the same surprise again. After poking around with some get- cmdlets and verifying it against a working environment I found a solution. The Service Provider Foundation database is unaware of the Service Management Endpoint. I’m still looking at the root cause, but you can use the some cmdlets on the SPF server to update the SPF database with the endpoint information.

If you encounter the issue described in this blog, make sure you have the SMA endpoint registered in Windows Azure Pack and run the following cmdlets on the SPF server.

import-module spfadmin

Get-SCSpfStamp | fl

$stamp = get-SCSpfStamp –name “Name of the Stamp you got from the Get-SCSpfStamp

New-SCSpfServer –name “IaasAutomation” –ServerType None –Stamps $stamp

$Server = Get-SCSpfServer –name “IaasAutomation”

New-SCSpfSetting –Name EndpointURL –SettingType EndpointconnectionString –Value “https://YourSmaEndpoint:9090/” –Server $Server

After a refresh the admin portal should now reflect the changes we made.

After SPF cmdlets

Yesterday I got a call from Darryl van der Peijl who was deploying a new lab environment and he encountered the exact same issue.If you also see this issue please add a comment to this blog or ping me on twitter @_marcvaneijk

Wish List For Next Version of Windows Azure Pack

Microsoft Azure Pack 2013 can be considered a version 2 of Microsoft’s on premise cloud services. Its predecessor, Windows Azure Services for Windows Server, was offered for free with System Center 2012 SP1 and required Windows Server 2012 and Hyper-V as a platform. WAS4WS can be considered as a typical V1 and offered a first look on how Windows Azure like services can also be offered as a hosted or private cloud. Windows Azure Pack 2013 has been available now for only 6 months, but we can’t imagine it not being available to us. The Windows Azure Pack has had a huge impact on the world of Microsoft cloud services and WAP is here to stay. It not only offers first class IaaS services like the VM Role but also PaaS services with Web Sites and Databases. WAP also offers the platform for new Automation capabilities called SMA or Service Management Automation, which will soon fully replace Orchestrator. Learning PowerShell Workflows has never been more important since the release of SMA.

Windows Azure Pack so far has proven to be a fairly solid product and this can be explained because the Service Management API and the portal software is identical to the public Windows Azure. If you compare the Windows Azure Pack portal to the Windows Azure portal, you can clearly see the advancements that public Windows Azure has made and some of the new functionality that it has acquired.

WAP currently offers Web Site Clouds, VM Clouds, Service Bus Clouds, SQL Servers, MySQL Servers, Automation, Plans and User Accounts.

Windows Azure offers many other services such as Mobile Services, Storage, HDInsight, Media Services, Visual Studio Online, Cache, Biztalk Services, Recovery Services, Traffic Manager, Management Services, Active Directory, Add-Ons and Settings.

For a list of recently added features visit Scott Guthrie’s blog.

In general we could say that both platforms share a common Service Management API and some of the services are identical such as Web Sites and Service Bus. Others rely specifically on features in Virtual Machine Manager and some are only possible because of possibilities/limitations in the operating system. Windows Azure is currenlty still built on Windows Server 2012 and does not support VHDX disks and Console Connect functionality which are both possible in Windows Azure Pack. On the other hand Windows Azure VMs allows adding new disks online, whereas Windows Azure Pack requires the VM to be turned off for adding or expanding a VHDX. This is not a limitation of Windows Server 2012 R2.

Unlike Windows Azure, Windows Azure Pack heavily relies on System Center 2012 R2 and specifically Virtual Machine Manager and Service Provider Foundation. If Usage is added, also Operations Manager is also a required component.


Windows Azure Pack with ADFS and Windows Azure Multi-Factor Authentication – Part 3

In the previous part of this blog series Windows Azure Pack was configured to use ADFS for authentication for the Tenant Site and the Admin Site. We have done numerous implementations of Windows Azure Pack where ADFS was part of the design. In the first production deployments we struggled with setting the correct claim values for Co-Admins on subscriptions and admin access for the Admin Site based on groups, like the issue described at the and of the previous part of this blog series. Since then we have learned (or at least we tried) and there are a couple of ways that you can use to gain some insight into the actual issued claims by ADFS. Now please understand me correctly, there will probably be more ways to do the same. I just collected the procedures that we stumbled upon during the troubleshooting moments. We have used the following functionalities to look at issued claims.

  • ADFS Auditing
  • Get-AdfsToken
  • WIF SDK Claim App

There are probably more or better ways to look at the claims issued by ADFS. If you know any, please don’t hesitate to add them to the comments at the end of this blog post.

ADFS Auditing

Active Directory Federation Service provides a built in functionality to log success and failure audits in the event log of the ADFS server. The success audits contain the actual claims provided by ADFS. Besides enabling this functionality in ADFS, auditing rights must also be enabled for the ADFS service account on the server running ADFS.

The first step is to enable auditing rights for the ADFS service account on the server running ADFS. You can configure this with a local policy or a group policy. Open the local or domain policy that will apply to your ADFS server and browse to the Computer Configuration > Policies > Windows Settings > Security Settings > Local Policies > User Rights Assignment entry.

01 GPO

Open the Generate security audits setting and add the domain service account used in the ADFS configuration wizard in part one of this blog series (domain\SVC_ADFS). Update the policy settings on the ADFS server by running the following command

gpupdate /force

Enable auditing on the ADFS server by running the following command

auditpol.exe /set /subcategory:”Application Generated” /failure:enable /success:enable

Open the ADFS management console. Right-click the root of the entries and select Edit Federation Service Properties.

02 Federation Service

Select the events tab and enable the Success audits checkmark.