Today I had a conference call with the people from Emulex responsible for the network adapters, firmware and drivers. As many of you know we’ve had a long lasting issue with HP/Emulex 554FLB CNAs in HP BL460c Gen 8 blade servers in c7000 blade enclosures. After we had replaced Windows Server 2012 with Hyper-V by Windows Server 2012 R2 with Hyper-V on the same hardware, we started to notice virtual machines losing connection. We have multiple guest clusters on top of the Hyper-V clusters and sometimes during Live Migration of one of the cluster nodes, we would see that connectivity was lost, even to the point that a cluster node would be forced to leave the cluster and come back later when the network connection was re-established. In fact that was our single best test to reproduce the problem.
Marc van Eijk en Peter Noorderijk wrote blogs about it and together they got over 200 comments from customers all over the world seeing the same problem, the majority of them having Emulex through an OEM like HP, Dell, IBM and Cisco. In fact we also read similar cases with HP rack servers, other NICs and different hardware combinations which led me to believe there could also be an issue in the networking/teaming stack in Windows Server 2012 R2.
During the months of November and December 2013 we collected a list of 10 registered support cases with either HP, Microsoft or both. At that time my primary contact was a senior escalation engineer at Microsoft who was able to collect several customer cases mostly from European customers. Unfortunately we could not register a support case with HP ourselves because we did not have a support contract with them. But the amount of customers that discovered they had the same problem, grew and grew. There must have been hundreds of open support calls with both HP and Microsoft and the storm of discontent was growing.
According to HP they were dependent on Emulex and HP did not get any feedback either. Also Microsoft was left in the dark for a long time.
READ MORE »
If you like System Center, Hyper-V, Azure as well as meat, don’t forget to register for System Center Summer Night “The MasterChef edition” which is rapidly approaching. In 10 days from today, nine experts, seven of them are MVPs, will present five interesting presentations.
Because we have plenty of space left we invite those who have registered to bring a friend without additional cost. If you haven’t registered yet, please do and have a great afternoon which is completed with a nice barbecue.
Date of event
15:00 – 16:15 How many System Center fits on one grill by Ronny de Jong / James van den Berg [MVP] / Helmer Zandbergen / Marc van Eijk [MVP] / Dieter Wijckmans [MVP]
16:15 – 17:30 How Service Manager can do everything you need – a best-of-the-best Swiss cheese selection by Marcel Zehner [MVP]
17:30 – 17:45 Break
17:45 – 19:00 Light up the fire on your Hyper-V by Hans Vredevoort [MVP] & Peter Noorderijk on Hyper-V Architecture
19:00 – 20:15 Become a Masterchef on Microsoft Azure Automation by Maarten Goet [MVP]
20.15 BBQ time
This event is organized by SCUG.nl and Hyper-V.nu
Please register at https://www.eventbrite.nl/e/tickets-system-center-summer-night-2014-9265847399
4103 NM Culemborg
Back in 2011, four Hyper-V MVPs decided to take on a server virtualization master class series focusing on Hyper-V for which they adopted the Spanish sounding name “The Hyper-V Amigos”. In the previous months German Hyper-V MVP Carsten Rachfahl, well known for his Hyper-V podcasts and videos, had already made three episodes with Belgian Hyper-V MVP, Didier van Hoye aka @WorkingHardInIT which gave us a lot of background of both Carsten and Didier in Episode 1, as well as some great explanatory showcasts on Unmap and Live Migration in Windows Server 2012 R2 in Episodes 2 and 3.
This time Carsten also invited Aidan Finn, Hyper-V MVP out of Ireland and myself to what seemed like a reunion. Three of the MVPs had visited TechEd North America and in Episode 4 they look back on how they came to know as The Hyper-V Amigos and what their thoughts were on the TechEd 2014 event.
I invite you to watch the next episode of The Hyper-V Amigos:
The update rollup for June 2014 fixes the issues that are documented in the following Microsoft Knowledge Base (KB) articles:
Update for data deduplication to improve scalability in Windows Server 2012 R2
You are prompted for BitLocker recovery key when Windows enters the automatic repair process
“Hyper-V Replica Cluster Broker is not installed” error when you replicate private clouds to Windows Azure
DNS server crashes after you install update 2919355 for Windows Server 2012 R2.
Connectivity lost between two nodes when a node reconnects to a Windows Server 2012 R2-based cluster
CPrepSrv.exe process crashes or Failover Cluster Manager freezes when you validate storage in Windows Server 2012 R2
Removed nodes can access shared disk resources unexpectedly in Windows Server 2012 R2
You cannot stop the cluster service on a Windows Server 2012 R2-based failover cluster
Storage spaces take a long time to move to another node after a node fails on a Windows Server 2012 R2 failover cluster
STS passive sign-in fails when a sign-in request is sent to a Windows Server 2012 R2-based STS server through STS proxy
AD FS device authentication is slow or fails in Windows Server 2012 R2
Authentication failures and event 422 when AD FS STS servers and AD FS proxy servers are in Windows Server 2012 R2
Long wait when you first open File Explorer in Windows RT 8.1 or Windows 8.1
Virtual machine network fails when you start the second VM on a Windows Server 2012 R2-based Hyper-V server
Windows Update does not download drivers for shared printers in Windows 8.1 or Windows Server 2012 R2
Error occurs when you run Get-VirtualDisk|Get-ClusterResource cmdlet in Windows 8.1 or Windows Server 2012 R2
OneDrive improvement update for Windows RT 8.1 and Windows 8.1: June 2014
“0x80041013″ error on a WMI provider in Windows RT 8.1, Windows 8.1, or Windows Server 2012 R2
“There was a problem” error when you redeem a promotional CSV token in Windows 8.1 or Windows Server 2012 R2
Cannot select Chinese suggestion words from on-screen keyboard in Windows 8.1 or Windows Server 2012 R2
Settings are migrated incorrectly after you refresh the system by using PBR in Windows RT 8.1 or Windows 8.1
Logon UI crashes when you connect to a remote server that is running a Windows Server 2012 R2 Core installation
Update to improve the OneDrive experience in Windows RT 8.1 and Windows 8.1
Excel freezes when you convert Japanese characters in Windows
Audit event ID 4661 triggers an invalid XML error in a Windows Server 2012 R2 or Windows Server 2008 environment
“The CA certificate could not be retrieved, element not found” error occurs when the CA server host name is longer than 52 characters
- 2936341(http://support.microsoft.com/kb/2936341/ ) The WebClient service does not send cookies in Windows
During my last updating round, I noticed that a number of VMs in my Windows Azure Pack lab, had problems with security update KB2920189. Reading the Microsoft Security Advisory, it states that Microsoft is revoking the digital signature for four private, third-party UEFI (Unified Extensible Firmware Interface) modules that could be loaded during UEFI Secure Boot.
These UEFI (Unified Extensible Firmware Interface) modules are partner modules distributed in backup and recovery software. When the update is applied, the affected UEFI modules will no longer be trusted and will no longer load on systems where UEFI Secure Boot is enabled. The affected UEFI modules consist of specific Microsoft-signed modules that are not in compliance with our certification program and are being revoked at the request of the author.
Microsoft is not aware of any misuse of the affected UEFI modules. Microsoft is proactively revoking these non-compliant modules in coordination with their author as part of ongoing efforts to protect customers. This action only affects systems running Windows 8, Windows 8.1, Windows Server 2012, and Windows Server 2012 R2 that are capable of UEFI Secure Boot where the system is configured to boot via UEFI and Secure Boot is enabled. There is no action on systems that do not support UEFI Secure Boot or where it is disabled.
I concluded that this update only targeted Hyper-V Generation 2 VMs with Secure Boot enabled, which was in fact the case for all VMs involved.
No matter how many times I tried, each update ended up failed. READ MORE »
Recently I needed to update a Windows Azure Pack installation from RTM to Update 2. Because the servers had no Internet access, I needed an offline approach.
One way is to download the Windows Azure Pack 2013 Update 2 components, extract the files and install them one by one, thereby carefully checking that only the installed components are updated. It is very easy to make a mistake and add unwanted ones on a WAP portal server.
Because this approach only works for the WAP 2013 Update 2 components and doesn’t give you the partner components such as Gridpro and Cloud Cruiser, the offline Web Platform Installer approach is the best choice. In fact WebPI has been upgraded from version 4.6 to 5.0 containing Update 2 plus the 3rd party components.
READ MORE »
Fellow blogger at Hyper-V.nu Marc van Eijk did an amazing job. At his first TechEd he delivered no less than three presentations. I went to every single one of them and was really proud to see him be so successful. Great job Marc!
Lessons Learned: Designing and Deploying the Windows Azure Pack in the Real World
Effortless Migration from VMware to Windows Server 2012 R2 Hyper-V
Transforming Bare Metal into Logical Switches Using Microsoft System Center 2012 R2 Virtual Machine Manager
Cameron Fuller and Maarten Goet realized that there was still a full day available and many MVPs were still hanging around in Houston on Friday. In cooperation with the Houston Area System Management User Group (HASMUG) they found enough sponsors to organize #TE14D5 or an informal community event offering two tracks: Enterprise Client Management (ECM) and Cloud and Datacenter Management (CDM).
Both Marc van Eijk and Hans Vredevoort delivered a presentation:
Windows Azure Pack Usage Service and the System Center Components that it depends on
by Hans Vredevoort
WAP showdown – VM Template vs. VM Role
by Marc van Eijk
Today I saw several users having problems with their Data Protection Manager 2012 R2 Server after installing KB2958100 which is the specific update for DPM in Update Rollup 2.
My advice is to postpone the DPM update and allow Microsoft to look into the user reports. Correction is quite complex and requires modifications in the DPM database.
See the thread for more details:
Along with Update Rollup 2 for System Center 2012 R2 (KB2932881), also Update 2 (KB2932946) for Windows Azure Pack was released. Update 2 fixes 10 known issues, amongst others a fix for the problem created with the Usage Collector in Update 1. Because WAP does not currently support rolling upgrades, it is necessary to schedule downtime for the WAP machines. Please be careful running this update in your production environment and expose them to a test environment first. There are so many components which depend on each other so take any precautions in the form of checkpoints (formerly called Hyper-V snapshots) and make sure you have recent full backups of your environment.
Although it is stated nowhere, you would be advised to first start with the System Center 2012 R2 Update Rollup 2 and run the documented SQL Scripts for VMM and SCOM first (important!) before you approve the Windows Azure Pack Update 2 installation. The SQL Script for VMM can be found in http://support.microsoft.com/kb/2932926. A reference to the path of the SQL Scripts for the SCOM OperationsManager and OperationsManagerDW databases can be found in http://support.microsoft.com/kb/2929891.
READ MORE »
Update (16 April 2014): see bottom of blog
I recently blogged about Windows Server 2012 R2 Update. As usual any update and certainly an update as large as this, has some risks. Therefore we usually advise to postpone Windows Updates, Update Rollups an Hotfixes and leave a couple of weeks before deploying updates in production. Always test in a lab if you can and if you can’t, keep an eye on the forums and the blogs from MVP’s specializing in the related technology.
Testing Host Level Hyper-V Backup
This weekend I came across a tweet from Richard Skinner who reported an issue related to Veeam Backup & Replication and Hyper-V Backup after applying the Windows Server 2012 R2 (Spring) Update. Meanwhile, Veeam had already confirmed the problem and was working frantically into the weekend to fix this nasty problem.
I decided to report a support case with Veeam as well, even though I’m only running it in a Windows Azure Pack lab. I found that the problem was easily reproducible, but only if VSS was enabled in the backup job.
The Problem: Using Hyper-V Checkpoints
When this selection is made as an alternative to Veeam’s default changed block tracking (CBT), the backup fails because it cannot deal with the file path of the checkpointed VHDX files. When you zoom in on the directory of a VM that is being protected, as soon as VSS kicks in, a checkpoint is made of the active disks. This causes the writes to be redirected from the VHDX to a corresponding AVHDX file which makes it possible for the backup software to take a clean and ‘frozen’ copy of the virtual hard disk. When the backup is ready, the written data to the AVHDX file is merged back into the VHDX file. Only briefly you’ll see n AutoRecovery.avhdx file created which is deleted when it is ready with the merge operation.
Important: Microsoft started to coin the term checkpoint in VMM. After Hyper-V had used the term Snapshot for a long time, this changed with Windows Server 2012 R2. We can now better distinguish between VSS snapshots and Hyper-V checkpoints. Backup software now uses Hyper-V checkpoints just as in Hyper-V Replica.
If you want to read more about the changed method of Hyper-V backup in Windows Server 2012 R2, please take a look at fellow MVP Aidan Finn’s post:
READ MORE »