Posts by Hans Vredevoort

Update Emulex VMQ Issue: They finally did it!

Another update from the VMQ front, this time with the formal announcement of a declaration of peace.

After a very long struggle – which started in November 2013, not long after the release of Windows Server 2012 R2 – I dare now confirm that VMQ can be re-enabled on Windows 2012 R2 Hyper-V hosts with HP/Emulex 554FLB host bus adapters.

image

If you have followed along, you will know that we have published several blogs on Hyper-V.nu discussing the matter. We initially discovered the problem in our own lab environment, then explained how to avoid the network disconnections by disabling VMQ, followed by how we tested the numerous HP/Emulex firmware and driver updates and about our interactions with Microsoft, Emulex and to a lesser extent HP. Many customers who were using the HP/Emulex, but also many other using OEM versions, were badly hit by the networking problems.  The problem with VMQ became well known all over the world, not to say notoriously known.

Several weeks ago I was able to test a combination of new HP/Emulex firmware and drivers (February 2015), combined with a private Microsoft hotfix (The KB for the final hotfix is KB3031598), provided by the Microsoft Networking team in Redmond. We already saw promising results using older versions, but there were a couple of remaining issues which had to be solved in the Windows NIC Teaming (LBFO) stack. One issue that I kept seeing was that when there were more VMs running on a particular host in the cluster than there were VMQs available, things started to collapse. Not only were the additional VMs not getting a VMQ Queue, they would also not get assigned to the default VMQ Queue. Other VMs that were already assigned to a VMQ Queue, were disconnected and it took some time for this problem to settle and make connections possible again.

Using the latest firmware/drivers with the Microsoft LBFO hotfix solves that problem (amongst others).

It is time to say that we have the worst behind us.

READ MORE »

Sold Out First “Theme Night” with eBook Giveaways”

The first “Theme Night” with focus on hybrid cloud organized by Hyper-V.nu and SCUG.nl on March 25th already sold out 2 weeks ago.

Today we can announce that both Pack Publishing and Apress accepted our request for a couple of free ebooks:

 

Hyper-V Network Virtualization Cookbook  by Ryan Boud

image_thumb

Windows PowerShell Desired State Configuration Revealed  by Ravikanth Chaganti

image_thumb[1]

Both books are a perfect match for what this first Theme Night has to offer and we thank both Ravi and Ryan as well as their publishers for making this possible.

Much appreciated guys!

System Center User Group NL & Hyper-V.NU presenting “Theme Night – Hybrid Cloud” – 25 maart 2015

Update 4 maart 2015

Tijdens dit event stelt uitgever PACK Publishing een aantal exemplaren van Ryan Boud’s Hyper-V Network Virtualization Cookbook beschikbaar.

Ook uitgever Apress biedt een aantal exemplaren van Ravikanth Chaganti’s Windows PowerShell Desired State Configuration Revealed

 

imageimage

Na het succes van de jaarlijkse “System Center Summer Night” zijn Hyper-V.nu en de System Center User Group een nieuwe weg ingeslagen. SCUG.NL en Hyper-V.nu slaan wederom de handen in één en organiseren gezamenlijk dit jaar meerdere “Theme Nights”. Tijdens deze bijeenkomsten praten wij jou bij over wat nieuw is binnen Azure, Hyper-V & System Center! De ontwikkelingen in IT-land gaan ontzettend hard, het wordt dan ook steeds moeilijker om alle facetten van Azure, Hyper-V & System Center in één avond te bespreken en daarnaast te blijven voldoen aan ieder zijn verwachting. Dus volgens ons is er maar één oplossing; “Theme Nights”.

image

Wat houden deze “Theme Nights” in?
Wij zorgen ervoor dat jullie in een paar uur helemaal bijgepraat worden door (internationale) topsprekers en MVP’s, binnen een bepaald thema. Ben jij een echte Service Manager liefhebber en interesseert Hyper-V je eigenlijk weinig, geen enkel probleem! Schrijf je in voor de Service Manager “Theme Night” en je zal geen live migration voorbij zien komen.

READ MORE »

Quickly Checking the Azure Pack Version

Updates for Windows Azure Pack are offered on a quarterly basis. These so-called Update Rollups offer bug fixes and functionality enhancements. The latest version is UR4 and the next one is due February 10th, 2015.

When installing Azure Pack via the Web Platform Installer, you are sure to get the latest version. Likewise if you use the downloader.ps1 script used by PowerShell Deployment Toolkit for offline installation of WAP, you also get the latest version.

If WAP was installed some time ago, you can quickly find the version by pressing CTRL+SHIFT+A

image

This data can also be copied to your clipboard in json format:

{"items":[{"key":"UTC Time","value":"2015-01-20 13:20:25Z"},{"key":"Browser","value":"Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; Touch; .NET4.0E; .NET4.0C; .NET CLR 3.5.30729; .NET CLR 2.0.50727; .NET CLR 3.0.30729; rv:11.0) like Gecko"},{"key":"Language","value":"en-us"},{"key":"Portal Version","value":"3.19.8196.21 (rd_auxsmp_stable_v2_gdr.141021-1031)"},{"key":"PageRequestId","value":"fb2cc9e9-83c5-4d20-950a-1933da326ca6"},{"key":"Email Address","value":"DOMAIN\\Administrator ({1})"},{"key":"Subscriptions","value":""}],"errors":""}

 

You can register here fore the Windows Azure Pack Community newsletter which gives you early access to new features in Update Rollups.

Azure Pack Wiki Available Via OneNote

Hi Azure Pack fans!

In the past 2 years or so, we have acquired a lot of experience with Azure Pack. We created a Wiki which was intended to collect the scattered locations of all the blogs, videos, articles and whitepapers. But as we observed, this wiki has also stimulated the growth of content, as how cool is it to be on this now famous Azure Pack Wiki <grin>.

image

Many features you will first see in Microsoft’s public cloud, will gradually become available on premise. I have added several new sections relevant to both public Azure and private Azure.

The Wiki has grown to such extreme length that the TechNet Wiki was no longer suitable and publishing to it became a nightmare.

For this reason I make the Azure Pack Wiki available in OneNote Online.  If you edit the Wiki in your local version of Microsoft OneNote, you can also use the hyperlinks.

image

You can find the latest update of the Azure Pack Wiki here for viewing and here for editing if you want to add content. Please add NEW behind the title of the added content.

Hope you like it!

Best regards,

Hans Vredevoort

Hyper-V MVP

Twitter: @hvredevoort

Marc van Eijk

Windows Azure MVP

Twitter: @_marcvaneijk

http://www.hyper-v.nu

http://www.hyper-v.nu

Review of Hyper-V Network Virtualization Cookbook

Introduction

Two weeks ago on Twitter, Ryan Boud who is  a senior cloud consultant working for Inframon, asked me if I would be interested in reviewing his book on Hyper-V Network Virtualization (HNV). Sometimes things can go very quickly. About one minute after his tweet, I confirmed my interest, not knowing this was in fact a book published by Packt.

As a side note, I’m personally not very fond of this publisher after a bad experience with them in their rather aggressive approach to candidate writers and quickly learnt they don’t pay very well for the vast amount of work authors put into their writings. As a technical editor for several books by Aidan Finn et al., and having contributed to a book called Microsoft Private Cloud Computing, I had gotten used to being paid for my work. Packt does not pay a cent to technical editors or reviewers, so that’s why I had previously declined my participation on another book.

imageimage

Review

But as you can see in the tweets above, I had already accepted the review and quickly received an email from Packt, offering me a log in account to their site and a free download of Ryan Boud’s book in pdf, epub, and mobi formats. So I decided to download the PDF and see what this book had in store. Of course I’m interested in anything written on Hyper-V Network Virtualization as it directly relates to the many CloudOS, System Center and Azure Pack projects I am involved in with different service providers and enterprises in Europe.

Hyper-V Network Virtualization was in fact introduced in Windows Server 2012 but was not very useful without a NVGRE gateway. We had to wait until Windows Server 2012 R2 and VMM 2012 R2 came out with all the necessary ingredients to let this baby live in the real world. Microsoft had decided to make an in-box gateway available, which could be implemented on a dedicated Hyper-V server or preferably an HNV cluster. By deploying a VMM Service Template, a single or high available NVGRE Gateway guest cluster could be deployed and managed through VMM. READ MORE »

Another Update on the VMQ Issue with Emulex and HP

I wrote my last blog on this topic 6 months ago. Meanwhile I have seen several firmware and driver updates from Emulex, usually followed by HP several weeks later. I’m still talking about the ongoing VMQ problems with  HP/Emulex 554FLB CNAs in HP BL460c Gen 8 blade servers in c7000 blade enclosures. Meanwhile I have tested several incarnations of this firmware/driver combination in our own Azure Pack cloud environment. I already found out in previous attempts, that a switch to new Emulex firmware and HP/Emulex drivers, including the switch from VMQ disabled to enabled, can be disruptive and hosts and VMs need to be restarted if things turn bad.

And it did turn bad on all four occasions I tried until now. In July 2014 our hopes were raised by Emulex when Mark Jones from Emulex posted the first update on July 23 2014. I tried them out but not until HP released their OEM specific driver for the CNA. It didn’t take me very long to find out that my test guest cluster quickly got disconnected during a live migration with one node being evicted from the cluster during the black-out. First opportunity: failed

On August 4 a special release was issued for non-OEM Emulex branded cards. I decided not to try this out on our HP/Emulex CNAs. I later found out that some people who tried had to call in HP to replace their servers. Second opportunity: Ignored

Just over a month later, another update appeared on the Emulex blog letting us know that HP versions of the firmware/drivers had been made publicly available. Previous versions had been considered but we found a mystery insider in the comments of our blog stating:

Without revealing NDA information, the HP driver released in September is not really a fix, it is also a workaround, with some caveats that don’t appear in the published notes. My understanding is that the fix delivered by Emulex was considered unstable by HP, and a truly “fixed” driver won’t be released until sometime in Q1 2015.

I didn’t have to think long. Third opportunity: Ignored

image

On October 21st 2014, another update appeared on the Emulex blog. I had become very suspicious of again another update and lacked the time or drive to once more spend many hours testing this version. What changed my mind was when fellow Hyper-V MVP Patrick Lownds sent a message to one of the MVP distribution lists, letting us know that HP had released their OEM version of the HP/Emulex 554FLB firmware and driver. READ MORE »

New blood for Hyper-V.nu

I’m very happy to announce that three very talented young men have agreed to officially start blogging for Hyper-V.nu. This time not as guest bloggers, but as official Hyper-V.nu bloggers.

If you check Hyper-V.nu on a regular basis, you will have noticed that in recent weeks very few blogs have appeared on the site. This is largely due to the enormous success of Windows Azure Pack which is more or less keeping us fully occupied.

Apart from the many CloudOS related projects that Peter Noorderijk, Marc van Eijk and I run on a daily basis, we also maintain the Azure Pack Wiki, some of the Hyper-V hotfix lists, do presentations at IT events, write books, blog for the MS Building Clouds blog, evangelizing hybrid cloud with Azure Pack, and as MVPs have very regular meetings with Microsoft product teams. In other words, there are not enough hours in the day to make this all work.

So that’s enough explanation why we need more bloggers to help fill the pages of Hyper-V.nu. In the previous year, you may have already seen several guest blogs by Darryl van der Peijl, Ben Gelens and Mark Scholman. All three also happen to be colleagues at INOVATIV, but that is not why they join Hyper-V.nu. It is their real world experience with Azure and Azure Pack technology that makes them special and why we let them join as bloggers.

Let me quickly introduce Darryl, Ben and Mark.

Darryl van der Peijl

image

Darryl was working for a service provider where I met him during the deployment of what was then called Windows Azure Services for Windows Server. Darryl is a very clever young man and quickly came up to speed with the Microsoft System Center and Cloud offering. He is also very proficient in PowerShell which is of course a must-have knowledge these days. Darryl has been implementing Azure Pack ever since, often sharing scripts he developed such as the Azure Pack Tool , the Windows Azure Pack Update Script on the TechNet Gallery. After several guest blogs, he just submitted his first blog on Scale-Out File Servers.

Darryl tweets at @DarrylvdPeijl and has his own blog at http://www.darrylvanderpeijl.nl/

Ben Gelens

image

I met Ben virtually via Twitter and was amazed at the quality of this blogs on VMM, storage, bare metal deployment. I praised his blogs a couple of times and got to know more of what Ben was doing. He happens to be also very versed at PowerShell and PowerShell Workflow, which is as you might know the center of focus in Service Management Automation (SMA), which was first exposed via the Windows Azure Pack admin portal. We then talked about several guest blogs about Bare Metal Post-Deployment using SMA and VM Role.

Ben tweets at https://twitter.com/bgelens and blogs at http://mssecbyben.wordpress.com/

Mark Scholman

image

Mark also quickly made fame while promoting his blogs via Twitter on networking, Azure Pack, NVGRE and Network Virtualization. These are all qualities which are highly desirable if you start implementing Windows Azure Pack in the real world. Mark recently starting investigating Azure Pack Websites for one of the projects we currently engage in. Learning and writing always ends up in a great blog for Mark and the Installing and Configuring HA Azure Pack Websites series is just one example.

Mark tweets at https://twitter.com/markscholman and blogs at http://sysctr.nl/

Let me finish by saying that these three guys are worth following and hopefully they’ll share many blogs on Hyper-V.nu.

Hans Vredevoort
@hvredevoort

Additional Background on the VMQ Issue with Emulex and HP

Today I had a conference call with the people from Emulex responsible for the network adapters, firmware and drivers. As many of you know we’ve had a long lasting issue with HP/Emulex 554FLB CNAs in HP BL460c Gen 8 blade servers in c7000 blade enclosures. After we had replaced Windows Server 2012 with Hyper-V by Windows Server 2012 R2 with Hyper-V on the same hardware, we started to notice virtual machines losing connection. We have multiple guest clusters on top of the Hyper-V clusters and sometimes during Live Migration of one of the cluster nodes, we would see that connectivity was lost, even to the point that a cluster node would be forced to leave the cluster and come back later when the network connection was re-established. In fact that was our single best test to reproduce the problem.

Marc van Eijk en Peter Noorderijk wrote blogs about it and together they got over 200 comments from customers all over the world seeing the same problem, the majority of them having Emulex through an OEM like HP, Dell, IBM and Cisco. In fact we also read similar cases with HP rack servers, other NICs and different hardware combinations which led me to believe there could also be an issue in the networking/teaming stack in Windows Server 2012 R2.

During the months of November and December 2013 we collected a list of 10 registered support cases with either HP, Microsoft or both. At that time my primary contact was a senior escalation engineer at Microsoft who was able to collect several customer cases mostly from European customers. Unfortunately we could not register a support case with HP ourselves because we did not have a support contract with them. But the amount of customers that discovered they had the same problem, grew and grew. There must have been hundreds of open support calls with both HP and Microsoft and the storm of discontent was growing.

According to HP they were dependent on Emulex and HP did not get any feedback either. Also Microsoft was left in the dark for a long time.

READ MORE »

System Center, Hyper-V, Azure and Meat

If you like System Center, Hyper-V, Azure as well as meat, don’t forget to register for System Center Summer Night “The MasterChef edition” which is rapidly approaching. In 10 days from today, nine experts, seven of them are MVPs, will present five interesting presentations.

Because we have plenty of space left we invite those who have registered to bring a friend without additional cost. If you haven’t registered yet, please do and have a great afternoon which is completed with a nice barbecue.

Date of event

June 26th

Program

15:00 – 16:15 How many System Center fits on one grill by Ronny de Jong / James van den Berg [MVP] / Helmer Zandbergen / Marc van Eijk [MVP] / Dieter Wijckmans [MVP]

16:15 – 17:30 How Service Manager can do everything you need – a best-of-the-best Swiss cheese selection by Marcel Zehner [MVP]

17:30 – 17:45 Break

17:45 – 19:00 Light up the fire on your Hyper-V by Hans Vredevoort [MVP] & Peter Noorderijk on Hyper-V Architecture

19:00 – 20:15 Become a Masterchef on Microsoft Azure Automation by Maarten Goet [MVP]

20.15 BBQ time

This event is organized by SCUG.nl and Hyper-V.nu

Registration

Please register at https://www.eventbrite.nl/e/tickets-system-center-summer-night-2014-9265847399

Location

Unieplaza
Multatulilaan
4103 NM Culemborg