Posts in category Hans Vredevoort

Windows Azure Pack–Mystery of the Missing Memory

CRIME SCENE

One of the tenants in our INOVATIV Cloud notified us that he saw an inconsistency in the memory usage measured by Windows Azure Pack compared to the actual memory used by the VMs. Tenant Tom claimed he had only used 13824MB but Windows Azure Pack reported he had used 25344MB. Apparently, the memory was melting before his eyes. Who or what had taken the missing memory?

“Think Watson!”

This is how the concerned tenant sees his resource allocation in the Windows Azure Pack Service Management Portal. As you can observe, 83% of memory was consumed, almost double the amount of what the tenant had actually consumed by the virtual machines that had been so easily deployed to the privately hosted Windows Azure Pack cloud. “Easy come easy gone,” Tom thought.

clip_image001_thumb4

EVIDENCE COLLECTION

A Service Plan in WAP can be configured by the Windows Azure Pack administrator with usage limits for the following resources:

  • # of Virtual Machines
  • # of Cores
  • RAM in MB
  • Storage in GB

If you have the Hyper-V Network Virtualization Gateway in place, you can also limit the following virtual network resources:

  • # of Virtual Networks
  • # of Site-to-Site VPNs per network

Within Virtual Networks, network usage can be capped for network read/sec and network write/sec, either to keep your most enthusiastic buddies under control or alternatively sell additional network capacity to more demanding tenants.

FURTHER INVESTIGATION

On further examination, Watson decided to look at several of the other subscriptions to see if there were any similar cases. Why did we never get any complaints before? In fact only recently our team of consultants devoured more than half the capacity of a 2-node Hyper-V 2012 R2 cluster. We started getting the dreaded overcommitted message in Virtual Machine Manager which by default keeps a cluster reserve of 1. We could have made some concessions to the individual reservation parameters, but that would be too easy.

To check the cluster reserve, right-click the cluster and select Properties.

image_thumb1

READ MORE »

Hyper-V.nu moved to Windows Azure

Hyper-V.nu has been around since 2008. A lot of content has been added to the blog since then. Many hours of research and writing form the basis for that. Our site has been running in a hosted VM, managed by Jaap Wesselius since 2008.

We know some Hyper-V, System Center, CloudOS. But this WordPress….. that is a different story. MySQL database, PHP and all other scary stuff. With Jaap leaving Hyper-V.nu we were doomed to manage and troubleshoot the site our own.

We needed a new strategy. We needed a solid foundation with as little WordPress to manage as possible and with an infrastructure we understand. The answer was easy. We needed Windows Azure.

After numerous evenings of testing and a lot of troubleshooting to cleanup some legacy issues in the (now old) environment. We are proud to announce that we succesfully migrated Hyper-V.nu to Windows Azure.

Combined with the migration we changed the layout of the site. We hope you enjoy the new look and feel of hyper-v.nu R2.

Experts Live 2013 Presentations Available for Download

The presentations of Experts Live can be found on this SkyDrive link: https://skydrive.live.com/?cid=43040b57da517977&id=43040B57DA517977!1281&ithint=folder,.pptx&authkey=!AEIwmJbuaDwgbFE

Two members of Hyper-V.nu, Peter Noorderijk and Marc van Eijk contributed their presentations for Experts Live. In case you missed it, take a look at the video for an impression of this superb event.

image

It is even possible to pre-register for the Experts Live 2014 edition: http://www.expertslive.nl/index.php/aanmelden.

November 2013 UR KB2887595 causing network problems

We recently published a number of blogs (blog1, blog2) about network connectivity problems with our HP ProLiant Gen8 blade servers with Emulex adapters. Via the comment section on those blogs we saw we were not alone and several others were suffering from problems or even BSODs using other adapters than Emulex. Also Broadcom and Chelsio were reported as having issues. We can now also add Intel network adapters to the list (see afterthought at the end of this blog).

I just saw an interesting pointer by Didier van Hoye to a blog by Michael Rueefli aka Dr. MIRU from Switzerland, who suffered from BSODs after installing KB2887595 (November 2013 Update Rollup) on his Windows Server 2012 R2 Hyper-V clusters using Intel 82599 dual-port 10Gbps network adapters.  Michael traced the BSOD back to the Mslbfoprovider which is the core driver for the inbox NIC teaming in Windows Server 2012 R2.

Previously I referred to as KB2887595 as as an update containing a stealth update (replacing Windows binaries without documentation). Evidently, this update has changed at least part of the lbfo network adapter teaming subsystem. Instead of fixing some of the network connectivity problems we had already encountered, it seems that in some cases, BSODs may occur on Windows Server 2012 R2 hosts with teamed Intel network adapters.

Like in our case with Emulex network adapters, the temporary workaround was to remove KB2887595 and reinstall the v2 version of KB2887595, then disable VMQ hardware offload. See Michael’s blog for instructions.

The Intel 82599 controller is also used by HP and is known as HP 560FLB Flexible LOM for HP Gen8 blade servers.

image image

Afterthought: if so many different network adapters suffer from serious networking problems, it can be coincidence that all network vendors overlooked something in the testing process. On the other hand we could well face a bug in the NIC teaming subsystem of Windows Server 2012 R2. We will probably not find out before the end of the year.

Despite the trouble you may face, we at Hyper-V.nu wish you a quiet Christmas and a stable 2014!

image

Update List for Windows Server 2012 R2 Hyper-V

Taylor Browne has very recently created a new Wiki for the latest Hyper-V service packs, updates, and hotfixes for Windows Server 2012 R2 Hyper-V: http://social.technet.microsoft.com/wiki/contents/articles/20885.hyper-v-update-list-for-windows-server-2012-r2.aspx

Currently only one KB article KB2883200 is on the list but it will not take long before this list will grow. KB2773200 was the General Availability Rollup which was delivered at Windows Server 2012 R2 GA release in October 2013.  This KB article also recommends to install KB2884846 which is the Windows Server 2012 R2 update rollup for October 2013 and includes “performance and reliability improvements”.

Many of the monthly update rollups for Windows Server 2012 R2 which come via Windows Update hide the exact details of what exactly has been changed or solved. It is very difficult to determine if it is specific to Hyper-V, clustering, networking, storage or all of them.

The November 2013 update rollup KB2887595 contains a fix for Hyper-V guest OS does not shut down when you restart the host computer that is running Windows Server 2012 R2 or Windows 8.1 which I encountered quite a lot in our environment. What seriously worries me however is that this same November 2013 update rollup also fixes things that are not mentioned in the list.  For instance during a support case with Microsoft which dealt with VMs losing network connections on Windows Server 2012 R2 Hyper-V hosts, we learnt that a particular update part of the November rollup was also supposed to fix a problem with LBFO teaming. It turned out that it didn’t fix the problem for everyone as we still have to disable VMQ to keep our VM networking operational.

The December 2013 update rollup KB2903939 is more explicit on what updates have been made but I have a hard time figuring out what critical updates are made for enterprise servers. I have to filter through things like issues with client computers, Lumia 2520 camera app crashes, errors in Windows Live Messenger, mouse pointer stutters, input devices that fail and something that never enters a DRIPS, whatever that may be!

image

Here is another update that is not yet on the list and applies to Windows Server 2012 R2 and Hyper-V: Known issues after you enable data deduplication on CSV.

My advice to Microsoft:

  • be much more specific about what has been fixed
  • avoid stealth fixes (as we call them among Hyper-V MVP’s) which solve problems but are not described at all
  • separate the server updates from client and mobile.
  • be open and honest if updates turn out to be bad or broken (as we have seen several times in the past)
  • formalize the wiki by offering a hotfix/update list authorized by the product teams like the cluster team does with: http://support.microsoft.com/default.aspx?id=2784261: Recommended hotfixes and updates for Windows Server 2012-based Failover clusters.

This last support page is a very good initiative but it can take a while before updates are posted. A vital hotfix KB2908415 was announced only yesterday: Clustered shared volumes go offline or the Cluster service stops during VM backup on a Windows Server 2012 Hyper-V host server but is not yet on either the Cluster Update Wiki or the Official list for Windows Server 2012.

In other words: work to be done!

Windows Azure Pack Is Clearly The Place To Be

You may or may not have noticed it, but the daily stream of updates around Windows Azure Pack is becoming more and more amazing. The WAP Wiki initiative I took with Marc van Eijk several weeks ago, makes us very aware of all the blogs and articles that are published around Windows Azure Pack, Service Provider Framework, VM Role, Service Management Automation, Usage and Billing and to a lesser extent Database as a Service, Service Bus and Web Sites.

It is more than obvious that a huge amount of Microsoft resources have been dedicated to making the Windows Azure Pack a success. The Building Clouds section on the Microsoft Server & Tools blog which was kicked off by Brad Anderson himself is very active indeed.

Sometimes I see several posts a day coming from the different team members. Guys like Anders Ravnholt, Tiander Turpijn, Jim Britt, Charles Joy, Mark Stanfill, Shiram Natarajan, Thomas Roettinger, Michael Greene, Eamon O Reilly, Justin Incarnato and many more write really interesting blogs that light up the for many still undiscovered possibilities with the Windows Azure Pack. In the community we see good blogs from Marc van Eijk, Walter Eikenboom, Daniel Neumann, Nathan Lasnoski, Michael Rueefli, Joe Thompson, Damian Flynn, Stanislav Zhelyazkov, Lai Yoong Seng and others.

image

Of course this is only the beginning and we have tons of feature requests for additional functionality. But then, there is so much to learn in this corner of the cloud that many of us can hardly keep up. As one of the initiators of the Hyper-V.nu blog, I’m fully aware that we can build on a superb platform using Windows Server 2012 R2, Hyper-V, System Center 2012 and now with the Windows Azure Pack, Microsoft has just begun to realize its full potential.

If you are looking for an organized index to Windows Azure Pack related subjects, just navigate to the WAP Wiki! There is also @WAzureP tweeting on Windows Azure Pack and related stuff. You can follow @hvredevoort and @_marcvaneijk for regular updates on these topics.

Please help us promote this with a tweet, blog or face… (no nothing on FB please)
We use the #WAPack hashtag and suggest you do too!

image

Participate in Geekmas Competition and win Microsoft Private Cloud Computing book

Today is the 7th day of a Christmas-themed competition that is running through the TechNet Blog. It’s based around the ‘Twelve Days of Christmas’ carol, and @TechNetUK is giving away a number of prizes on a daily basis between now and 20th Dec. Today our Microsoft Private Cloud Computing book features as one of the prizes,

Head over to the TechNet UK page for further details on how to win of of the prizes.

Thanks to Steven Mullaghan for organizing this competition and featuring the book that was co-authored by Aidan Finn, Damian Flynn, Patrick Lownds and myself.

Note: only entries from the UK will be eligible

image

WHAT CAN YOU WIN?

image

Of course if you are not one of the lucky winners, you can still order Microsoft Private Cloud computing here:

Assigning Owner and User Role to VMs in Windows Azure Pack

At Inovativ, the company I work for, we more or less use Windows Azure Pack in production now. We have begun to operate as a hoster and each consultant is a tenant in our Inovativ Cloud. They can go to the Windows Azure Pack tenant portal and consume the resources we have available in our cloud. If you don’t set any rules your tenants are likely to grab all the resources that are available. Currently we have not yet set any quotas, but I’m afraid we will have to very soon.

One problem that we encountered was that VMs which were created outside of the portal did not have the correct Owner and User Role. This would mean that those VMs were not visible as owned by a certain tenant in the Windows Azure Pack tenant portal. Also when you set quota on for instance the number of VMs that could be created or the amount of memory that could be consumed, only the VMs were counted that were actually owned by the tenant.

image

This would be a serious problem if you are a hoster an wanted to move existing VMs to a tenant’s subscription or service plan as it is called in Windows Azure Pack. Another problem that we think is linked to this is that Cloud Cruiser which collects usage information for billing or showback is not able to process the usage records.

This is what it looks like from the admin portal when you go to VM Clouds and find all VMs for a certain tenant and subscription to a tenant plan. As you can see there not all of my VMs are listed.

image

It was quite easy to give access to the manually added VMs via the VM properties in Virtual Machine Manager 2012 R2. The first requirement is assign the VM to the same VMM Cloud that is used in Windows Azure Pack. If the VM does not have an Self-Service owner, you can select one. The problem however is that the only accounts you can choose are Active Directory accounts. This is exactly what we don’t want.

READ MORE »

Disk Performance counters in Windows Server 2012 R2 Task Manager

Have you ever wondered why you DO see performance data in Windows 8/8.1 under the performance tab in Task Manager, but DON’T see this same information in Windows Server 2012/2012 R2?

Well I kind of missed seeing that information in Server but never really bothered to really investigate.

Windows 8/8.1
Windows Server 2012/2012 R2
image image

I know the solution is too simple for words, but just in case you also forgot about this good old Diskperf command which you can run from an administrative command or Powershell prompt:

Diskperf –Y to switch on the Disk Performance counters. You don’t need an academic degree to figure out how to disable them again.

image

image

So much for an easy tip with great results.

Hotfix: VM Virtual Fibre Channel Loses Access to LUN after Live Migration

A hotfix has been released today for Windows Server 2012 Hyper-V servers which are unable to access LUNs over a Synthetic Fibre Channel after a VM is live migrated to another host in the cluster.

This problem can occur if the following conditions are met:

  • You have two Windows Server 2012-based computers that have the Hyper-V role installed
  • You install a virtual machine on one of the Windows Server 2012 Hyper-V hosts
  • You set up a guest failover cluster, and then you make the virtual machine a cluster node
  • The virtual machine is configured to access LUNs over a Synthetic Fibre Channel
  • You try to perform live migration to move the virtual machine to another host.

This issue is caused by the inability of the Hyper-V host to restore the Synthetic Fibre Channel LUN on behalf of the virtual machine during live migration.

The problem applies to Windows Server 2012 Standard and Datacenter.

For further details look up the Microsoft Support Article: http://support.microsoft.com/kb/2894032

 

clip_image001

Our Sponsors





Powered by