VMM Update Rollup 2 affects wildcards in PowerShell

Here is a short guest post by one of our sponsor’s cloud admins at Service Provider Amsio, Ivo Hoefakker

VMM 2012 R2 wildcards no longer seem to be working in (some) PowerShell cmdlets after upgrading to Update Rollup 2. Today I’ve updated my environment with Update Rollup 2. After the upgrade some of my scripts were no longer working.

I came to the conclusion that some PowerShell cmdlets no longer accept the wildcard (*) character.

For example:

Before the update all of the below cmdlets work:
Get-SCVirtualMachine | ? name  -eq ‘amsio-vm01′
Get-SCVirtualMachine | ? name -ge ‘amsio*’
Get-SCVirtualMachine | ? name -like ‘*vm*’
Get-SCVirtualMachine | ? owner -eq ‘ivo@amsio.com’
Get-SCVirtualMachine amsio-*

After installing Update Rollup 2 the following cmdlet no longer works:
Get-SCVirtualMachine amsio-*

I have confirmed this (issue) on the following PowerShell commands:

Get-SCUserRole
Get-SCVirtualMachine
Get-SCVMHost
Get-SCVMHostGroup

Whether or not this is really a “issue” is debatable. You could also say you should improve your scripting.

Reported problems after installing Update Rollup 2 for DPM 2012 R2

Today I saw several users having problems with their Data Protection Manager 2012 R2 Server after installing KB2958100 which is the specific update for DPM in Update Rollup 2.

My advice is to postpone the DPM update and allow Microsoft to look into the user reports. Correction is quite complex and requires modifications in the DPM database.

See the thread for more details:

http://social.technet.microsoft.com/Forums/en-US/e9f4801e-6a6f-440f-ad05-65758007db69/dpm-2012-r2-ru2-kb2958100-dpm-accessmanager-service-crash?forum=dataprotectionmanager

Update 2 for Windows Azure Pack (April 2014)

Along with Update Rollup 2 for System Center 2012 R2 (KB2932881), also Update 2 (KB2932946) for Windows Azure Pack was released. Update 2 fixes 10 known issues, amongst others a fix for the problem created with the Usage Collector in Update 1. Because WAP does not currently support rolling upgrades, it is necessary to schedule downtime for the WAP machines. Please be careful running this update in your production environment and expose them to a test environment first. There are so many components which depend on each other so take any precautions in the form of checkpoints (formerly called Hyper-V snapshots) and make sure you have recent full backups of your environment.

Although it is stated nowhere, you would be advised to first start with the System Center 2012 R2 Update Rollup 2 and run the documented SQL Scripts for VMM and SCOM first (important!) before you approve the Windows Azure Pack Update 2 installation. The SQL Script for VMM can be found in http://support.microsoft.com/kb/2932926. A reference to the path of the SQL Scripts for the SCOM OperationsManager and OperationsManagerDW databases can be found in http://support.microsoft.com/kb/2929891.

Windows Azure Pack Update 2 solves the following issues:

Issue 1

The PowerShell cmdlets “Get-MgmtSvcSubscription”, “Get-MgmtSvcPlan”, “Get-MgmtSvcAddOn” cannot retrieve a Subscription, a Plan, or AddOn through a specific Subscription, Plan, or AddOn ID respectively.

Symptom: When using these PowerShell cmdlets to retrieve the corresponding objects, the objects are retrieved by name rather than by Id.

Resolution: Now the parameter with corresponding Id can be used to retrieve the corresponding object.

Issue 2

Usage Collector Service loses its current web.config settings after applying Windows Azure Pack Update 1 if instructions in the corresponding KB Article are not followed.

Symptom: When you deploy Update 1 over an existing deployment with the Usage Collector Service configured and fully functional, you will notice that the Usage Service stops working.

Resolution: This issues does not occur with Update 2, the Usage Service configuration is preserved and no manual steps are required. Notice that if you already applied Update 1 and did not fix this issue with the manual steps provided in the corresponding KB article, Update 2 will not fix it automatically; you still need to solve it manually.

Issue 3

Plan Creation Events exposed by the Usage API have a NULL value for the display name.

Symptom: Any Service receiving notifications or querying usage data will find the Plan Display Name to be NULL.

Resolution: With Update 2 the Display name value is passed instead of NULL. Notice that any already stored data prior to Update 2 will still show NULL for the Display Name.

Issue 4

The Web Application Gallery Service generates a large amount of temp files in its Temp folder.

Symptom: Every time the WebAppGallery service restarts it creates a new temp folder– the temp folder stores the cached feeds containing the Metadata of the available gallery applications and their dependencies and the actual application packages. The service is scheduled to restart every 1740 minutes which explains the growing number of temp files over time.

Resolution: The old WebAppGallery temp files are removed.

Issue 5

Resizing dynamic memory VMs through machine profiles results in an error when the new Startup Memory value wasn’t in the range between original Minimum memory and Maximum memory. Also changing between static to dynamic on vice versa in the machine profile results in error.

Symptom: When a tenant user sets the Start value of the dynamic memory VM to a value higher than the original maximum value or less then original minimum value an error is generated and the operation fails. Changing from Static to Dynamic did not change the kind of memory, only new memory value was set as Startup value in the profile.

Resolution: This is now possible with Update 2 and no errors are generated.

Issue 6

Active Directory was not enabled for Windows Azure Pack WebSites Resource Provider.

Symptom: Administrators had no way of configuring WebSites Clouds to use Active Directory. Tenants have no way of enabling Active Directory Account Management and Authentication for their Web Sites.

Resolution: In Update 2 this is available for Administrators to configure and for Tenants to use. The Admin has the option of always requiring Windows Authentication, in which case, tenants no longer have an option.

Issue 7

It is difficult to discover how to create Co-Administrators for an existing subscription.

Symptom: Tenant users tend to go to “My Account” and “Subscription” drawer and pane, but do not see an option there to create Co-Administrator.

Resolution: In this Update we have added a link to the “My Account” drawer set of options. This link will be enabled once the user has a valid subscription.

Issue 8

Error messages in Admin Site that refer to un-configured services in a Plan are not explicit as to what are the problems.

Symptom: when trying to update a Plan, for example when making it public, an error is generated if any of the Resource Providers it contains are not configured, this message does not mention what is the problem or which Resource Providers are affected.

Resolution: In Update 2 the messages are more informative and the user can take corresponding actions.

Issue 9

Obscured error is returned when a Tenant creates a database using a security token whose corresponding password does not meet the SQL Server domain policies.

Symptom: The user receives an unrelated SQL error regarding a select statement containing an invalid column (or an Internal Server Error message, depending on whether the created database belong to an Always On availability group).

Resolution: With Update 2 the tenant now sees the correct error and can take action to resolve it by changing the credential password.

Issue 10

Windows Azure Pack Management Databases were not created as partially contained when SQL Always-On was desired.

Symptom: When using SQL Always-On the user was forced to manually replicate the associated database logins.

Resolution: With Update 2 the administrator does not need to manually replicate the database logins.

 

Installation Instructions

These installation instructions are for the following Windows Azure Pack components:

  • Tenant site
  • Tenant API
  • Tenant Public API
  • Administration site
  • Administration API
  • Authentication
  • Windows Authentication
  • Usage
  • Monitoring
  • Microsoft SQL
  • MySQL
  • Web Application Gallery
  • Configuration site
  • Best Practices Analyzer
  • PowerShell API

To install the update .msi files for each Windows Azure Pack (WAP) component, follow these steps:

  1. If the system is currently operational (handling customer traffic), schedule downtime for the WAP machines. WAP does not currently support rolling upgrades.
  2. Stop or redirect customer traffic to sites you consider adequate.
  3. Create backups of the computers.

    Notes:

    1. This update does not include database changes, backing up and restoring databases is not necessary. In general, it is recommended to keep current database backups.
    2. If you are using Virtual Machines, take snapshots of their current state. Otherwise, go to the next step.
    3. If you are not using VM’s, take a backup of each MgmtSvc-* folder inside inetpub directory on each machine having a WAP component installed.
    4. Also collect information and files related to your certificates, host headers, or any port changes.
  4. Important: Make sure to create a backup of the Windows Azure Pack Usage Extension Site and Application Pool settings before doing the update. You can do so by following this step:
    1. In the computer containing the Usage Collector create a backup of the Usage Collector Site and Application Pool settings by running the following 2 appcmd commands:

       

    %systemroot%\system32\inetsrv\appcmd.exe set site MgmtSvc-UsageCollector /name:MgmtSvc-UsageCollector.backup

     

    %systemroot%\system32\inetsrv\appcmd.exe set config /section:applicationPools /[name='MgmtSvc-UsageCollector'].name:MgmtSvc-UsageCollector.backup

     

    1. The Update .msi process will use these backups to restore the settings for this site. This manual step is required because the corresponding site .msi for the RTM and Update 1 releases did not include this step. With this Update (Update 2) the msi does include it, so that the next update will not require this manual step.
  5. Perform Update using Microsoft Update either directly on each node or through the Windows Server Update Services (WSUS) Server.
  6. For each node under Load Balancing, run the updates for components in the following order:
    1. Update Resource Providers (RP’s) services (SQL Server, My SQL, SPF/VMM, Web Sites) as needed, make sure the RP sites are running.
    2. Update the Tenant API Site, Public Tenant API, and Admin API nodes.
    3. Update the Admin and Tenant Sites.
  7. If everything is updated and functioning as expected, you can proceed to open the traffic to your updated nodes. Otherwise see Rollback steps.

Update Rollback Instructions

 

  1. In case an issue occurs and you determine that a rollback is necessary, follow these steps:
    1. If a snapshot is available from step 3.a then apply snapshots, if there are no snapshots continue with the next steps
    2. Uninstall the update from Microsoft Update or Uninstall programs
    3. Reinstall RTM bits from Web PI and DO NOT run the Config wizard. And reinstall the Update 1 bits from Windows Update if you have them to start with.
    4. Replace following items from backup folders to RTM bits
      1. Content folder
      2. Manifest folder
      3. Web.Config file
      4. Apply any changes associated to your certificates, host headers, and/or port changes
  2. Do not leave the system in partially updated state. Perform rollback operations on all machines having WAP installed even if update failed on one node.
  3. Recommended: Run the Windows Azure Pack Best Practice Analyzer on each WAP node to ensure configuration items are correct
  4. You can now proceed to open the traffic to your updated nodes.

     

File List

File that changed Size (KB) Version
MgmtSvc-AdminAPI.msi 4164 3.14.8196.32
MgmtSvc-AdminSite.msi 16428 3.14.8196.32
MgmtSvc-AuthSite.msi 12184 3.14.8196.32
MgmtSvc-ConfigSite.msi 7492 3.14.8196.32
MgmtSvc-Monitoring.msi 4356 3.14.8196.32
MgmtSvc-MySQL.msi 3644 3.14.8196.32
MgmtSvc-PowerShellAPI.msi 3724 3.14.8196.32
MgmtSvc-SQLServer.msi 3680 3.14.8196.32
MgmtSvc-TenantAPI.msi 4148 3.14.8196.32
MgmtSvc-TenantPublicAPI.msi 4196 3.14.8196.32
MgmtSvc-TenantSite.msi 17756 3.14.8196.32
MgmtSvc-Usage.msi 4304 3.14.8196.32
MgmtSvc-WebAppGallery.msi 3904 3.14.8196.32
MgmtSvc-Bpa.msi 888 3.14.8196.32
MgmtSvc-WindowsAuthSite.msi 4136 3.14.8196.32

 

Veeam Rapidly Fixes Host Level VSS Backup for Hyper-V after KB 2919355 (WS2012 R2 Update)

Update (16 April 2014): see bottom of blog

I recently blogged about Windows Server 2012 R2 Update. As usual any update and certainly an update as large as this, has some risks. Therefore we usually advise to postpone Windows Updates, Update Rollups an Hotfixes and leave a couple of weeks before deploying updates in production. Always test in a lab if you can and if you can’t, keep an eye on the forums and the blogs from MVP’s specializing in the related technology.

Testing Host Level Hyper-V Backup

This weekend I came across a tweet from Richard Skinner who reported an issue related to Veeam Backup & Replication and Hyper-V Backup after applying the Windows Server 2012 R2 (Spring) Update. Meanwhile, Veeam had already confirmed the problem and was working frantically into the weekend to fix this nasty problem.

I decided to report a support case with Veeam as well, even though I’m only running it in a Windows Azure Pack lab. I found that the problem was easily reproducible, but only if VSS was enabled in the backup job.

The Problem: Using Hyper-V Checkpoints

When this selection is made as an alternative to Veeam’s default changed block tracking (CBT), the backup fails because it cannot deal with the file path of the checkpointed VHDX files. When you zoom in on the directory of a VM that is being protected, as soon as VSS kicks in, a checkpoint is made of the active disks. This causes the writes to be redirected from the VHDX to a corresponding AVHDX file which makes it possible for the backup software to take a clean and ‘frozen’ copy of the virtual hard disk. When the backup is ready, the written data to the AVHDX file is merged back into the VHDX file. Only briefly you’ll see n AutoRecovery.avhdx file created which is deleted when it is ready with the merge operation.

Important: Microsoft started to coin the term checkpoint in VMM. After Hyper-V had used the term Snapshot for a long time, this changed with Windows Server 2012 R2. We can now better distinguish between VSS snapshots and Hyper-V checkpoints. Backup software now uses Hyper-V checkpoints just as in Hyper-V Replica.

If you want to read more about the changed method of Hyper-V backup in Windows Server 2012 R2, please take a look at fellow MVP Aidan Finn’s post:
http://www.aidanfinn.com/?p=15759

READ MORE »

Boot from VHDX for Windows 8.1 Update

You may have seen my previous blogs on how to very quickly make your computer multi-boot into another operating system.

Now that Windows 8.1 Update and Windows Server 2012 R2 Update has been released, I’ll briefly repeat the steps.

  1. Download Convert-WindowsImage.ps1 and copy it to a temporary directory
  2. Start Windows PowerShell ISE in Administrator mode
  3. Run it with .\Convert-WindowsImage.ps1 -ShowUI
  4. Choose the required ISO file
  5. Choose the SKU
  6. Choose the VHD/VHDX Format, Type and Size
  7. Modify the Working Directory if necessary
  8. Type a name for the VHDX
  9. Optionally add an existing Unattend.xml file for further customization
  10. Hit Make the VHD!

You will see

My Windows 8.1 Enterprise VHDX was only 7.6GB small which came as a pleasant surprise.

The VHDX should still be mounted under a drive letter. If not, right-click the VHDX and mount it.
In my case I had ejected the mounted disks and manually re-mounted under drive F:

The following steps are needed to make your computer boot from the VHDX file:

  1. Open an administrative command prompt via WIN+X Command Prompt (Admin)
  2. Type bcdboot F:\Windows
  3. Type bcdedit /v to see the result in the Windows Boot Loader section

Taking the identifier you can change the description in your bootlist by typing:

bcdedit /set {545a3023-1918-11e2-bed1-bd8926e5c774} description “Windows 8.1 Enterprise with Update”

If you had configured Hyper-V on your Windows 8.1 computer, don’t forget to enable the hypervisor launchtype:

bcdedit /set hypervisorlaunchtype auto

Memory Leak on HP ProLiant Servers with NIC Teaming

HP recently published a customer advisory explaining that HP ProLiant servers running Microsoft Windows 2012 as well as Windows Server 2012 R2 and using Microsoft Windows NIC Agent 9.40 may report a memory leak up to 5Mb/hour.

The memory leak is caused by HP’s Microsoft Windows NIC Agent 9.40.

The problem can be easily observed in Task Manager under Processing as you can see in below diagram.

Windows Server 2012 R2 Update is Important

Today we received a note from Microsoft about the importance of Windows Server 2012 R2 Update which is coming to Windows Update on April 8th, 2014. For MSDN and TechNet subscribers this update is already available in the form of an ISO of Windows Server 2012 R2 with the Update included and a smaller collection of 6 updates for Windows 8.1 and Windows Server 2012 R2

In a blog post published on April 2nd, the Microsoft Windows Server Team explains the importance of this update as it provides an easy way to get up to date with the patches, bug-fixes, and improvements that Microsoft has provided since the release of Windows Server 2012 R2. For failover clustering, this update is certainly important as it contains many of the “hoster patches” that Microsoft has worked on in the past six months. The Update consists of six files:

Microsoft recommends to install the Update in this order:

  1. KB2919442
  2. KB2919355
  3. KB2932046
  4. KB2937592
  5. KB2938439
  6. KB2949621

I tried the update on my Windows 8.1 tablet. Because of the large size of KB2919355 (707MB) this update can take a while, so a little patience is required. Of course for multiple clustered Hyper-V hosts, the manual update method is not to be advised. If you have not yet tried Cluster Aware Updating (CAU), an automated cluster aware update tool included with Failover Clustering and available since Windows Server 2012, I strongly advise you to try this out. It not only allows you to install Windows Updates, but also hotfixes, drivers and firmware for your server hardware. Take a look at fellow MVP Didier van Hoye’s blog and video Hands on with Hyper-V Clustering Maintenance Mode & Cluster Aware Updating TechNet Screencast. Another good explanation can be found in chapter 8 of Windows Server 2012 Hyper-V Installation and Configuration Guide by Aidan Finn, Patrick Lownds, Michel Luescher and Damian Flynn.

 

 

Makeover Hyper-V Update List for Windows Server 2012 and R2

There are several update lists available on TechNet. Some are curated by the product team and some are kept up-to-date by MVP’s and other people in the community. For easy reference we decided to place a shortcut to these lists in the header of Hyper-V.nu.

The following hotfix and update lists are available:

  • Hyper-V: Update List for Windows Server 2012
  • Recommended hotfixes and updates for Windows Server 2012-based failover clusters (updated by Cluster product team)
  • Hyper-V: Update List for Windows Server 2012 R2
  • Recommended hotfixes and updates for Windows Server 2012 R2-based failover clusters clusters (updated by Cluster product team)

Because notably the Windows Server 2012 list had become a bit of a mess, I have rearranged the list, removed outdated or replaced hotfixes and added a sorted date column.

I have also updated the XML file for both Windows Server 2012 and R2 so that you can use a PowerShell cmdlet to quickly scan your Hyper-V hosts if a hotfix or update is installed or not. These files can be downloaded from my OneDrive.

Run the update checker like this:
.\HyperV2012R2UpdatesCheck.ps1 [name host1]

 

 

Recommended hotfixes and updates for Windows Server 2012-based failover clusters

VConnect – A Windows Azure Pack Extension for VMware Hosts

While searching for new content for the Windows Azure Pack Wiki, I stumbled on a blog by RaviCK called Ravi’s Cloud 360o which pointed me to a video on how to integrate VMware hosts with Windows Azure Pack. In a recent project we integrated VMware hosts with Windows Azure Pack by means of Virtual Machine Manager and adding vCenter Servers and indirectly adding VMware hosts to a Microsoft Cloud. This approach has a few disadvantages because only standalone Virtual Machines can be deployed and Console Connect does not work for VMs deployed to VMware hosts. All the wealth of VMRole Gallery Items are lost in this solution.

So I was surprised to find that someone has actually written a custom extension for Windows Azure Pack called VConnect from Cloud Assert which brings VMware hypervisors to the platform. Administrators of Windows Azure Pack can now setup plans that provides Virtual Machine services based on VMware hosts.

VConnect is still in beta and only supports a few basic operations such as:

  • Adding a VSphere endpoint of a VMWare hypervisor server
  • Lists the Virtual Machines from all the added servers
  • Basic operations such as Power On, Power Off, Suspend and Reset VM
  • Connect to the VM via Remote Desktop (VMWare tools has to be installed on the VM)
  • Take a screenshot of the Virtual Machine screen
  • ShutDown, Standby and Reboot of Guest OS (VMWare tools has to be installed on the VM)

Take a look at the demo at https://www.youtube.com/watch?v=NUw-PimK6rQ

HP 3PAR and support for ODX

Over the weekend my fellow MVP Flemming Riis from Denmark contacted me about a new customer bulletin (c04205854) from HP about HP 3PAR StoreServ Storage with HP 3PAR OS 3.1.2 which warns for a serious issue when used with Windows Server 2012 and Windows Server 2012 R2. In the past year we saw and heard of several cases with Live Storage Migration causing corruption of VHDX caused by a bad implementation of ODX in HP 3PAR’s firmware. Previously we advised users to disable ODX at the operating system level as a temporary workaround. Now it seems, HP has found the issue and offers a patch for HP 3PAR OS.

Description

An issue has been discovered with the HP 3PAR OS and the use of Windows Server 2012 Off-loaded Data Transfer (ODX) commands, which may result in a number of blocks incorrectly zeroed beyond the requested range under certain conditions. This issue is not observed with ODX disabled.

Detailed analysis

When the Write Using Token requests using Block Device Zero Token is greater than 16 MBs in size, and the size is not a multiple of 16 MBs, a number of blocks may be zeroed beyond the end of the requested range. Disabling ODX eliminates the use of Write Using Token requests using Block Device Zero Token.

Scope

Windows Server 2012 or Windows Server 2012 R2 hosts with ODX in use with HP 3PAR StoreServ Storage running HP 3PAR OS version 3.1.2 GA, 3.1.2 MU1, 3.1.2 MU2, 3.1.2 EMU2, or 3.1.2 MU3.

Resolution

Upgrade the HP 3PAR OS on the HP 3PAR StoreServ Storage to 3.1.2 MU2 or later if running a lower HP 3PAR OS version. Next apply the patch as follows:

  • For 3.1.2 MU2 and 3.1.2 EMU2, apply Patch 11 followed by Patch 36.
  • For 3.1.2 MU3, apply Patch 30.

Workaround

Disable ODX on the Windows 2012 or Windows 2012 R2 hosts. To disable ODX:

Set-ItemProperty HKLM:\SYSTEM\CurrentControlSet\Control\FileSystem -Name “FilterSupportedFeaturesMode” -Value 1

A server reboot is required for all servers in which the registry value is modified.

If you have an opportunity to test this patched version of HP 3PAR in your own environment, I would strongly advise you to do this first!

You can find the customer advisory here