My fellow blogger MVP Hans Vredevoort installed the most recent updates on an operational Windows Azure for Windows Server environment. After applying the updates the connectivity from the Service Management Portal to System Center VMM failed.
In the Service Management Portal the Service Provider Foundation was still registered. The amount of VM clouds returned to zero. Before the updates the amount of VM clouds in the Service Management Portal matched the amount Clouds in SCVMM. Uninstalling the updates from the servers did not resolve the issue.
Applying the most recent updates in my lab caused the same issue. I created a temporary environment for troubleshooting. This test setup consists of five virtual machines.
- dc.azure.lab – Windows Server 2012 / domain controller
- sql.azure.lab – Windows Server 2012 / SQL Server 2012 SP1
- vmm.azure.lab – Windows Server 2012 / System Center VMM 2012 SP1
- spf.azure.lab – Windows Server 2012 / System Center SPF 2012 SP1
- smp.azure.lab – Windows Server 2012 / Service Management Portal Express
The end to end solution for Service Providers can have different designs. This setup is the bare minimum required for troubleshooting the issue. I’m writing a complete guide for all the individual parts, the possible designs and related configuration, which will be posted on www.hyper-v.nu soon.
The installation and configuration of the test setup without applying the updates resulted in a correct functioning connection between the Service Management Portal and System Center VMM through System Center SPF.
The first thing to do is create some snapshots of the environment. The Windows Update feature detected about thirty available updates for each server. I decided to run the updates in batches of ten updates to speed up the process of finding the troublesome update. I starting with the SPF server, since I had a feeling that this server was causing the error.
After installing all the required updates on the SPF server the VM clouds still matched the number of clouds in System Center VMM. I rebooted the SPF server and run IISReset.exe on the SMP server just to be sure. It was still functioning correctly. So far for my hunch.
With the SPF server now fully patched I turned to the VMM server. I repeated the process of installing batches of ten updates. Rebooting the VMM server and running IISReset.exe on the SMP server after every batch. When the VMM server was fully patched the Service Provider Foundation connectivity from the Service Management Portal showed no errors.
I really doubted that the SQL server or even the domain controller were interfering in the issue, so the logical next server to patch was the server hosting the Service Management Portal itself. As you might guess by now, after applying all the critical updates in the SMP server the connectivity to the Service Provider Foundation was still functioning correctly.
Before moving my attention to the SQL server and the domain controller I enabled update support for Microsoft Update. A rescan of available updates on the VMM server displayed two optional updates.
Installing both updates did not cause the issue.
Enabling Microsoft Update on the SPF server resulted in one optional update.
After installing the update the connectivity from the Service Management Portal to the Service Provider Foundation failed.
Now that we have the update causing the error, let’s have a look if we can solve the issue.
The first thing to do is to check the knowledge base article describing the update.
The section Installation instructions > Installation instructions for Service Provider Foundation describes known issues for this update that gives insight.
After you install the Update Rollup 1 package on the Service Provider Foundation server, the updates do not appear in the Add or Remove Programs item in Control Panel
Uninstalling the updates in my original lab or in the environment Hans Vredevoort was working on did not result in a correct functioning environment. This explains why uninstalling the updates did not provide the solution since this update cannot be uninstalled in the Add/Remove Programs interface.
After you install the Update Rollup 1 package on the Service Provider Foundation server, the VMM endpoint Application Pools identity will be set to Network Services. To resolve this issue, use the IIS management console to reset the VMM Application Pools identity
The Application Pool Identity fulfills an important role in the connection from the Service Management Portal to the Service Provider Foundation. I will get into more detail on this in the upcoming blog series.
I reverted the snapshots of all the virtual machines to the correct functioning environment before the updates were installed. Opening IIS on the Service Provider Foundation server displays three Application Pools that have the domain service account as Application Pool Identity specified.
Installing Update Rollup 1 for System Center 2012 Service Pack 1 changes the VMM application pool identity to Network Service.
To change the Application Pool Identity back the domain service account right click the Application Pool and select advanced. Select the More button next to the Network Service Account and specify the domain service account
A refresh on the Service Management Portal resulted in a correct number of VM clouds. Even without running IISReset.exe on the Service Provider Foundation.
Installing Update Rollup 1 for System Center 2012 Service Pack 1 creates an additional Application Pool named Usage that allows for virtual machine usage data reporting by connecting to Operations Manager. This is a very interesting development but deservers a blog on its own.