Today I encountered an issue that some might categorize as “DUH!”. For others it might be a handy tip and a warning to not install SCVMM 2012 RC in a high availability configuration as I did. This configuration was not intended to be this way but for some reason I ended up with it. Lucky me!

Okay, let’s start. So what was the issue and how did I solve it. But first I will tell you what my configuration was.
I have a two node failover cluster to makes SCVMM 2012 high available. Both servers are Windows Server 2008 R2 SP1 and have all roles and features installed that should be installed when configuring a failover cluster. They look similar, but are they! If you read the recommendations from Microsoft the setup that I have is not ideal but should work. Because of hardware / finance limitations I had to put some roles on the same servers. The image below shows how I set it up.

SCVMM2012-HA

Both servers are virtual machines with the following installed:

  • VMM Management Server
  • VMM Console
  • VMM Self-Service Portal

On top of this I also installed Microsoft SQL 2008 R2 locally. As stated before, this is not the recommended setup. It is better to spread the features over multiple servers. Good starting point to figure out what the recommended configuration is the TechNet website: Virtual Machine Manager.
So far nothing special! After successfully installing all features I did my first failover and it failed over successfully! Yeah for me Smile !!!

But then I did some more failovers and it just broke. One of the services in the SCVMM 2012 cluster failed. To be more specific the “VMM Service”. The following messages appear in the event log on the node where I did the failover to.

SCVMM2012-HA-EventID-1

Log Name: VM Manager

Source: Virtual Machine Manager

Date: 30-11-2011 22:13:23

Event ID: 1

Task Category: None

Level: Error

Keywords: Classic

User: N/A

Computer: SERVER02

Description:

System.FormatException: The string was not recognized as a valid DateTime. There is a unknown word starting at index 0.

at System.DateTimeParse.Parse(String s, DateTimeFormatInfo dtfi, DateTimeStyles styles)

at System.Convert.ToDateTime(String value)

at Microsoft.VirtualManager.Engine.SqmRefresher.IsRefreshRequired(String refreshTime)

at Microsoft.VirtualManager.Engine.SqmRefresher.RefreshData(SQMSessionManager manager)

at Microsoft.VirtualManager.Engine.RefreshDriver`1.RefreshThreadFunction(Object obj)-2146233033

Event Xml:

<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">

<System>

<Provider Name="Virtual Machine Manager" />

<EventID Qualifiers="0">1</EventID>

<Level>2</Level>

<Task>0</Task>

<Keywords>0×80000000000000</Keywords>

<TimeCreated SystemTime="2011-11-30T21:13:23.000000000Z" />

<EventRecordID>6771</EventRecordID>

<Channel>VM Manager</Channel>

<Computer> SERVER02</Computer>

<Security />

</System>

<EventData>

<Data>System.FormatException: The string was not recognized as a valid DateTime. There is a unknown word starting at index 0.

at System.DateTimeParse.Parse(String s, DateTimeFormatInfo dtfi, DateTimeStyles styles)

at System.Convert.ToDateTime(String value)

at Microsoft.VirtualManager.Engine.SqmRefresher.IsRefreshRequired(String refreshTime)

at Microsoft.VirtualManager.Engine.SqmRefresher.RefreshData(SQMSessionManager manager)

at Microsoft.VirtualManager.Engine.RefreshDriver`1.RefreshThreadFunction(Object obj)-2146233033</Data>

<Binary> SOME BINARY STUFF HERE ;-) </Binary>

</EventData>

</Event>

SCVMM2012-HA-EventID-19999

Log Name: VM Manager

Source: Virtual Machine Manager

Date: 30-11-2011 22:13:23

Event ID: 19999

Task Category: None

Level: Error

Keywords: Classic

User: N/A

Computer: SERVER02

Description:

Virtual Machine Manager (vmmservice:3820) has encountered an error and needed to exit the process. Windows generated an error report with the following parameters:

Event:VMM20

P1(appName):vmmservice

P2(appVersion):3.0.5007.0

P3(assemblyName):Engine.Common

P4(assemblyVer):3.0.5007.0

P5(methodName):M.V.E.SqmRefresher.IsRefreshRequired

P6(exceptionType):System.FormatException

P7(callstackHash):5104

.

Event Xml:

<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">

<System>

<Provider Name="Virtual Machine Manager" />

<EventID Qualifiers="0">19999</EventID>

<Level>2</Level>

<Task>0</Task>

<Keywords>0×80000000000000</Keywords>

<TimeCreated SystemTime="2011-11-30T21:13:23.000000000Z" />

<EventRecordID>6772</EventRecordID>

<Channel>VM Manager</Channel>

<Computer>SERVER02</Computer>

<Security />

</System>

<EventData>

<Data>Virtual Machine Manager (vmmservice:3820) has encountered an error and needed to exit the process. Windows generated an error report with the following parameters:

Event:VMM20

P1(appName):vmmservice

P2(appVersion):3.0.5007.0

P3(assemblyName):Engine.Common

P4(assemblyVer):3.0.5007.0

P5(methodName):M.V.E.SqmRefresher.IsRefreshRequired

P6(exceptionType):System.FormatException

P7(callstackHash):5104

.

</Data>

<Data>3820</Data>

<Data>VMM20</Data>

<Data>vmmservice</Data>

<Data>3.0.5007.0</Data>

<Data>Engine.Common</Data>

<Data>3.0.5007.0</Data>

<Data>M.V.E.SqmRefresher.IsRefreshRequired</Data>

<Data>System.FormatException</Data>

<Data>5104</Data>

</EventData>

</Event>

A quick BING got me the result to do the following on the VMM database.

update dbo.tbl_VMM_GlobalSetting set PropertyValue=NULL where propertyName=’LastSQMEngineRefreshTime’

Unfortunately this is a fix for a known bug in SCVMM 2008 which is explained here >>> SCVMM Crashing with Event ID 19999, M.V.E.SqmRefresher.IsRefreshRequired System.FormatException.

Then I looked at the EventID 1 again and noticed the phrase “The string was not recognized as a valid DateTime”. Okay, could it be that the regional settings be causing this?

Remember that I said that the server looked the same. Well they look same, but they weren’t. When looking closer at the regional settings I noticed that SERVER01 was setup with all DU-NL regional settings and that SERVER02 was configured  with all EN-US regional settings. As SERVER01 was the first server to be installed I decided to change the regional settings on SERVER02 to match those of SERVER01. I changed all settings to DU-NL.

SCVMM2012-HA-Regional-Settings_04

Next action is to make it the default for all system accounts and new accounts. After this a reboot is needed.

After the reboot of SERVER02 I tried the failover again and it still failed. Why? The reason for this is that some regional settings are stored in the users profile and some are computer-wide. Apparently I need to delete all the local profiles of the service accounts used for SCVMM 2012 before fixing this issue. The only way to do this correctly is to go into the advanced system settings and delete them via “User Profiles”. Restarted the server again and tried again.

SCVMM2012-HA-Failover

After this the failovers are as they should, successfully! I got my SCVMM 2012 high availability back again!

So, what is the lesson from all of this? Make sure that your regional settings are set identical before starting the installation of SCVMM 2012! Duhhhh Winking smile !!!

Also, if someone of Microsoft is reading this and as has influence on the RTM process, it might be an idea to implement a check in the wizard when creating a failover cluster!