Windows Azure Pack High Availability

This blog was published a couple of days after Microsoft announced (at TechEd 2013 North America) that Windows Azure Pack is the new name for Windows Azure for Windows Server. This blog is based on the RTM bits of Windows Azure for Windows Server. When the Windows Azure Pack software is available I’ll check the installation and if any changes are required I’ll update this blog.

My previous post System Center VMM 2012 SP1 High Availability with SQL Server 2012 AlwaysOn Availability Groups describes a complete step by step to install and configure a Private Cloud with high availability on the host level, the database level and the VMM level. With System Center 2012 SP1 Service Provider Foundation High Availability multi tenancy was added to the Private Cloud enabling Service Providers to provide their customers with an extensible OData web service and even leverage the investment in their current portal by connecting it to the extensible OData web service.

To top it off this blog will describe a complete step by step for installing Windows Azure for Windows Server in a High Available configuration. You could add two servers to the environment that was built in the previous blogs.

01 WA4WS HA Design Single Domain Express

It is also possible to create a separate domain for Windows Azure for Windows Server with a dedicated host cluster, a SQL Server 2012 AlwaysOn Availability Group and two virtual machines with Windows Azure for Windows Server in a high available installation.

02 WA4WS HA Design Separate Domain Express

The procedure for installing and configuring a high available Windows Azure for Windows Server is the same for both environments. I will add two virtual machines Portal1 and Portal2 to the domain that was built in the previous posts.

High availability in Windows Azure for Windows Server can be enabled with a hardware load balancer or with Network Load Balancing. In my previous blog System Center 2012 SP1 Service Provider Foundation High Availability you can find a walkthrough on how to enable Network Load Balancing on virtual machines.

Prerequisites

Virtual Machines and Network Load Balancing

For this walkthrough I added two virtual machines (Portal1 and Portal2) to the domain. Each virtual machines is running Windows Server 2012. I installed the Network Load Balancing feature and created a Network Load Balancing cluster with a single NIC per server in Unicast mode.

03 NLB Cluster Parameters

The full internet name for the NLB cluster is portal.hyper-v.nu.

Service account for registering the Service Provider Foundation

To register Windows Azure for Windows Server to System Center Service Provider Foundation a local user account is required on the SPF server. In a high available environment there are two or more SPF servers. We need to create a local user account on each SPF server and it should have the same name and the same password.

04 Local SPF service account

Add the local service account to the following local groups SPF_Admins, SPF_Provider, SPF_VMM on each SPF server.

Certificates

The main purpose of Windows Azure for Windows server is to give customers a simple and intuitive web interface for managing their services that are running at the Service Provider. This requires the Service Management Portal Tenant Site and optionally the Service Management Portal Tenant API made accessible from the Internet. For secure access a web certificate must be obtained from a public Certificate Authority with common name equal to the full internet name of the NLB cluster (in this example portal.hyper-v.nu).

SQL Server mixed mode authentication and SA password

Windows Azure for Windows Server requires SQL Server authentication in mixed mode for a high available installation. In you followed the steps from the previous blogs SQL Server is running in Windows Authentication mode. To change these settings logon to SQL1 and open SQL Server Management Studio. Right click the server name in the object explorer and select properties. On the security tab select SQL Server and Windows Authentication mode. Applying these settings requires a restart of the SQL Server service. Repeat these steps on SQL2

05 SQL Authentiction mode

In the database configuration step of the Windows Azure for Windows Server configuration wizard you are required to specify credentials for SQL Server authentication mode. The System Administrator (SA) account is disabled when you change from Windows Authentication mode to SQL Server and Windows Authentication mode. It possible to create a dedicated system account or use the built in SA account. To enable the SA account logon to SQL1 and open SQL Server Management Studio. Open the Security > Logins entry, right click the SA account and select properties. On the status tab change the login setting to enabled.

06 Enable SA account

On the general tab specify the password for the System Administrator and apply the settings. Repeat these steps on SQ2. Take note that the SQL account and password should be identical on each SQL Server.

Express mode or distributed installation

Windows Azure for Windows Server can be installed in express mode (all components on a single server) or in distributed mode (spread the components over multiple servers). Each mode can be made high available.

07 WA4WS HA Design Separate Domain Ditributed

The steps to install the express mode and the distributed mode are described in my blog Installing and configuring Windows Azure for Windows Server – Part 1. I will use the express mode and make it high available in this blog. The steps to make the distributed mode high available are the same.

Installing Windows Azure for Windows Server

In my previous blog Installing and configuring Windows Azure for Windows Server – Part 1 you can find a detailed step by step for the installation procedure. I will just highlight the steps here.

  • Move the SQL Server Availability Group to SQL1
  • Logon with a domain administrator account on Portal1
  • Run the Microsoft Web Platform Installer
  • Select the products tab and in the left menu select Windows Azure
  • Add the Service Management Portal and Service Management API Express and select install

When the installation wizard completes the configuration page opens with URL https://localhost:30101/. On the Database server setup screen specify the SQL Availability Group DNS listener (in this example SQL), select SQL Server Authentication and enter the SA account and credentials we set in the prerequisites of this blog. Provide a configuration store passphrase. This passphrase will be used to add additional servers to the same configuration for high availability.

08 Database Server Setup Portal1

In the next screen the configuration wizard creates all the components. When the installation wizard is complete the wizard opens the Admin Site with URL https://localhost:30091/. In the default configuration it is only possible to logon to the admin site with the local administrator. You can force using this account by entering the username in the following format .administrator with the correct password.

09 Logon with local admin

It is also possible to validate against the Admin Site with domain accounts. To enable this feature open the Local User Manager (lusrmgr.msc) and add the domain users or domain groups that require access to the Admin Site to the local group MgmtSvc Operators.

10 MgmtSvc Operators

Close the Admin site and open it again.

11 Domain account on admin site

When you are still logged on to the server as a domain administrator and you added this account to the MgmtSvc Operators group you should be able to login with your current credentials.

Add the Windows Azure for Windows Server databases to the Availability Group

Before configuring anything in the Admin Site first enable and test high availability of the Windows Azure for Windows Server databases. The step required to add the databases to the Availability Group are described in the previous blog. I will mention the steps in short.

  • Logon to SQL1 (which we made primary for the Availability Group before installing Windows Azure for Windows Server)
  • Windows Azure for Windows Server uses five databases
    • Microsoft.MgmtSvc.Config
    • Microsoft.MgmtSvc.PortalConfigStore
    • Microsoft.MgmtSvc.Store
    • Microsoft.MgmtSvc.Usage
    • Microsoft.MgmtSvc.WebAppGallery
  • Change the recovery model of all five databases to full and then make a backup of all five databases
  • Run the Add database to Availability Group wizard and add the five databases the Availability Group

The Availability Group should now contain all the Windows Azure for Windows Server Databases.

12 DBs added to Availability Group

Availability Groups and SQL Server user accounts

When the databases are added to the Availability Group we are able to make the second SQL Server (in this example SQL2) the primary node. Use the failover cluster manager to move the role to SQL2. Logon to Portal1 (the server just used to install Windows Azure for Windows Server) and refresh the Admin Site by hitting F5. There is clearly something not working correctly.

13 Portal warnings

There are some event logs created by the Windows Azure for Windows Server installer. You can find the logs in the Event Viewer > Applications and Services Logs > Microsoft > WindowsAzure. The Server-Management log is filled with authentication errors.

14 Eventlog

Opening an event will show a login failed for a SQL user.

15 Event Properties

We have seen this issue before in the previous blogs. SQL Availability Groups replicates databases but not user accounts. Opening SQL Server management studio to compare Logins on both SQL Servers will show that the Windows Azure for Windows Server created a bunch of user accounts on SQL1 that are not available on SQL2.

16 SQL1 accounts17 SQL2 accounts

In the previous blogs we just added the domain service account to the second SQL Server and we were good to go. But there is a little problem here. The user accounts created by Windows Azure for Windows Server are SQL Server accounts and not domain accounts. Even if we managed to create exactly the same accounts we would not know how to match the password that were created by the installer.

This is where the great Microsoft Windows Azure Pack product team stepped in and provided us with a solution. Thanks to Marc, Bradley and Bala for helping out!!

It is possible to export the user accounts (domain and SQL accounts) to a plain text file with hashed password by following this knowledge base article.

Basically you create two Stored Procedures by running the script in the knowledge base article on your master database. Run the script on SQL1 (which we made primary for the Availability Group before installing Windows Azure for Windows Server and therefore contain the local user accounts).

18 Stored Procedures

Open a new query and run EXEC sp_help_revlogin

The output from the Stored Procedure contain the domain and SQL accounts.

19 Execute stored procedure

Compare the output to the user accounts on SQL2 and remove the lines that contain user accounts that are already available on SQL2. Connect with the SQL Server Management Studio to SQL2 and open a new query screen. Paste the output from the Stored Procedure we just adjusted and execute it. This will create the required local SQL accounts on SQL2 with the correct passwords.

Logon to Portal1 and refresh the Admin Site again. All warnings should disappear and the Admin Site should operate correctly.

Enable high availability for Windows Azure for Windows Server

Now that high availability is enabled on the database level for Windows Azure for Windows Server we are almost ready to install Windows Azure for Windows Server on the second server. To verify that the confirguration is using the same database we need to create on object in the database. The easiest way to configure an object without requiring additional configuration is to make a Hosting Plan with a MYSQL Servers service. Open the Admin Site on Portal1 (https://localhost:30091/) and select PLANS in the left menu. Click Create A New Hosting Plan. Give a friendly name for the plan (for example Hosting Plan for testing HA), select MYSQL Servers in the Select services for a Hosting Plan screen and finish the wizard.

20 Refresh admin site

The installation procedure on Portal2 is equal to Portal1

  • Logon with a domain administrator account on Portal2
  • Run the Microsoft Web Platform Installer
  • Select the products tab and in the left menu select Windows Azure
  • Add the Service Management Portal and Service Management API Express and select install

When the installation wizard is complete the configuration wizard will open with URL https://localhost:30101/ and you are prompted with the Database Server Setup screen. Specify the SQL Availability Group DNS listener (in this example SQL) in the server name. Select SQL Server Authentication. Specify the System Administrator account and password you used for the configuration of the first server (in this example SA) and specify the Configuration Store Passphrase that you created when configuring the first Server.

21 Database Server Setup Portal2

When you have entered the correct Configuration Store Passphrase in the first field you will notice that the Confirm Passphrase field is greyed out. This validates the passphrase against the current database and confirms that you will add this server to the current configuration.

When all components are installed successfully the Admin Site will open with URL https://localhost:30091. You can add the required domain users or domain groups to the local MgmtSvc Operators Group to enable these accounts to open the Admin Site on this server.

Verify high availability for Windows Azure for Windows Server

Open the Admin site on the second server (Portal2) and verify that the Hosting Plan we created earlier is available. Select VM Clouds in the left menu and select Register Service Provider Foundation. Provide the URL to the Service Provider Foundation web service using on port 8090 (in this example https://spf.hyper-v.nu:8090). Specify the local service account and password we created in the prerequisites (in this example SVC_SPF_REG).

22 Register SPF

The Tenant Site will most likely be made accessible on the internet. To enable secure access to the Tenant Site make sure that each server (Portal1 and Portal2) has the public web certificate installed that was requested from a public Certificate Authority in the prerequisites section of this blog. Open Internet Information Services (IIS) Manager on Portal1, right click the MgmtSvc-TenantSite website and select edit bindings. Change port 30081 to 443 and select the public web certificate (in this example portal.hyper-v.nu).

23 Tenant Site bindings

Follow the steps in my blogs Installing and configuring Windows Azure for Windows Server – Part 2 and Installing and configuring Windows Azure for Windows Server – Part 3 for further configuration.

You should now be able to shut down one server on the host level, the database level, the VMM level, the SPF level and the Windows Azure for Windows Server level without service disruption.

More Information

Windows Azure Pack

Enabling On-Premises IaaS Solutions with the Windows Azure Pack

At the moment of publishing this blog this session is not available yet but keep an eye on this one

3 Comments

  1. November 29, 2013    

    Hi
    I have followed your article steps & trying to install WAP in HA but when I am shutting down portal-1 VM my admin site from Portal-2 is not working.

    Cloud you please let me know what went wrong in case why it is not working from other server.

    Thanks
    BG

  2. January 15, 2014    

    Hi Marc,

    I have deployed KATAL in HA mode, with proper steps provided by you but sometimes in Tenant Sites, we are unable to get VM Templates

    more details are on mentioned link

    http://social.technet.microsoft.com/Forums/windowsazure/en-US/d03f54bf-f1d9-4e92-8a9f-20b419ab9165/creating-vm-via-tenant-portal-fails?forum=websitesvirtualmachinesonwinserver#fc50c11a-cfb6-4113-b657-5e64023d290d

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>