With Windows Azure there are a lot of options on how to manage and implement deployments, upgrades, scale and availability.  Mission Critical applications require more effort and planning on which Azure features to leverage based on its availability requirements. 

Understanding Updates & Upgrades

There are three types of Update events that can occur on the Windows Azure Platform.

• ·         Application Upgrade
• ·         Guest OS Update
• ·         Host OS Update

An Application Upgrade occurs when you do an in-place upgrade on your application, options for this is covered in the Deployment section.

The Guest OS Version is controlled via the Azure Service Configuration by setting osFamily and the osVersion attributes.  The osFamily attribute currently can be a 1, Windows Server 2008, or a 2, Windows Server 2008 R2.  The osVersion controls a group of OS patches and updates for a given time, by setting the osVersion to “*” instead of an explicit version, this will install OS patches automatically.  For Mission Critical applications you will have to determine if having the latest patches, including security vulnerabilities installed automatically is more of risk to your solution, than manually performing the upgrade.  There are two ways to perform this update manually either through the Service Configuration or through “Configure OS” within the Management Portal.

The Host OS Update occurs one the Host system.  Windows Azure tenants have no control as to when this update happens. To ensure the 99.95% SLA, Windows Azure will not update all hosts at once but perform host updates on host on different Fault domains, which is why you must have at least 2 instances for each role for the guaranteed 99.95% SLA.  The Scalability & Elasticity section goes into more detail on how you can ensure capacity during these updates.

Increasing Scalability & Elasticity

A lot of times it is easy to choose the largest instance possible, surely adding more memory or CPU will increase the performance of your application?  The problem is that most application are not written specifically to leverage multiple CPU cores and I have yet see an application that actually needed 14 GB of memory that the Extra Large VM provides.

Which is better:  4 Mediums or 2 large instances?

I would argue that 4 Mediums would be “better” as it would give you elasticity to increase OR decrease the number of instances based on my load at any given time, yet reducing cost by not over provisioning resources.  For example if you have 2 Large Instances – You couldn’t scale down without sacrificing the 99.95% SLA provided for roles that have at least 2 instances and you couldn’t scale up without paying for an entire new Extra Large VM.   

Also by choosing the medium instances over the large instances, it would allow you to increase the number update domains from 2 to 4, this allows you to have higher capacity availability during Host OS updates, Guest OS updates and Application updates.   For example if a Host OS update occurs, which you cannot control when it happens,  and you were using 2 Large instances, during this time frame your solution could only handle 50% of your maximum capacity.  On the other hand if you had used 4 Medium instances you could choose to create 4 update domains, one for each instance, which means during any update your solution, would be able to handle 75% of your maximum capacity.   

What is your solutions tolerance for reduced capacity?  Maybe 25%-50% reduced capacity for a short period is acceptable, if not what are your options?

The simplest option is to create an additional update domain that runs the same number of instances as all your other domains.  In the example above of using 4 Mediums instances, this would mean running a 5^th update domain with 1 additional medium instance.

Another option would be to consider auto scaling or reducing feature availability for features that are CPU and/or memory intensive.  The Enterprise Library Integration pack for Windows Azure includes WASABi , Windows Azure Autoscaling Application Block.  This application block allows you to create rules to scale up/down and to reduce resource intensive features on the fly. Developers can download this application block from: http://www.microsoft.com/en-us/download/details.aspx?id=28189

Increasing Availability with Geo-Redundancy

With two fault domains in a single datacenter and multiple upgrade domains, you are setup nicely in the event of an inner data center failure.  What about a complete datacenter failure?  As with an inner datacenter failure, you have to decide what is your tolerance for a complete datacenter failure?  If your tolerance is low, then you should consider deploying your application into two data center and if you are not using auto scaling type features – this means running a complete duplicate of all of your instances in a secondary data center to run at 100% capacity you had prior to the failure.  Once you have your solution deployed into two data centers, you can leverage the Traffic Manager feature to create a new ‘Failover’ policy for your secondary deployment. 

It sounds simple right? Well there are other considerations when planning to have a complete failover, such as are you leveraging any other Windows Azure features, such as storage, Service Bus, Cache or Access Control?  Since these services are datacenter dependent in the event of a failure, these services may not be available.

This is not a simple task and will take additional development and planning.  While I’ve geared this section to be towards for a complete data center failure, this could be leveraged for a specific service failure.  For example, if your solution leverages Table Storage, if the storage service for the data center your application has a failure, your Windows Azure Compute instances will be running, but any features leveraging storage will not be available and depending on the feature this could be a critical feature of the application.

Deployment Process Recommendations

With Windows Azure there are multiple options for deploying applications. You can use Visual Studio, the management portal, or something custom using the Management Service API. There isn’t one way that is better than the other, but you should deploy your application in a consistent method across multiple environments: Development, Testing, Staging and Production.   

If it is possible within your team or organization, you should consider automated deployments.  This option takes additional time, but allows you to deploy consistently.  Automated deployments can be achieved in various ways, but currently require a custom development effort. The largest benefit to automated deployments is for operations teams to be able to rapidly re-deploy services in other data centers in of a catastrophic data center failure.   For thoughts around how an automated deployment could be implemented, read the “Automating Deployment and using Windows Azure Storage” in the Moving Applications to the Cloud, written by Microsoft’s Patterns and Practices team. http://msdn.microsoft.com/en-us/library/ff803365

If automate deployment isn’t feasible, you should consider leveraging Windows Azure PowerShell Cmdlets, available for download here: http://wappowershell.codeplex.com/

These PowerShell Cmdlets provides consistent management accessibility that conforms to other Microsoft products that are leveraging PowerShell as a management interface.

Another common question is “Should I use a VIP Swap or leverage the In Place Upgrade?”  With the 1.5 Windows Azure SDK – In Place upgrades features are on-par with a VIP Swap.  So what is the advantage of using a VIP Swap?  The advantage is the ability to follow a process, to smoke test your application prior to it being published “live”.    IF you were to use an in-place upgrade, once complete the application is “live”.   Mistakes happen in IT, what if someone published the wrong version?  If you used VIP swap, you could smoke test the application first and even after you perform the VIP swap, you could revert back to the previous version instantaneously!

Solution Monitoring

Even if you are using an auto scaling framework such as WASABi – Monitoring is an important aspect.   Monitoring will assist in diagnosing issues within your application and knowing important key metrics about your application performance, errors and load. While Windows Azure can log Windows Events, Performance Counters and Logs to your storage accounts there is no “Out of the Box” Solution to view this data in a user friendly, graphical format.

System Center Operations Manager offers a Windows Azure management pack, which allows you to view alerts for Events and Performance metrics of your applications in a friendly manner. 

The Windows Azure Management pack is available for download:  http://www.microsoft.com/en-us/download/details.aspx?id=11324

Disaster Recovery

While it is true that Windows Azure does backup your Storage and SQL Azure data in triplicate, however this isn’t for you to leverage to restore your data on-demand.  These backups are used by the Windows Azure teams in case of a catastrophic failure so they can restore an entire datacenter and not necessarily recover data for individual Windows Azure tenants.  While this provides some relief, it is best to have backups of your application, configurations and data so that you can restore onto another Azure datacenter manually.   While creating backups of Storage data would be a custom development effort, relational data stored on SQL Azure can be backed up and restored to a file using the Import/Export feature in the Windows Azure.

• Application Settings - Don't use the web.config to store application settings, specifically environmental application settings. For example: SQL Connection strings, should be stored in the
  Service Configuration not the web.config. The problem is since the web.config is part of the deployment package, you will either have to redeploy a new package or Remote Desktop into all of the instances to make a change.  Think about how you move builds across environments: Development --> Test --> UAT --> Production

            Common Web Config Settings that should be moved to the Service Config

• SQL Connection Strings
• Including Entity Framework
• AppFabric Caching
• Yes – Azure lets you copy the web.config settings for caching, but caching it typically going to be environment based so move it to the Service Configuration.

• ASP.NET Membership Provider 

• Deployment – Leverage the VIP Switch functionality, don’t just deploy to the production node. Deploy to the staging node, smoke test, then do a VIP Switch. If something isn’t right, this gives you a great way to rollback. Yes – you have to pay for the staging compute instances. There are some cases where you can’t leverage this feature, such as a change to a WCF contract, where WCF client needs to be updated as well. Plan your deployment strategy for this scenario.

• Scale Out, Not Up – Too many times it’s easy to say I’ll just use the biggest VM size available.  Your memory and CPU footprint should account for load, but you can spread that load across multiple instances.  IE: 2 x Large Instances or 4 x Medium or 8x Small instance? In a lot of cases – you’d be better off with 4x Medium or 8x Small than 2x Large and it’s the same cost.  

• Performance and Diagnostic Data – Collect this data, it makes troubleshooting alot easier for you. At a minimum capture the crash dumps, so they will be stored Blob storage – Ideally you have some kind of Monitoring solution deployed on Premise – like SCOM with the Windows Azure management pack or another third party solution.

• Don’t enable Profiling or Remote Desktop for production instances – Profiling will be a performance hit in production, Remote Desktop exposes another potential attack surface. Leverage Diagnostic and Performance data to troubleshoot issues.

• Don’t leverage CTP features in Production – CTP features are great, but don’t have SLAs. (Once in Beta features typically have SLAs). IE: Traffic Manager is a great feature, but while it is in CTP, it is extremely risky as a single point of failure with no guaranteed SLA.   

Have some thoughts? Share them with me: midunn@microsoft.com

About my new team: http://www.microsoft.com/microsoftservices/en/us/support_premier_addons_pmc.aspx

This means I'm pretty much done with Unified Communications, I still take questions around the APIs from time to time, but I’ll be changing the theme of the content from UC to Azure. I have lined up some posts about best practices and tips on using Windows Azure for Mission Critical applications:

• Common Mistakes
• Deployment Best Practices
• Geo Redundancy Possibilities

Hopefully there are a few of you, that still check up on this blog and maybe some new readers.

What kind of scenario does this open? One example improved desktop experience. Well SAPI is the obvious first choice when creating a desktop application, but the SAPI experience is really dependant on the OS. IE: TTS voices are different from XP to Vista. The benefit to having a server side solution you can control the experience of the end user has.

While looking to build something there are some problems to overcome, for me building a simple sample solution, two main problems stick out.

1.) Streaming Audio - I don't really want to pass around and process WAV files. You could but the experience wouldn't be as nice as you couldn't do immediate speech recognition.
2.) As I won't be using System.Speech or SAPI, I am going to have to access hardware resources such as the microphone and speakers.

For streaming audio there are alot of different options, but being a big fan of WCF, this is first avenue I looked at and sure enough you can stream content to/from WCF. Is there anything WCF can't do!

 http://msdn.microsoft.com/en-us/library/ms731913.aspx

 Next access microphone and speakers via .NET, I don't want to write low level code I want something down and dirty. I found a very cool CodePlex project, NAudio. It abstracts a lot of the audio APIs into a single, easy to use API. This will work perfectly for my client side application.

http://www.codeplex.com/naudio

Here is a simple diagram of what I am building, and actually very close to having something running.

Note: If this was something which I was planning to actually put in a production environment, I'd separate the Synthesizer and Recognizer. These will be independent calls and would benefit from being on their own servers.

Code Sample will be available soon, hoping to have it up on MSDN samples by November 16th.

Since MSPL hasn't really changed from LCS to OCS, we can use the sample provided for LCS to implement a custom disclaimer for OCS.  You can download the sample from: http://www.microsoft.com/downloads/details.aspx?familyid=ee41345b-836c-4dcf-9810-32709ae9f5c4&displaylang=en&tm

You'll have to make some project modifcations because it doesn't compile out of the box. These are easy like missing references etc..

If you want to first run the console application, for debugging and/or learning purposes, you'll notice that the application manifest, ".am",  is missing. Just copy the the DisclaimerService.am from the service proejct, open it in notepad and change the AppUri to http://www.microsoft.com/LC/Disclaimer/Console.

Finally when you are ready to install the service portion of the solution, you'll have to change the ServicesDependsOn property on serviceInstaller1 in ProjectInstaller.cs as it is currently set to "LCSProxy.exe".  Just set this to be blank.

After these minor changes, simply continue following the instructions included with the sample.

You can find the code, binaries and an initial "How To":
http://code.msdn.microsoft.com/ucmahost

The home page of the project site gives a basic walk through. One item I sort of mentioned on the site was the ability to interface with the service via MSMQ. Instead of specifying a Inbound or Outbound workflow, you can just provide the name of a private queue. This will watch that queue and send an alert via a default Outbound WF or the provided Outbound WF.

To use this simply write this string to MSMQ, sipuri={0}&message={1}&type={2}, providing the details of who you want to send the message to, the actual message and type. The type can be 1, Audio, or 2, IM.

These workflows are great for those needing to great simple yet powerful UC enabled applications. The workflow activities themselves are pretty self explanatory. I'll cover the activities themselves in another post.

 The "difficult" portion of creating a UCMA workflow application isn't the actual workflow, but creating the host. Yes the default template wraps your WF application is nice little console application, but this is really only for debugging purposes. Are you really going to deploy your console application in production? Those familiar with Speech Server are probably scratching their heads, while Speech Server had a nice service and administrator console that allow you to easily point it to your Speech WF application, UCMA does not. You need to build it yourself and build it for every UC application you build, and to truly do that we need to know at least a little bit about the UCMA Core API. Joe Calev has a great blog which walks you through the Signaling portion of the Core API. For the WF API you really should have some understanding of the Collaboration portion of the Core API.

 First you will need to make sure that you have a certificate on for your application and you will need to provision an application on the OCS Front End. You can do this by using the Application Provisioner and following these instructions: http://msdn.microsoft.com/en-us/library/dd253360(office.13).aspx

 This is the key information that our application will need:

• Application Name
• Application Port
• Application SIP URI
• Application Server FQDN
• OCS Pool FQDN
• OCS Pool Port
• GRUU
• Certificate Information

Note: You should never call the asynchronous methods, synchronously. IE object.End(object.Begin). You'll see some of this in sample code and blog posts, do not use this is a production application!

Armed with the necessary information, we can start looking at some code:

First we need to use create an instance of the CollaborationPlatform, for this post the importance of this object is that it is used to describe our application. It has two constructors, one that takes a ClientPlatformSettings parameter and the other that takes a ServerPlatformSettings.

The different here is the ClientPlatformSettings will use an AD account, while a ServerPlatformSettings will use a Contact Object. We reference the Contact Object via the GRUU provided by the Application Provisioning tool. There are some features lost if you use the ClientPlatformSettings, such as Impersonation. I typically draw the line, if the application is going to consume the Audio or Instant Message call and perform some logic based on the input I use the ServerPlatformSettings. IE: Response Bots. If my application is just providing an abstraction method for a user that already has credentials to sign in and message users, then I would use the ClientPlatformSettings. IE: Build your own authenticated Web Messaging.

For this post, we will use the ServerPlatformSettings.

ServerPlatformSettings platformSettings = new ServerPlatformSettings(appName, appServerFqdn, applicationPort, gruu, cert);
CollaborationPlatform collabPlatform = new CollaborationPlatform(platformSettings);

collabPlatform.EndStartup(collabPlatform.BeginStartup(null, null))

Once the CollaborationPlatform object has completed starting up, we need to setup our EndPoint. There are two choices, ApplicationEndPoint and UserEndPoint, if you are using the ClientPlatformSettings then you need to use the UserEndPoint, since we are using the ServerPlatformSettings we need to use the ApplicationEndPoint.

ApplicationEndpointSettings settings = new ApplicationEndpointSettings(applicationUri, ocsFqdn, ocsTlsPort);
ApplicationEndpoint endpoint = new ApplicationEndpoint(collabPlatform, settings);

If our WF application is an Inbound application, we need to register for the type of calls the WF will handle, but remember that the WF can only accept AV and IM calls and can't accept conferencing calls. That’s not to say you couldn’t register to receive a conference invite, and handle the conferencing features using the Core API itself, but that is another post.

endpoint.RegisterForIncomingCall<AudioVideoCall>(AudioVideoCallReceived);
endpoint.RegisterForIncomingCall<InstantMessagingCall>(InstantMessagingCallReceived);

Once we have done that, we can establish our endpoint.

endpoint.EndEstablish(endpoint.BeginEstablish(null, endpoint));

Next we need to start the Workflow Runtime and add the UCMA WF Services, this isn’t so much related to UCMA as it is to WF itself.

WorkflowRuntime  workflowRuntime = new WorkflowRuntime();

workflowRuntime.AddService(new CommunicationsWorkflowRuntimeService());

workflowRuntime.AddService(new TrackingDataWorkflowRuntimeService());

workflowRuntime.StartRuntime();

Finally we need to route the call to our workflow, via the handler methods we registered for earlier.

AudioVideoCallReceived
InstantMessagingCallReceived

When a call comes in, we need to pass the call to our workflow application and from there the Workflow handles the rest:

WorkflowInstance workflowInstance = workflowRuntime.CreateWorkflow(WorkflowType);

CommunicationsWorkflowRuntimeService communicationsWorkflowRuntimeService = (CommunicationsWorkflowRuntimeService)workflowRuntime.GetService(typeof(CommunicationsWorkflowRuntimeService));

communicationsWorkflowRuntimeService.EnqueueCall(workflowInstance.InstanceId, call); //Call object is passed by the receiving handler

communicationsWorkflowRuntimeService.SetEndpoint(workflowInstance.InstanceId, endpoint); //Endpoint object is a local variable

communicationsWorkflowRuntimeService.SetWorkflowCulture(workflowInstance.InstanceId, new CultureInfo("en-US"));

workflowInstance.Start();

I was thinking about it and all of this is pretty repetitive code from one workflow to another workflow application, this all has to be done, everytime for every WF project. I’ve created a solution that I will release on CodePlex last this week. Basically the solution is a Windows Service that will take the parameters, GRUU, port, etc… log into OCS and route calls to the workflow you provide as a compiled assembly. Also part of this solution utilizes MSMQ, in much the same way Speech Server did. You can associate a MSMQ name to an application, if a message is in that queue if will fire off the workflow you specify or use the default built in WF application.

The Jist of it is, instead of worrying about writing all the code above, you can concentrate on just the workflow portion. Compile the WF as an Assembly instead of a Console application, add the application parameters to the App.Config of this UCMA Application Host service and you are done! A picture is worth a thousand words, so here is a high level picture of the solution.

I am also working on releasing the following solutions on CodePlex:

• Updated Web Chat sample. The current released Web Chat uses UC AJAX, this updated one will use UCMA 2.0, WCF & SilverLight.
•  Client Framework. An abstraction layer to both UC Client and Communicator Automation.

Sign up for the OCS 2007 R2 Virtual Launch on Feburary 3rd.
http://www.microsoft.com/communicationsserver/virtualevent/languageselect.aspx

And just to keep entertain until Feburary 3rd, here are some R2 launch teaser videos:
https://www.youtube.com/profile?user=OCSR2Launch&view=videos

http://www.microsoft.com/downloads/details.aspx?FamilyID=fad62198-220c-4717-b044-829ae4f7c125&displaylang=en

I promise I will get the the UCMA 2.0 Workflow post I promised, but those of you wanting to get a taste of UCMA now, check out Joe Calev's blog, he currently has a full seris on UCMA 2.0 Core right now. http://blogs.msdn.com/jcalev/default.aspx

Right now I am in Redmond for 3 weeks for the first rotation of Microsoft Certified Master program for Office Communications Server 2007 R2. All I can say it that it will be very very long days, including doing training on the weekend. As much as I am looking forward to it, I am also dreading it. 3 weeks away from home, actually 4 weeks as I will also be attending TechReady, our internal conference, and probably 11-12 hours in a classroom...  This class probably is not for the faint of heart.

If are not sure what it is MCM is, you can read more here: http://www.microsoft.com/learning/mcp/master/OCS/default.mspx

The long time away from home will actually give me some time to finally blog.