I realize this post is pretty useless, but it seemed kind of weird to start posting again after 3 years without some sort of segue ...

Availability

With Windows Azure, SXP service releases are literally push-button activities with zero planned downtime. The SXP team uses the Virtual IP (VIP) Swap feature on the Windows Azure Platform Management Portal to promote its staging environment to a production environment. Service releases have occurred approximately every six weeks since SXP launched in April 2010. The process on Windows Azure is so refined and so simple that the team no longer accounts for service releases when anticipating events that may cause downtime ...

We put this theory to the test earlier this week.  Most of our engineering and operations team were on vacation for the holidays when the recent ASP.NET security advisory was published - https://technet.microsoft.com/en-us/security/advisory/2659883  This seemed like a pretty significant issue but fortunately had a straight forward work-around that we chose to implement.  Engineering made the code changes, built the packages, and ran our test suite against the new services.

Operations was able to patch 3 services in 16 minutes without missing a single request.  This is another data point in the ease of running services on Windows Azure and one more reason we prefer to deploy on Windows Azure whenever possible.

One of the promises of Windows Azure and cloud computing in general is elasticity.  The ability to quickly and easily expand and contract computing resources based on demand.  A common example is that of retailers during the holiday season.  There is a huge spike of activity beginning on “Black Friday” and, traditionally, a lot of money is spent preparing for the peak load.

Social eXperience Platform (SXP) is a multi-tenant web service that powers community and conversations for many sites on microsoft.com.  A great example of one of our tenants is the Cloud Power web site.  SXP is responsible for providing the conversation content on the site while the remainder of the site is served from the standard microsoft.com infrastructure.  When the Cloud Power site sees an increase in traffic, SXP also sees an increase in traffic, which is exactly what happened in April.

In this case, the web traffic spikes were due to ads, which typically ran for a day or two.  Compared to March’s average daily traffic (represented by the 100% bar), SXP’s traffic spiked to over 700%.  Here’s a graph that shows 72 hours of traffic while an ad campaign was active.  The blue area is normal traffic and the red area is the additional traffic generated by the ads.  Since the ad was targeted to the US, there is a heavy US business hour bias to the traffic on both the soft launch as well as the full run.

There are lots of examples of traffic spikes like this taking web sites down or causing such slow response that they appear to be down.  Traditionally, the only way to handle such spikes was to over-purchase capacity.  The majority of the time, the capacity isn't needed, so it sits mostly idle, consuming electricity and generating cooling costs.  Windows Azure has a better approach.

The engineering team had a discussion with our business partners and learned that due to a couple of ad buys, SXP would see increased traffic on several days in April.  In advance of the ads running, we worked with the Microsoft.com operations team and decided to double our Windows Azure compute capacity to ensure that we could handle the load.

This is where Windows Azure really made things easy.  We went from 3 servers to 6 servers on our web tier within an hour of making the decision.  The total human time to accomplish this was a couple of minutes.  Our Ops lead changed one value in an XML file and Windows Azure took care of the rest.  Within half an hour, we validated via the logs that we had doubled our capacity and all web servers were taking traffic.

We didn’t have to allocate servers or VMs.  We didn’t have to install and patch an OS.  We didn’t have to install our application.  We didn’t have to run penetration tests.  We didn’t have to reconfigure the firewall or the load balancer.  All we did is modify a value in an XML file.  Since Windows Azure provides a REST API as well as Power Shell scripts, automating this process is straight forward. 

We monitored SXP closely during our first spike and determined that we didn’t need the additional capacity, so we turned the additional instances off.  Again, this took a couple of minutes to modify the XML file and about a half hour to take effect.

Our total, full-retail cost for the burst capacity was $70 plus about 5 minutes of operations time. 

Compared to the traditional models, Windows Azure made the process very fast and very simple.  Windows Azure compute has a minimum time commitment of one hour, so complex systems can burst their capacity multiple times per day if needed.  This is a very different model than traditional charge-back models for servers or VMs where you often get billed for a month or longer even though you don’t need the capacity. 

If you have the need for elasticity, Windows Azure offers a compelling solution at a very reasonable price point.

When we launched SXP just under a year ago, we had just under 25K requests per day.  Less than a year later, our peak load has grown by 40X.  Measured on an hourly basis, our peak load has grown 67X.

SXP broke 500K transactions in a day for the first time on March 21, so we have doubled our peak load over the last 20 days while being 100% available for 19 consecutive days. 

All performance numbers look good - our processor peaked at 20%, nothing queued, nothing hung - so we still have plenty of capacity for growth.

The Windows Azure Platform continues to deliver great performance, availability, and value.

One year ago this month we launched our first Windows Azure platform-based web service, the Social eXperience Platform (SXP). The Windows Azure Platform had just launched, so we were entering uncharted territory. Looking back over the last 12 months, it has been a great experience and the Windows Azure platform has allowed us to realize many of the promises of cloud computing.

Scale
When we launched, we were serving about 10K requests per day. We are now averaging over 450K requests per day with a peak of 687K requests in a single day. More importantly, we have the capability to deliver at least 2M requests per day without having to increase the number of instances.

Agility (part 1)
Windows Azure has made it easy to add functionality to SXP as the business has needed it. SXP has about 4X the lines of code today compared to version 1 which shipped just under a year ago. We have deployed 10 separate times in a year and haven’t incurred any downtime (measured at the transaction level) during any of our upgrades. With the exception of some learning curve issues, all of our deployments have gone very smoothly, even smoother than most of our on-premise deployments. This has given us the confidence necessary to deliver quick updates for the business.

Agility (part 2)
When we designed SXP, we designed it as a multi-tenant, cloud-based service. Our goal (hope!) was that other properties across Microsoft would adopt SXP. We like to talk about moving “capabilities” to the cloud in addition to moving applications to the cloud. SXP is an example of enabling capabilities via the cloud which enables other teams to build composite applications that leverage our service. We have grown from one tenant when we launched to over 35 tenants with over a dozen in the pipeline. Some of our tenants are using SXP in ways that we never dreamed about. The Windows Azure Platform has been a big enabler of this agility.

Elasticity
As we’ve on-boarded new partners, we’ve had 3 separate instances where traffic “doubled overnight”. The Windows Azure Platform allowed us to scale up in anticipation of the increased traffic and to scale back days or even hours later. So far, we’ve had the headroom to handle the increased load, but it’s nice to know that we can very easily add and remove capacity as load increases.
With Windows Azure compute, we were able to double our web service tier capacity in a matter of minutes. Once we were confident we could handle the increased load with fewer web instances, we scaled back to the correct level. It only took minutes to scale back down and the full retail price of the additional capacity was under $100. Compare this to having to order servers or even provision VMs and the increased agility and decreased cost for the elasticity is dramatic.

Performance
SXP’s performance continues to be excellent, with average server response times below 100 ms. We have a separate service that monitors SXP from within the same data center. The service sends a synthetic transaction to SXP every 5 seconds and records the availability and performance of the service. Since we are within the same data center, there are no networking issues involved. Our average ping response for the year is just above 10 ms.
SXP is a global service, but there is some US business hour bias to our load. We notice a slight increase in ping response times during prime US business hours, but the average is still around 12 ms. While this is 20% longer, it is only a 2ms delta, which seems reasonable given the increased load.

Availability
This has been one of the best surprises from our move to Azure. We measure availability at the transaction level and since going live, we have delivered over 4 9s of availability at 99.9957%. To put things in perspective, we’ve had 2,543 errors out of 59M requests. Of those errors, 1,030 occurred during a 1 hour window and over half occurred during a 2 hour window. Neither of those two hours of major downtime were Azure-specific issues.
The Azure team really deserves a lot of kudos for this as Azure is a v1 product that only shipped a few weeks before we went live. They are solving some pretty tough problems, so I was willing to cut them some slack early on. Fortunately, they over-delivered. Now we’re pushing them to help us deliver 5 and 6 9s availability.

Cost
This has been another pleasant surprise. We have been able to significantly reduce our monthly hosting costs by moving to the Windows Azure platform. To be fair, it’s not an exact apples-to-apples comparison, but, on the other hand, SXP delivers far more than our previous solution, so here are the details:
Previously, we used a 3rd party solution to provide comments and ratings for our Showcase site (www.microsoft.com/showcase). Since the solution had been customized for Showcase, our monthly hosting costs were $15K. Because of the customizations, we weren’t able to [easily] leverage the solution across the other microsoft.com sites and also weren’t able to upgrade to the latest version of the software, which drove our support costs up. Basically, we had an expensive application and we wanted a scalable platform.
SXP, our Windows Azure based solution provides increased functionality and we now have 35 sites (tenants) live. Because we were building a service for use across microsoft.com, we built a multi-tenant service from the ground up. This gave us a distinct operational advantage over the 3rd party solution as we are able to on-board new tenants in minutes rather than weeks. This also gives us a huge cost advantage as we’re able to spread the operations costs across multiple tenants.
Our monthly cost for the entire SXP production system using full retail pricing is $1,473, which is a 90% savings over our on-premise solution. Using standard commitment based pricing, that cost drops to $924.19 per month. January through March, Showcase generated 32% of the SXP traffic. Allocating 32% of the costs to Showcase is $296, which is a 98% savings over our old solution. If we divide the cost by 35, each tenant’s cost is $26.41 per month. We’re confident we can drive the monthly per tenant operations cost below $20 and perhaps below $10 or even $5.
At the same time, we increased our availability and greatly increased our agility in responding to business needs. Here is our before and after picture:

While everything hasn’t been perfect over our first year of operations, we are very pleased with our Windows Azure platform experience as evidenced by the fact that we are aggressively moving additional workloads to the platform.

Our video is here: https://technet.microsoft.com/en-us/library/gg575755.aspx

We talk about the real-world lessons learned from running a high-volume web service on Windows Azure as part of Microsoft.com.

Our application is a stateless, RESTful web service that is used by several Microsoft.com web properties.  In designing the service, we chose not to cache any data so each request is a round trip to one or more SQL Azure databases.  We measure uptime at the transaction level, so if the web server or SQL Azure database is unavailable, we report that as an error.

In Q1, we served 7.76M requests with an availability of 99.995%.  Our September availability was 5 9s.

Again, we chose not to write any special code to improve uptime.  This is generic IIS / ADO.NET code (with the SQL Retry logic from here).

I find it amazing that a general purpose platform can deliver these kinds of results at commodity pricing levels.  To achieve over 3 9s availability is generally very expensive and often requires some coding changes to achieve.  Windows Azure and SQL Azure make it easy.

We are releasing a new version of SXP in the next couple of weeks with some cool new features and are already planning version 3 which moves even more capabilities to Azure.  SXP is in.

Case study is here: https://technet.microsoft.com/en-us/library/ff973966.aspx

Changes
The first change I made was to do my math correctly in the sleep computation ...

The second change I made was to add a SqlConnection.ClearPool(conn); call to the retry exception handler.  This should ensure that the other connections in the pool don't have to retry.  We tend to see our retries in "batches", so hopefully this is the line of code to fix that.  You could use ClearAllPools() instead, but since we have a database per tenant, clearing the specific pool is a better choice for us.

The third change was to move the conn.Close() and SqlConnection.ClearPool(conn) outside of the if block so that they get executed every time.

The fourth change was to add one final Sleep(sqlRetrySleep) call in the exception handler.  This and the ClearPool call gives us our best chance to succeed when we return.

The last thing I did was to use Actipro CodeHighlighter to make the code in the blog post more readable.

Thanks to everyone for the feedback.  Here's the latest version of the code: 

public string GetSqlContextInfo(SqlConnection conn)
{
    string sqlContext = string.Empty;

    // Omitted the code to read these from configuration
    int sqlMaxRetries = 4;
    int sqlRetrySleep = 100;
    int sqlMaxSleep = 5000;
    int sqlMinSleep = 10;

    // start a timer
    TimeSpan ts;
    DateTime dt = DateTime.UtcNow;

    for (int retryCount = 1; retryCount <= sqlMaxRetries; retryCount++)
    {
        try
        {
            conn.Open();

            // get the SQL Context and validate the connection is still valid
            using (SqlCommand cmd = new SqlCommand("SELECT CONVERT(NVARCHAR(36), CONTEXT_INFO())", conn))
            {
                sqlContext = cmd.ExecuteScalar().ToString();
            }

            ts = DateTime.UtcNow - dt;

            // log opens that take too long
            if (ts.TotalMilliseconds >= 75)
            {
                SxpLog.WriteKnownEvent(8001, ts.TotalMilliseconds, "Connect", conn.DataSource, conn.Database, conn.WorkstationId, sqlContext);
            }

            break;
        }

        catch (SqlException ex)
        {
            conn.Close();
            if (retryCount == 1)
            {
                SqlConnection.ClearPool(conn);
            }

            if (retryCount < sqlMaxRetries)
            {
                SxpLog.WriteSqlRetry(5902, ex, conn.DataSource, conn.Database, conn.WorkstationId, "Connect", retryCount);

                // don't sleep on the first retry
                // Most SQL Azure retries work on the first retry with no sleep
                if (retryCount > 1)
                {
                    // wait longer between each retry
                    int sleep = retryCount * retryCount * sqlRetrySleep;

                    // limit to the min and max retry values
                    if (sleep > sqlMaxSleep)
                    {
                        sleep = sqlMaxSleep;
                    }
                    else if (sleep < sqlMinSleep)
                    {
                        sleep = sqlMinSleep;
                    }

                    // sleep
                    System.Threading.Thread.Sleep(sleep);
                }
            }
            else
            {
                // Log the exception
                SxpLog.WriteSqlException(ex, conn.DataSource, conn.Database, conn.WorkstationId, "Connect", retryCount, "Final");

                // we thought about rethrowing the exception, but chose not to
                // this will give us one more chance to execute the request - we might get lucky ...

                // sleep
                System.Threading.Thread.Sleep(sqlRetrySleep);
                conn.Open();
            }
        }
    }

    // we log this value and null might cause an issue
    if (sqlContext == null)
    {
        sqlContext = string.Empty;
    }

    return sqlContext;
}