Affinity Groups

If different components of your Azure solution is deployed to different Azure data centers (DC) there’s a big chance that you will have to deal with additional network related issues.

Imagine a scenario where your web front tier is deployed in an Asia DC, web services tier in a Europe DC and database in a US DC. Doesn’t sound logical, right?

For optimal performance it would be best to deploy as many solution components as possible in the same data center.

You can check Importance of Affinity Groups in Windows Azure  for more information. (Thanks to Holger Sirtl for reminding this topic to me)

Multi tenancy of Windows Azure SQL Database (formerly known as SQL Azure)

Windows Azure SQL Database is a self managed service starting as low as $5 (as of August 2012)

In certain scenarios Azure SQL Database users can face performance issues due to the multi tenanted nature of the service.

The only way to make a SQL Database service available at a very low cost is to make sure multiple customers (tenants) share the same hardware resources together.

As of August 2012 the cost of Azure SQL DB is determined “only” by the size of the DB. The smaller the size, the less you pay. And naturally the more tenants you share the hardware with. Even if you have a 50 MB small DB that needs to process a lot of transactions, as of today there is not a way to “pay more” and “get more” hardware resources.

For these types of scenarios you can choose to install our good old SQL Server on a persistent Azure VM (aka Infrastructure as a Service or simply IaaS)

With SQL Server on IaaS we can get dedicated VMs for our DB, no matter what the size of the DB is and utilize all the hardware resources dedicated to our VM.

As you can easily guess there are some disadvantages of this approach. Firstly you have to pay more since you hold dedicated hardware resources for your DB. And also you have to manage the SQL Server yourself which will also increase the cost of maintenance.

For more information you can check SQL Server in Windows Azure Virtual Machine Early Adoption Cook Book

Note: Persistent VM feature is published on June 2012 and will probably stay as a preview service for a few more months to come. You can use it for production purposes but Microsoft will not give you SLA until the service is made generally available (GA)

Update: If you need more disk IO performance on an IaaS VM you can use multiple network attached disks to boost performance. You can do disk striping (RAID 0) and since Azure network drives are fault tolerant by default you get both better performance and reliability. Also please remember to use different storage accounts for each disk to further boost performance. Please check this 10 minute for more information, https://channel9.msdn.com/Shows/TechNet+Radio/TechNet-Radio-IT-Time--Part-5-Real-World-Azure--Provisioning-Storage-for-IO-Intensive-Applications-o

Update: For higher SQL Server disk IO performance using multiple file groups are preferred instead of disk striping. Please check http://msdn.microsoft.com/en-us/library/windowsazure/dn133149.aspx for more info.

Update: If you are designing massive scale applications you might find the following quite interesting to read
Best Practices for the Design of Large-Scale Services on Windows Azure Cloud: http://msdn.microsoft.com/en-us/library/windowsazure/jj717232.aspx
Designing and Deploying Internet-Scale Services: www.usenix.org/events/lisa07/tech/full_papers/hamilton/hamilton.pdf  (beware, it's a bit heavy in theory and not an easy read)

Latency Issues

Also known as ping time, latency is the round trip network time between the client and the server. (Azure web roles by default doesn't support ping for security reasons. But it would be quite easy to measure the latency without adding ping support. Please check the IE developer toolbar section at the bottom.)

Latency, locally in a country can be around 25 msec (milliseconds).

Latency, to a global cloud provider’s closest data centre can range from 25 (if you are quite close geographically) to 200 msec (if you are in a remote country like Australia). For most countries we can assume latency will range from 25 to 100 msec.

This means that if we are in Australia and we have a solution that is deployed to a global cloud vendor, the latency can be 8 times higher compared to a local hoster.

Now, does this mean our solutions overall performance will be 8 times less? Not really. It might actually be much more faster, depending on the solution itself.

Let's examine some scenarios; I will take a Turkish company, ACME as an example where the latency to closest Azure data centre is around 100 msec. (I deliberately choose a country that doesn’t have the best or worst latency)

Scenario 1 - ACME host their solution on a local hoster. A web page in their solution does some heavy lifting and takes 10 seconds to complete. When they have more users this can take up to 20 seconds. It's not easy to add more servers on demand with their local hoster so they are facing performance issues from time to time. When a user opens up this page on a busy period, it can take up to 20,025 msec to show up. 20,000 msec (20 seconds) for the page to generate and 25 msec to transfer.

Then ACME decides to move their solution to Azure. They implement autoscaling so they can add more capacity under heavy load. With the additional capacity we can guarantee that the page generates in 10 seconds, most of the time. Unlike the previous 20,025 msec case, even under heavy load the same page is served in 10,100 msec (10 sec page generation + 100 msec transfer time).

When we have less load Azure would perform nearly the same as local hosted solution. 10,025 vs. 10,100 msec. The difference is 75 msec which is less than one tenth of a second. It would be nearly impossible for human eye to realise the difference for a page that takes 10 seconds to generate. However under heavy load Azure would performance nearly 2 times faster, 20,025 vs 10,100 msec.

The bottom line for this scenario would be, Azure solution would perform either the same or better compared to the local hosted solution. So we have no issues on this one. 

Scenario 2 – ACME has a web page that only takes 500 msec to execute. The math is simple, local hosted solution takes 525 msec for user to see the page vs. 600 msec on Azure. The difference is still 75 msec, again 1/10 of a second but this time a very careful eye might realise a little bit of a difference.

The bottom line, Azure performs nearly the same as local hosted version and the difference is negligible.

Scenario 3 - ACME has a web page that again takes 500 msec to execute, but this time the page has some AJAX features. There is a table in the page that has 50 rows and 10 columns. And for each cell a javascript call is being made to the server for a lookup which only executes in 5 msec.

In this scenario the local hosted page would take 500 (page execution) + 25 (latency) + 500 (number of cells) x [ 5 (lookup execution) + 25 (latency) ] = 525 + 500 x 30 = 15,525 which is roughly 15 seconds.

Now let's calculate the Azure hosted version. 500 (page execution) + 100 (latency) + 500 (number of cells) x [ 5 (lookup execution) + 100 (latency) ] = 600 + 500 x 105 = 53,100 This time it is 53 seconds, nearly a minute.. Quite slow compared to the local hosted 15 seconds page eh?

Although this scenario is a bit exaggerated, performance issues due to wrong AJAX usage is one of the most common issues for cloud solutions.

Solution: As you can easily guess the solution in the above scenario would be to simply optimize the page to make one AJAX call to get all the data needed in the table, instead of making a call for each cell.

With this optimization the Azure version could take 500 + 100 + 5 (lookup) + 100 (a single AJAX call) = 705 msec. Less than a second, more than 15 times faster compared to the unoptimized local hosted version!

Scenario 4 - ACME has a web page built in ASP.Net and they use update panel and many other ASP.Net AJAX controls. ACME loves the way they built it, simply dragging and dropping controls and all the "magic" happens behind the scenes.

ACME feels the local hosted version works "just fine" but when they move to Azure, they "feel" that the page is just "slower". It was hard for ACME to highlight which part is slow since ASP.net AJAX controls does the magic to continuously load "stuff" and they don't really know what happens when.

Solution: Unfortunately tackling issues like this one is a bit trickier compared to the one on scenario 3. (And Murphy is never wrong, we face more of this type) So what do we do?

Solution option 1: Similar to the previous scenario, we still need to optimize the page to make fewer javascript calls. In this case your best friend would be tools like Internet Explorer developer tools (opens with F12) or similar add-ons for other browsers. (Please check the bottom of this post for a 10 second crash course to detect latency issues, number of calls etc.) I have to warn you though, this might not be very easy since you might detect many javascript calls that are simply auto generated for you by the AJAX framework. My suggeston would be to replace AJAX controls with your own javascript calls if you can't optimize the number of calls. I would also strongly suggest using a framework like jQuery to ease this task instead of starting from scratch.

Solution option 2: Simply use ASP.Net MVC (+ jQuery). The paradigm is quite different and unlike classic ASP.Net, MVC developers has much more tighter control on the actual html generated and most javascript calls being made. jQuery is embedded to your project by default and it is quickly becoming the de facto standard for UI coding. The down side is that, this solution might apply only for new projects and it might not be feasible to refactor your existing app towards MVC. It might actually be easier to rewrite the UI from scratch. Not applicable for most cases.

Scale Up issues

All cloud vendors including Azure has a hard upper limit for scaling up. As an example (and as of May 2012) a single Azure compute instance (VM) has the upper limit of 8 CPU cores and 14 GB Ram. This is as much as you can scale up, today.

Solution: It's quite easy to guess, right? Your application must be designed to scale out, not up. Regarding compute, you should simply be able to add more instances to your deployment. And the good news is, most Azure services are designed with auto scale out features out of box. For instance Azure storage (BLOB, Table, Queue) all scale out to more and more VMs behind the scenes if needed.

Relational databases are a bit tricky in this regard since it’s not very easy to scale them out. SQL Azure scales up automatically to a stage but as of May 2012 it can't really compare to the 32 core physical box that you can buy for your on premise data centre. The solution can be partitioning the DB to scale out (never that easy) using multiple DBs (never easy either) or using non-relational stores like Azure BLOBs and tables (which is also not very easy if you really need relations)

Scaling out relational databases is beyond the scope of this blog but after a quick search you should be able to find plenty of resources on this topic.

Bandwidth Issues

Nowadays bandwidth capacity is quite high for most countries. Afterall more and more users want to watch online videos and internet is simply replacing TV.  However we might still face bandwidth issues because, a certain country might have permanent or temporary international bandwidth issues or the amount of content we deliver is simply massive and downloading it takes more than the desired time. (ex: to broadcast a full HD video in realtime)

Solution: We might consider using Azure CDN (Content Delivery Network) or a CDN from another vendor.

CDNs are distributed servers worldwide and used to cache content so that the clients receive them from a node that is closer from a “network topology” point of view.

So what is a network topology? Closer geographic distance doesn’t always mean better bandwidth or lower latency. Two cities (from different countries) might be 100 kilometers far away from each other. One of them might have a CDN node serving the whole country and the other city might have very poor connection to the neighbour country. This can easily mean users from the second city will automatically skip downloading content from the 100 km away CDN node and will choose to download it directly from the main datacentre or from a CDN node that is 500 kilometers away.

Optimizing your solution for CDN usage and understanding internet network topology are again beyond the scope of this blog post.

Compute Instance (VM) Performance

The bigger your instance is, the more bandwidth it has (on top of CPU cores and memory). Please check the below page for more information.

http://msdn.microsoft.com/en-us/library/windowsazure/ee814754.aspx

In certain cases increasing your VM size alone can boost the performance quickly to a desirable point. Still it would be best to collect some performance counters to see what is going on behind the scenes. For that we have;

Azure Diagnostics

For many performance issues if we can’t measure the current state then we can’t really optimize. Simple as that. Thankfully Azure has an extensive set of tools and performance counters for this purpose and they are quite easy to use.

You can check the below page for more information on Azure diagnostics.

http://msdn.microsoft.com/en-us/library/windowsazure/hh411552.aspx

Internet Explorer Developer Tools

Pressing F12 in IE opens up developer tools that have plenty of goodies. The last tab, network, can be your first stop to detect many Azure performance issues (especially latency).

A very easy way to test your latency would be deploying one liner HelloWorld.html and then in the developer toolbar go to Network tab and click “Start Capturing”. Then you can hit F5 to refresh the page and see how long it takes to load. Since helloworld.html is extremely simple I would assume most of what you see (80% ?) would be network related and therefore latency.

For more complex pages, network tab will give you a detailed view on your pages performance including javascript calls. Then you can focus on certain parts of the page that takes up the majority of the time. This can be static content that you load from Azure, so that you would know the optimization you need is more on server side. Or you can figure out the problem is that you are making too many javascript calls, so the optimization would be on UI level.

Happy Azuring..

UPDATE: I posted another blog where I talk about a few other performance related topics. You can check it from Tuning Windows Azure Performance, more thoughts

http://www.microsoft.com/download/en/details.aspx?id=29209 

It's worth noting that for most of the usage scenarios you don't really need an endpoint protection software for Azure. Yes, your web and worker roles are actually VMs but all of the components on those VMs are secured by Microsoft. Except your own software, and that might not really need MEP. However there are cases where you have to install 3rd party components or have users (admins) use the VMs remotely via RDP and MEP in these scenarios might be quite usefull. 

One common complaint I heard was the lack of metadata capabilities and this will be resolved in SQL Server 2012. I have never been a fan extensive metadata use in BI though. With other tools my experience was always that the time spent to fix the metadata to reflect underlying data changes is more than to fix the report itself. It's still good to support both methodologies and leave the decision to the users.

For Gartner's report please visit http://www.gartner.com/technology/reprints.do?id=1-196VVFJ&ct=120207&st=sb

http://www.windowsazure.com/en-us/pricing/details/#database

History almost always proved that increase in productivity creates more jobs and helps grow the economy. So what LSE says makes a lot of sense to me.

For more information please check http://cloudtimes.org/london-school-of-economics-study-says-cloud-computing-could-be-key-to-global-economic-growth/

BizSpark Plus, an extension of the BizSpark program, provides higher value offers to startup accelerators around the globe. How cool is that?

For more information please visit http://blogs.msdn.com/b/windowsazure/archive/2012/01/30/microsoft-bizspark-to-offer-startups-60-000-in-cloud-services-on-windows-azure.aspx

If you ever tried using this type of forms you’ll realize that you will have some limitations compared to the forms that run inside the InfoPath client.

One technique to come over these limitations would be using Web services in your forms. For instance you can query a database if you have a web services layer encasulating the database server. Below are some hints to make your life easier.

1. It is quite easy to setup a new data source that is accessing to a web service in InfoPath. You can develop and using your InfoPath client and click Preview to test it.
2. You can use rules that are tied to buttons or form start & close rules (Tools – Form Options – Open and Save – Rules) to set Web service parameters, execute and assign the result to you fields.
3. For your forms to be used they have to be either added as a content type or the connection definition in InfoPath has to be stored in a connection library
4. To add your form as a content type navigate to manage forms in Central Administration. Here you can add your form and enable it for your site. To use a connection library click the convert button on InfoPath while you have the data connections dialog box open. Don't forget that you have to do either of these to have a web enabled InfoPath form to call web services.
5. Now you can create a forms library where the default form is your form which is calling web services.
6. In one customer we had some errors executing the web service and figured out that the reason was anonymous access being enabled using Central Administration and disabling it solved our problem.

Testing Methodology

I used the OOB load test that comes with the open source project "SharePoint 2007 Test Data Population Tool" (http://www.codeplex.com/sptdatapop) to first test the environment without the huge documet usage. And then in parallel I wrote another test that simply reads/writes 100 MB documents in SharePoint and monitored the performance counters on all of the servers. The second test used around %1 of the total users in the first test (and this was our customers guess about the huge file usage)

Findings

• The average request/sec dropped around %18
• The average response time was quite similar (since less pages were processed by our test rig)
• CPU usage on all of the servers were quite similar except a slight increase in the front-end servers
• Available memory was again quite similar (slight decreases in the front end servers)
• SQL Server disk I/O (reads, writes and disk queue length) had some significant spikes and the averages had some significant (and for our scenario acceptable) differences
• Network interfaces in the front-end servers had some significant difference

These results might not be the same in your environment and I strongly suggest performing similar tests before coming to a conclusion. In our case the current infrastructure was quite capable of handling huge files but if we wanted to further increase the performance I'd personally start looking at the SQL physical discs first.

Oh by the way please also consider your overall disk capacity before allowing huge files.. Since SharePoint does not work like simple network shares you need to be very careful on disk capacity planning.