Let me introduce some of these counters when I explain several “visualized stories”. My motivation is to share and evangelize the methodology more than the data itself. As Azure invests more in Service Bus technology, some of the exploring work results might become useless, but the thinking and implementation approach will remain invaluable. 

Have all published events been sent to Service Bus successfully?

“\Service Bus Monitor\SendToLocal: # of Passed / sec” tracks number of events per second that have been successfully sent to Service Bus.

If we know all the incoming traffic are routed to local data center Service Bus, we should expect to see the chart of this perf counter (in yellow) completely matches to the chart of “\ASP.NET Application(_Total_)Requests/Sec” (in black).

The difference between the two charts is from “\Service Bus Monitor\SendToLocal: # of Failed / sec”, roughly speaking.

This is an easy way to check, within a given time frame, if there is any (local) Service Bus failure at its ingress interface, and how much impact it has had.

How to track Service Bus failure?

“SendToLocal: # of Failed / sec” logs count of failures and timestamp for local SB instance errors (could be caused by SB internal server busy state). This count should match “SendToRemote: # of Passed / sec”, according to current routing principle. If we see “SendToBoth: # Failed / sec” has more than “0”, it clearly indicates message(s) failed to send to both local and remote SB instances, which tells us the message(s) were lost in transferring to SB cluster. This will be a critical error. We may quickly check these graphs before we drill into event log for error details.

We are capturing “SendToLocal: # of Consecutive Failures” to see how consistently it behaves when the local SB instance fails to respond properly at its ingress. The monitoring results tell us this symptom repeats several times a day, but the failure went away quickly, like “service bus glitch”, I am continuing to track this number (max is ‘2’ by far), and see if it is possible to figure out a reasonable threshold for us to design an algorithm to reroute partial or all traffic under specific condition.

Latency comparison between local and remote data center

We need to design a Geo-distributed messaging system for high performance and low chance of failure. We set up two Service Bus instances
in two data centers.

Is sending data to the SB instance at local data center always better performed than sending it to the remote SB instance? What is latency trend to local/remote look like over time? At which point, we should consider rerouting traffic in the case of SB instance failure? Etc.

With these questions in mind, I added “SendToLocal: Latency” and “SendToRemote: Latency”. Monitoring threads are instrumented to send test message to the two SB instances frequently and gather their roundtrip time measurement as latency result.

This is one example trending graph. The red line represents Latency (ms) trend to remote SB. The purple line shows Latency (ms) trend to local SB. (The scale is not the same for the two lines from the tool I am using, by the way.)

This is what I have learned so far.

• Most of the time, latency to local is x% of latency to remote. This is what expected considering the basic network latency by geo location difference.

• Quite often, latency to local spiked and significantly “prevailed” latency to remote. This is interesting but “annoying”, as it adds mystery factor to the “black box” (SB). We know the incoming traffic prefers us to always route it to the local SB. We also know at some point of time it likes us to change the route temporarily due to SB internal state change, i.e. low/normal load to over load or short of broker resource.

•  Latency increase may be a leading factor for SB connection failure. This is a hypothesis. It follows common sense, but I have not got any concrete proof. The more I look at the data and their comparison, the less confidence I have on this statement. We will need to collect more data, or improve the approach we collect data. But I am sure probability and statistics should play a big role in this study. This is similar as how we are trying to measure, monitor, and “forecast” earthquake today.

Relation between latency and failure of SB

This is an example graph of getting latency counters and failure count trends together. The goal is finding a possible pattern that reveals failure(s) versus latency changes specific to SB instance. This is a continuing data study process. No conclusion to share at this moment.

(End) 

1. How do we enable anonymous users to send data to our system?

There are basically two ways to access Azure Service Bus, RESTful API or SBMP (Service Bus Messaging Protocol), both of which require secured connection (by connection string, or certificate to create token). Letting users to access our Service Bus directly doesn't sound a realistic solution.

2. How to distribute data interface to geographical locations to serve users at anywhere in US?

Azure Service Bus are configured by data center location, the same as Azure WebRole instances. It is important to assemble the two components together by location in order to follow principle of the lowest network latency and the highest service resilience. 

etc.

Here is a high level architecture design.

Since most users are on internet, web interface is designed to accept data published by users. We implement a web Api service that exposes data publishing "endpoint". It is by HTTP connection with SSL cert authentication to guarantee a secure server access to users. However, asking every user to be authorized before he/she could send data is a little bit challenging here.

Considering that users could be from anywhere in US, the web Api service is deployed to two data centers (i.e. US west, US east). To accommodate geo-distributed design requirement, we've prepared for two Service Buses, one in each data center (USW and USE).

All the WebRole instances have connections to the Service Bus instances (SBMP), but their primary data transferring target is the Service Bus instance in local data center. This means data received by webRole instance in US west data center is sent to Service Bus instance in the same US west data center. However, in case the "send" action fails due to local Service Bus unavailability issues, the same data is resent to the service bus in remote data center (US west data center in this example.) This will minimize risk of possible data loss during the transfer process.

All the published data is organized by topic name. We manage a list of TopicClients, each of which "maps" to a specific topic name on a specific Service Bus instance. Therefore, we can use SubscriptionClient to gather all data from the other end of two Service Bus instances, and filter data by topic name.

I will continue to discuss on this topic in the next post. Please stay tuned.

Recently I shared a story about one day (or night) I was suddenly getting 80 GB topic size. I thought it was an accidental discount by operational glitch, but it happened again last night. I have learned the "root cause" from an expert. Thanks to the recent released partitioning feature for Service Bus Topic, it "boosts up" Topic size dramatically. If it is turned on, when you create a new Topic, you will get 16 times the max size you are requesting. So, if you want to configure 5 GB with partition on (set "true"), you get 80 GB. I tried it with 3 GB plus partition equal true, and I've got 48 GB max size in return, as expected. Very cool!

Here is a code snippet just for reference.

        private static void CreateEventTopic(string connectionString, string topicName, long topicMaxSizeInMB, bool partitionEnabled)
        {
            if (string.IsNullOrEmpty(topicName))
            {
                throw new Exception("Error: Topic name is needed!");
            }

            NamespaceManager namespaceManager = NamespaceManager.CreateFromConnectionString(connectionString);
            // this method (TopicExists) is NOT case sensitive for topicName string
            if (!namespaceManager.TopicExists(topicName))
            {
                namespaceManager.CreateTopic(
                    new TopicDescription(topicName)
                    {
                        DefaultMessageTimeToLive = TimeSpan.FromMilliseconds(60 * 1000),
                        EnableBatchedOperations = true,
                        DuplicateDetectionHistoryTimeWindow = TimeSpan.FromMilliseconds(60 * 1000),
                        RequiresDuplicateDetection = true,
                        MaxSizeInMegabytes = topicMaxSizeInMB,
                        EnablePartitioning = partitionEnabled
                    });
            }
        }

With 80 GB (per topic) on hand, I feel much more comfortable to do more things with the toy. The partitioning feature looks promising in terms of messaging handling for both send and receive. You may find detailed description from https://msdn.microsoft.com/library/azure/dn520246.aspx.  With excitement and hope, I have enabled this feature for all topics. We will try and see how it is working in practice.

This link has an intuitive resource for beginners to start with: https://www.windowsazure.com/en-us/documentation/articles/service-bus-dotnet-how-to-use-topics-subscriptions/. However, it is very soon that you will find its capacity planning becomes important at the architecture design stage.  

There are a lot of numbers mentioned on this page (https://msdn.microsoft.com/en-us/library/windowsazure/ee732538.aspx). It took me quite some time to study each quota number. As an example of using Topic/Subscription feature, I'd suggest focusing on the following six numbers as a start:

1. Topic size: 1 GB or more

2. Number of topics per service namespace:  10,000

3. Number of subscriptions per topic: 2,000

4. Message size for a topic/subscription entity: max at 256KB

5. Number of concurrent connections on a topic/subscription entity: 100

6. Number of concurrent receive requests on a topic/subscription entity:  5,000

And, regarding 5 and 6, this article has a decent explanation: https://blogs.msdn.com/b/mvpawardprogram/archive/2013/11/18/windows-azure-service-bus-connection-quotas.aspx

Service Bus used to be offering 1GB up to 5 GB spaces until just recently (a week ago or so), when the portal UI "suddenly" decided to offer 1 GB size of space to every topic you set up either manually or programmatically. This is what I have experienced lately, but it has not been validated by official feature announcement (is there any, yet?).

I guess it was during the feature change transition period of time in production. One of my topic has got "80 GB" size!

See this picture as a sort of proof or "certificate of achievement" (I have masked some strings related to the project.)

It has made me feel so special. I love it, although I have not verified if it does give me that huge space?

Even when I was confused about the increasing usage size in "negative" direction (see below screenshot), I was still cool about it.

If you suspect whether this was just a one out of million cases, here is another "proof of UI concept".

Azure Service Bus is continuing to make improvement. When we look back at these pictures after a year or two, I am sure we will be really thrilled about the progress.

Let's enjoy every moment (good or not good) now, when we explore its adventurous features.

I have a simple WebRole cloud project. It had been building fine, until when I merged it to a different branch. The same package build process produced this error message box.

And, in the output window, it only has one line of error: "...>Error: Object reference not set to an instance of an object." The error message was really terse, I had no idea what was actually going on.

I changed the output option to "Diagnostics" or "Detailed", but it still only gave the same one line error message.

Having searched online resources without a luck to find any meaningful clue, I tried removing the web project reference from the cloud project. It didn't even allow me to do so. Here is the pop-up message box.

I know there is no single issue with the .csdef or .cscfg files, according to what the message indicated. I had to remove it manually from the project file. The "brutal" way got accepted.

During the process of experimentation, quite a lot of "annoying" things happened. The diagnostics.wadcfg file was gone, the important settings data inside .cscfg files were mysteriously lost, and etc.

When I added the same web project back to the cloud project, it popped up another "Windows Azure Tools for Microsoft Visual Studio" related message.

This time the message is very helpful. At least I got hint of the direction. Now I realized that I used to have two versions of "System.Spatial" (5.2 and 5.6) in the dependency package.

I thought ver. 5.2 was old, so I removed it from the package and kept the latest version (5.6) for "System.Spatial". The "WindowsAzure.Storage" ver. 2.1 is actually referencing "System.Spatial" ver. 5.2.

After I recovered the "System.Spatial" ver. 5.2 back in the package, the problem was solved!

The unexpected blocking issue was resolved. I am wishing any useful error information (like something above in the last dialogue box) would be included in the output window during the build process. This will save us tremendous time and effort.

Application: WaIISHost.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.TypeInitializationException
Stack:  at Microsoft.WindowsAzure.ServiceRuntime.Implementation.Loader.RoleRuntimeBridge.b__0()
at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
at System.Threading.ThreadHelper.ThreadStart()

It seemed clear the WindowsAzure.ServiceRuntime ran into critical error when it tried to bring up the application service it hosted. But, the error message is not intuitive enough to find the root cause. Another weird behavior is, the cloud package built from the same project on the other enlistment machine was just doing fine. There must have some significant difference between the two packages (.cspkg files) built from two enlistments. I had no luck in finding any tool online to help compare the two package files. Looking through the build logs and all the files inside the package didn’t surface any clue.

After having spent some decent time investigating inside the VM, I finally found a log file named “WaIISHost.log” Under this folder: “C:\Resources\Directory\625f7be7a6b241dfaa6924e3e6b8841e.xxxxxxxx.Service.DiagnosticStore”. A “surprising” detail of the exception error is clearly written in this file. It is something like this (Concerning the project confidentiality, I am using “xxx…” here to represent the dll names.)

WaIISHost Information: 0 : [00003952:00000001, 2014/02/14 18:05:29.433, INFO ] DebuggerAttachComplete START
WaIISHost Information: 0 : [00003952:00000007, 2014/02/14 18:05:30.676, ERROR] Unhandled exception: IsTerminating ‘True’, Message

‘System.TypeInitializationException: The type initializer for ‘xxxxxxxx’ threw an exception. —> System.IO.FileNotFoundException: Could not load file or assembly ‘Microsoft.WindowsAzure.ServiceRuntime, Version=2.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35′ or one of its dependencies. The system cannot find the file specified.
at xxxxxxxx..cctor()
— End of inner exception stack trace —
............

The log information is very helpful. The stack trace has enabled me to identify the root cause immediately. It turns out that a dependency dll was still referencing an older version of Azure SDK (ver. 2.0). The root of the root cause was actually from an unexpected time delay of a “Gated Check-in” feature implemented in VSTS recently. This is why the issue was seen from one enlistment, but not from the other(s). And, this issue may not be easily reproduced at local dev box, if multiple versions of azure runtime assemblies are installed on the dev box.  

If I started to search for “WaIISHost Log”, I would have quickly found this early post (https://blogs.msdn.com/b/kwill/archive/2011/05/05/windows-azure-role-architecture.aspx). My “painful” experience has motivated me to share the tip of using “WaIISHost.log” with you for effectively troubleshooting cloud service errors during its start up.

(This article was originally written on Feb. 15, 2014)