I wanted to be able to read and write a particular set of resources, in my sample case this was an Azure blob as well as some associated SQL Database records, from one or more applications, machines, or devices such that only one consumer thread would be allowed to modify these resource at any one time.  For the sake of this article, think of those resources as an atomic unit that should only ever be modified together and not individually, nor by more than one thread at a time.  If I were solving this problem within a single app domain, I could use a simple critical section using the Monitor class, or the lock keyword.  If I were solving this problem within a single machine (OS instance), I could then upgrade to the Mutex class while agreeing upon a shared mutex name for synchronization.  One of the beautiful aspects of these synchronization mechanisms is that they provide near instant notification once the mutual exclusion or lock becomes available for another thread to acquire; this facilitates low latency, event-based collaboration between threads.  They also implement a queued access model whereby access to the shared resource is granted on a first-come, first-serve basis.

These OS-level techniques break down once I introduce the requirement to synchronize in the same way across devices.  While I recognize that there are many ways to use other technologies as surrogates for these locking techniques when dealing with multiple devices, I sought to find one that looked and smelled as similar to .NET's Mutex class as possible.  After some hours of searching, I found nothing I quite liked, so I've invented one that leverages Windows Azure Service Bus on the backend.  The remainder of this article details a very early version of a class I call CloudMutex that implements the WaitOne and ReleaseMutex methods familiar to consumers of .NET's System.Threading.Mutex class.  Please note that this is a very preliminary attempt and I'm hoping for great feedback from my peers before investing too much.  If a ready-made solution already exists, I'd love to see that as well.

This article assumes that you have a basic understanding of Windows Azure Service Bus Queues along with solid C# fundamentals including thread synchronization mechanisms.  For lack of that, please do some preparatory learning before going much further.  Also note that hereafter, I'll just use the word "device" to refer to any OS instances, whether physical or virtual, on any operating system, anywhere with access to the Internet.  Further, for those so inclined to want a more detailed definition of a mutex, think of it as a binary semaphore with thread affinity; that is, only one thread can own it at a time.

Starting from the end

The first most salient unit test that I wrote sought to show that thread synchronization within a single process worked the same way as it would with the out-of-the-box Mutex class.   As such, I drafted the following test case:

[TestMethod]

              public void CanAcquireAndReleaseMutex()

              {

                     string mutexName = Guid.NewGuid().ToString();

                     var mutex = new CloudMutex(false, mutexName);

                     var acquiredByWorker = new ManualResetEvent(false);

                     string workerMessage = null;

                     ThreadPool.QueueUserWorkItem(new WaitCallback(o =>

                           {

                                  var workerMutex = new CloudMutex(false, mutexName);

                                  Console.WriteLine("Worker thread \"{0}\" has started.", Thread.CurrentThread.ManagedThreadId);

                                  workerMutex.WaitOne();

                                  Console.WriteLine("Worker thread \"{0}\" has acquired the cloud workerMutex and will wait 2 seconds.", Thread.CurrentThread.ManagedThreadId);

                                  acquiredByWorker.Set();

                                  Thread.Sleep(2 * 1000);

                                  workerMessage = Guid.NewGuid().ToString();

                                  Console.WriteLine("Worker thread \"{0}\" has waited 2 seconds, set the worker message to \"{1}\" and will release the workerMutex.", Thread.CurrentThread.ManagedThreadId, workerMessage);

                                  workerMutex.ReleaseMutex();

                                  Console.WriteLine("Worker thread \"{0}\" has released the workerMutex.", Thread.CurrentThread.ManagedThreadId);

                           }));

                     acquiredByWorker.WaitOne();

                     Console.WriteLine("Base thread \"{0}\" notified that the worker thread has acquired the cloud mutex.", Thread.CurrentThread.Name);

                     mutex.WaitOne();

                     Console.WriteLine("Base thread \"{0}\" has acquired the cloud mutex.", Thread.CurrentThread.Name);

                     mutex.Delete();

                     Assert.IsFalse(String.IsNullOrWhiteSpace(workerMessage));

              }

In words, the main test thread is taking the following steps:

The worker thread is taking the following steps:

• Acquiring the mutex
• Signaling to the main test thread that it has acquired the mutex
• Waiting two seconds
• Releasing the mutex

The above design ensures that the worker thread will always acquire the mutex before the main test thread attempts to do so.  The output from the test will look approximately as follows:

(T+0 sec) - Worker thread "5" has started.

(T+0 sec) - Worker thread "5" has acquired the cloud workerMutex and will wait 2 seconds.

(T+0 sec) - Base thread "9" notified that the worker thread has acquired the cloud mutex.

(T+2 sec) - Worker thread "5" has waited 2 seconds, set the worker message to "{76B921AA-AF9D-482F-97A9-496B9701A5C7}" and will release the workerMutex.

(T+2 sec) - Base thread "9" has acquired the cloud mutex.

Summarily, for this very simple case, the CloudMutex works just as a regular mutex would but uses the cloud as the backing mechanism as opposed to resources on a single machine.  I'm currently working on much more sophisticated cross machine test cases but, for now, trust me that this works whether the participants are in the same process or across the world from one another, so long as they both have internet access. and can someone agree on a mutex name (along with other Azure Service Bus details).

using System;

using System.Collections.Generic;

using System.Linq;

using System.Text;

using System.Threading;

using System.Diagnostics;

using Microsoft.ServiceBus;

using Microsoft.ServiceBus.Messaging;

using Yeast.WindowsAzure.ServiceBus;

namespace Yeast.WindowsAzure.Threading

{

       public class CloudMutex

              : WaitHandle

       {

              #region Constants

              public const string MetaLockQueueName = "cloudmutex_metalock";

              public const string MessageIDProperty = "ID";

              public static readonly Guid MetaLockMessageID = Guid.Parse("65A19C9A-40F3-4018-B16F-1923D3EA0457");

              #endregion

              #region Fields

             [ThreadStatic]

              private static Dictionary<Uri, CloudMutex> _threadOwnedMutexes = new Dictionary<Uri, CloudMutex>();

              #endregion

              #region Constructors & Initialization

              private CloudMutex()

              {

              }

              public CloudMutex(bool initiallyOwned, string name)

                     : this(initiallyOwned, name, ServiceBusSettings.FromConfiguration())

              {

              }

              public CloudMutex(bool initiallyOwned, string name, ServiceBusSettings settings)

                     : this(initiallyOwned, name, settings, CloudMutexConstructionMode.Full)

              {

              }

              protected CloudMutex(bool initiallyOwned, string name, ServiceBusSettings settings, CloudMutexConstructionMode mode)

              {

                     try

                     {

                           Settings = settings;

                           MetaLockQueueClient = Settings.Factory.CreateQueueClient(MetaLockQueueName, ReceiveMode.PeekLock);

                           EnsureMetaLockQueue();

                           if (mode == CloudMutexConstructionMode.Full)

                           {

                                  InstanceID = Guid.NewGuid();

                                  InitiallyOwned = initiallyOwned;

                                  Name = name;

                                  var builder = new UriBuilder(Settings.ServiceUri);

                                  builder.Path = Name;

                                  MutexQueueUri = builder.Uri;

                                  NextList = new Dictionary<Guid, CloudMutex>();

                                  MutexQueueClient = Settings.Factory.CreateQueueClient(Name, ReceiveMode.ReceiveAndDelete);

                                  EnsureMutexQueue();

                           }

                     }

                     catch (Exception exception)

                     {

                           throw new Exception(String.Format("Failed to create an instance of {0} on the Service Bus endpoint at {1}.", GetType().Name, Settings.ServiceUri), exception);

                     }

              }

              private void SeedMetaLockMessage()

              {

                     var singletonMessage = new BrokeredMessage();

                     singletonMessage.Properties[MessageIDProperty] = MetaLockMessageID;

                     MetaLockQueueClient.Send(singletonMessage);

              }

              protected void EnsureMetaLockQueue()

              {

                     try

                     {

                           if (!Settings.NamespaceClient.QueueExists(MetaLockQueueName))

                           {

                                  MetaLockQueueDescription = Settings.NamespaceClient.CreateQueue(MetaLockQueueName);

                                  SeedMetaLockMessage();

                                  CreatedMetaLockQueue = true;

                           }

                           else

                           {

                                  MetaLockQueueDescription = Settings.NamespaceClient.GetQueue(MetaLockQueueName);

                                  if (MetaLockQueueDescription.MessageCount == 0)

                                  {

                                         SeedMetaLockMessage();

                                  }

                           }

                     }

                     catch (Exception exception)

                     {

                           throw new Exception(String.Format("Failed to ensure the existence of, or to gain access to the meta lock queue with name {0}.", MetaLockQueueName), exception);

                     }

              }

              protected void EnsureMutexQueue()

              {

                     try

                     {

                           if (!Settings.NamespaceClient.QueueExists(Name))

                           {

                                  using (var scope = new MetaLockScope(MetaLockQueueClient))

                                  {

                                         OwnsMetaLock = true;

                                         if (!Settings.NamespaceClient.QueueExists(Name))

                                         {

                                                Trace.WriteLine(String.Format("{0} - Queue '{1}' does not exist at {2} and will be created.", DateTime.Now, Name, Settings.ServiceUri), GetType().Name);

                                                MutexQueueDescription = Settings.NamespaceClient.CreateQueue(Name);

                                                Trace.WriteLine(String.Format("{0} - Successfully created queue '{1}' at {2}.", DateTime.Now, Name, Settings.ServiceUri), GetType().Name);

                                                if (!InitiallyOwned)

                                                {

                                                       var mutexMessage = new BrokeredMessage();

                                                       MutexQueueClient.Send(mutexMessage);

                                                }

                                         }

                                         else

                                         {

                                                MutexQueueDescription = Settings.NamespaceClient.GetQueue(Name);

                                                if (InitiallyOwned)

                                                {

                                                       WaitOne();

                                                }

                                         }

                                  }

                                  OwnsMetaLock = false;

                           }

                           else

                           {

                                  MutexQueueDescription = Settings.NamespaceClient.GetQueue(Name);

                           }

                     }

                     catch (Exception exception)

                     {

                           throw new Exception(String.Format("Failed to ensure the existence of, or to gain access to the mutex queue with name {0}.", Name), exception);

                     }

              }

              #endregion

              #region Properties

              public Guid InstanceID

              {

                     get;

                     private set;

              }

              protected BrokeredMessage AcquisitionMessage

              {

                     get;

                     private set;

              }

              public CloudMutex Previous

              {

                     get;

                     private set;

              }

              public Dictionary<Guid, CloudMutex> NextList

              {

                     get;

                     private set;

              }

              public Uri MutexQueueUri

              {

                     get;

                     private set;

              }

              public bool OwnsMutex

              {

                     get;

                     private set;

              }

              public bool OwnsMetaLock

              {

                     get;

                     private set;

              }

              public bool CreatedMetaLockQueue

              {

                     get;

                     private set;

              }

              public bool InitiallyOwned

              {

                     get;

                     private set;

              }

              public string Name

              {

                     get;

                     private set;

              }

              public ServiceBusSettings Settings

              {

                     get;

                     private set;

              }

              public MessagingFactory Factory

              {

                     get;

                     private set;

              }

              protected QueueClient MetaLockQueueClient

              {

                     get;

                     private set;

              }

              protected QueueDescription MetaLockQueueDescription

              {

                     get;

                     private set;

              }

              protected QueueClient MutexQueueClient

              {

                     get;

                     private set;

              }

              protected QueueDescription MutexQueueDescription

              {

                     get;

                     private set;

              }

              #endregion

              #region Object Overrides

              public override int GetHashCode()

              {

                     return MutexQueueUri.GetHashCode();

              }

              public override string ToString()

              {

                     return MutexQueueUri.ToString();

              }

              public override bool Equals(object obj)

              {

                     bool areEqual = false;

                     if (obj is CloudMutex)

                     {

                           var operand = obj as CloudMutex;

                           areEqual = MutexQueueUri == operand.MutexQueueUri;

                     }

                     return areEqual;

              }

              #endregion

              #region Methods

              public static void ReinitializeMetaLock()

              {

                     ReinitializeMetaLock(ServiceBusSettings.FromConfiguration());

              }

              public static void ReinitializeMetaLock(ServiceBusSettings settings)

              {

                     settings.NamespaceClient.DeleteQueue(MetaLockQueueName);

                     var mutex = new CloudMutex(false, null, settings, CloudMutexConstructionMode.MetaLockOnly);

              }

              public override bool WaitOne(TimeSpan timeout)

              {

                     return WaitOne(timeout, false);

              }

              public override bool WaitOne(TimeSpan timeout, bool exitContext)

              {

                     return WaitOne((int)timeout.TotalMilliseconds, exitContext);

              }

              public override bool WaitOne()

              {

                     return WaitOne(1000 * 60 * 5);

              }

              public override bool WaitOne(int millisecondsTimeout, bool exitContext)

              {

                     if (OwnsMutex)

                     {

                           throw new InvalidOperationException(String.Format("Cannot acquire the mutex backed by queue {0} on the Service Bus endpoint at {1} because it is currently owned by this instance of {2}.", Name, Settings.ServiceUri, GetType().Name));

                     }

                     CloudMutex previousMutex;

                     if (_threadOwnedMutexes.TryGetValue(MutexQueueUri, out previousMutex))

                     {

                           Previous = previousMutex;

                           Previous.NextList.Add(InstanceID, this);

                           OwnsMutex = true;

                           AcquisitionMessage = Previous.AcquisitionMessage;

                     }

                     else

                     {

                           AcquisitionMessage = MutexQueueClient.Receive(TimeSpan.FromMilliseconds(millisecondsTimeout));

                           OwnsMutex = true;

                           _threadOwnedMutexes.Add(MutexQueueUri, this);

                     }

                     return OwnsMutex;

              }

              public void ReleaseMutex()

              {

                     if (!OwnsMutex)

                     {

                           throw new InvalidOperationException(String.Format("Cannot release the mutex backed by queue {0} on the Service Bus endpoint at {1}.  It is not owned by this instance of {2}.", Name, Settings.ServiceUri, GetType().Name));

                     }

                     if (Previous != null)

                     {

                           if (!Previous.NextList.Remove(InstanceID))

                           {

                                  throw new InvalidOperationException(String.Format("Failed to remove the {0} instance with ID {1} from the previous {0}'s (ID: {2}) next list.  The current instance was not found.  This may indicate a bug in your software.", GetType().Name, InstanceID, Previous.InstanceID));

                           }

                           Previous = null;

                     }

                     else

                     {

                           if (!_threadOwnedMutexes.Remove(MutexQueueUri))

                           {

                                  throw new InvalidOperationException(String.Format("Failed to remove the {0} instance with ID {1} and path {2} from the thread owned mutex list.  This may indicate a bug in your software.", GetType().Name, InstanceID, MutexQueueUri));

                           }

                     }

                     OwnsMutex = false;

                     var releaseMessage = new BrokeredMessage();

                     MutexQueueClient.Send(releaseMessage);

              }

              public void Delete()

              {

                     Trace.WriteLine(String.Format("{0} - About to delete queue '{1}' at {2}.", DateTime.Now, Name, Settings.ServiceUri), GetType().Name);

                     Settings.NamespaceClient.DeleteQueue(Name);

                     Trace.WriteLine(String.Format("{0} - Successfully deleted queue '{1}' at {2}.", DateTime.Now, Name, Settings.ServiceUri), GetType().Name);

              }

              #endregion

       }

}

The Meta Lock

I wanted this so that I could ensure that the basic infrastructure needed in service bus to support any one mutex could be created in a safe, isolated manner, free from race conditions.  All clients of this meta lock queue use the "peek lock" mechanism for accessing a singleton message within that queue.  The act of receiving the message from the queue is analagous to acquiring a lock on the entire service bus namespace for the purposes of these mutexes.  When the owner of the meta lock abandons the singleton message, that is analogous to releasing the lock. While this lock is held, one can be sure that no other thread is attempting to create service bus queues for mutexes so long as all parties respect the meta lock.  To make the meta lock easy to use in code, I've created a MetaLockScope class that implements the IDisposable interface so that the using keyword can be applied in code to demark the beginning and end of a protected code section.  The class has the following definition:

Acquiring and releasing the CloudMutex

With all of the scaffolding in place to support this cross machine synchronization mechanism laid out, I'll finally elaborate around how the two other key methods, WaitOne, and ReleaseMutex work internally.

 WaitOne

For each mutex, a unique service bus queue is required.  That queue will be in an "acquireable" state if a single message is present in the queue.  The act of receiving and deleting the message from the queue is done by the WaitOne method in order to acquire the mutex.  Because of the way that service bus queues behave, only one consumer will be allowed to receive that message and thus acquire the mutex.  All other callers will wait to receive a message, or return instantly without a message depending on the caller's choice.  Should multiple calls all attempt to acquire the mutex before it has been released, only the first caller will succeed while the others will themselves be queued up behind one another

 ReleaseMutex

To release the mutex, and this make it acquireable by another caller, a message must be placed back into the queue.  As such, the ReleaseMutex method underlyingly sends a message to the queue to effect the release.

 Notes

Supporting actor

The following enum exists in my source base that I'll share for clarity.  It doesn't do a terrible amount so I'll just post it for the curious.

using System;

Next steps

If anyone find this idea of a cloud mutex interesting, please let me know by commenting on this blog post and I'll look for some time to go deeper.

Finally, there is a class called ServiceBusSettings that I've included here along with a sister class called ServiceBusSection (a configuration section), that might deserve a blog post of their own.  That'll be my very next project unless something else catches my fancy first, if only to get feedback on other techniques that have worked well for others.  Suffice it to say though, these classes provide a means by which to access Windows Azure Service Bus credentials in a way that works whether running inside or outside of the compute emulator.  I consider that a general problem for which I have one general solution.

As always, I look forward to your input.   I post to share and I post to learn.

While upgrading to VS2012 Pro RC, my unit tests began to fail across the board when accessing various Azure APIs like RoleEnvironment.IsAvailable.  I would get the exception below consistently:

System.TypeInitializationException: The type initializer for 'Microsoft.WindowsAzure.ServiceRuntime.RoleEnvironment' threw an exception. ---> System.IO.FileLoadException: Mixed mode assembly is built against version 'v2.0.50727' of the runtime and cannot be loaded in the 4.0 runtime without additional configuration information.
Result StackTrace: 
at Microsoft.WindowsAzure.ServiceRuntime.RoleEnvironment.InitializeEnvironment()
   at Microsoft.WindowsAzure.ServiceRuntime.RoleEnvironment..cctor()
 --- End of inner exception stack trace ---
    at Microsoft.WindowsAzure.ServiceRuntime.RoleEnvironment.get_IsAvailable()

To reproduce this, I would do the following:

• Prerequisites: VS2012RC, .NET SDK for 1.7 Azure release (and it's dependencies from https://www.windowsazure.com/en-us/develop/net/)
• Open VS2012RC
• Create a new Test => Unit Test project
• Add a reference to Microsoft.WindowsAzure.ServiceRuntime from: c:\Program Files\Microsoft SDKs\Windows Azure\.NET SDK\2012-06\ref\Microsoft.WindowsAzure.ServiceRuntime.dll
• Add the following code to the default test file: 

The sample also illustrates how to hook into WCF internals, but that stuff is well documented elsewhere so I'll stay focused on the stuff that I found to be novel as I walk you through this SAMPLE code.

Sample Service

Nothing special.  Here's the interface definition.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Runtime.Serialization;
using System.ServiceModel;
using System.Text;

namespace Microsoft.Samples.DPE.WcfPerformanceCounters.Web
{
 // NOTE: You can use the "Rename" command on the "Refactor" menu to change the interface name "ISampleService" in both code and config file together.
 [ServiceContract]
 public interface ISampleService
 {
  [OperationContract]
  void DoWork();

  [OperationContract]
  void Cleanup();
 }
}

Here's where things start to get interesting.  The service implementation appears below.  I'll explain it below the code itself:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Runtime.Serialization;
using System.ServiceModel;
using System.Text;

using SampleFramework;
using SampleFramework.Diagnostics;

namespace Microsoft.Samples.DPE.WcfPerformanceCounters.Web
{
 [InterceptionBehavior(typeof(TrackMethodPerformanceBehavior))]
 [UnityInstanceProvider(typeof(ISampleService), true)]
 public class SampleService
  : ISampleService
 {
  #region Constructors

  public SampleService()
  {
  }

  #endregion

  #region Methods

  public void DoWork()
  {
   Random random = new Random();
   var secondsToWait = random.Next(2000, 10000);
   System.Threading.Thread.Sleep(secondsToWait);
  }

  public void Cleanup()
  {
   Random random = new Random();
   var secondsToWait = random.Next(1000, 50000);
   System.Threading.Thread.Sleep(secondsToWait);
  }

  #endregion
 }
}

First, note that the method implementations just sleep for a random number of seconds to simulate variable execution times and to make our performance counter data interesting.  Next, let's focus on the line "[UnityInstanceProvider(typeof(ISampleService), true)]".  This line is our primary hook into WCF internals.  It's definition appears below.  The purpose of this attribute is to allow us to replace the IInstanceProvider (https://msdn.microsoft.com/en-us/library/system.servicemodel.dispatcher.iinstanceprovider.aspx) in the WCF pipeline.  The instance provider is the thing that actually goes about creating instances of the SampleService class when it's requested.  It's within our custom instance provider that we'll place the code that allows us to intercept calls to the service methods.  Again, the attribute definition is here:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.ServiceModel;
using System.ServiceModel.Dispatcher;
using System.ServiceModel.Description;
using System.Diagnostics.Contracts;

using Yeast.Reflection;

namespace SampleFramework
{
 [AttributeUsage(AttributeTargets.Class)]
 public class UnityInstanceProviderAttribute
  : Attribute, IContractBehaviorAttribute, IContractBehavior
 {
  #region Constructors

  private UnityInstanceProviderAttribute()
  {
  }

  public UnityInstanceProviderAttribute(Type targetContract, bool yieldsSingleton)
  {
   Contract.Assume(targetContract != null);

   TargetContract = targetContract;
   YieldsSingleton = yieldsSingleton;
  }

  #endregion

  #region Properties

  public Type TargetContract
  {
   get;
   private set;
  }

  public bool YieldsSingleton
  {
   get;
   private set;
  }

  #endregion

  #region Methods

  public void ApplyDispatchBehavior(ContractDescription contractDescription, ServiceEndpoint endpoint, DispatchRuntime dispatchRuntime)
  {
   var providerGenericType = typeof(UnityInstanceProvider<,>);
   var providerConstructedType = providerGenericType.MakeGenericType(TargetContract, dispatchRuntime.Type);
   var provider = TypeFactory.CreateInstance(providerConstructedType) as IUnityInstanceProvider;
   provider.DispatchRuntime = dispatchRuntime;
   provider.YieldsSingleton = YieldsSingleton;
   dispatchRuntime.InstanceProvider = provider;
  }

  public void AddBindingParameters(ContractDescription contractDescription, ServiceEndpoint endpoint, System.ServiceModel.Channels.BindingParameterCollection bindingParameters)
  {
  }

  public void ApplyClientBehavior(ContractDescription contractDescription, ServiceEndpoint endpoint, ClientRuntime clientRuntime)
  {
  }

  public void Validate(ContractDescription contractDescription, ServiceEndpoint endpoint)
  {
  }

  #endregion
 }
}

The important point here is that we need access to the DispatchRuntime so that we can set the InstanceProvider property on it.  I've done some monkey business with generics here so that I could take advantage of .NET type system so that I could have one IInstanceProvider type dynamically created for each service type I'm manipulating.  This also allows me to use the new() operator when creating service instances as opposed to having to use Activator.  Below you'll find the source code for the instance provider.  This is where the magic happens and I invoke unity interception APIs:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.ServiceModel;
using System.ServiceModel.Description;
using System.ServiceModel.Dispatcher;
using System.ServiceModel.Channels;
using System.Diagnostics.Contracts;

using Microsoft.Practices.Unity;
using Microsoft.Practices.Unity.ObjectBuilder;
using Microsoft.Practices.Unity.StaticFactory;
using Microsoft.Practices.Unity.Utility;
using Microsoft.Practices.Unity.InterceptionExtension;

using Yeast;
using Yeast.Reflection;

namespace SampleFramework
{
 public class UnityInstanceProvider<ContractType, ServiceType>
  : IUnityInstanceProvider
  where ServiceType : class, ContractType, new()
  where ContractType : class
 {
  #region Fields

  private static object _singletonLockTarget = new object();

  #endregion

  #region Constructors

  public UnityInstanceProvider()
  {
  }

  #endregion

  #region Properties

  public static ContractType SingletonService
  {
   get;
   private set;
  }

  public DispatchRuntime DispatchRuntime
  {
   get;
   set;
  }

  public bool YieldsSingleton
  {
   get;
   set;
  }

  #endregion

  #region Methods

  public ContractType InstantiateService()
  {
   var behaviors =
    (from attribute in typeof(ServiceType).GetAttributes<InterceptionBehaviorAttribute>(false)
     let behavior = TypeFactory.CreateInstance(attribute.BehaviorType)
     select behavior as IInterceptionBehavior).ToList();
   
   var service = new ServiceType();
   var proxiedService = Intercept.ThroughProxy<ContractType>(
    service
    , new InterfaceInterceptor()
    , behaviors);
   return proxiedService;
  }

  public object GetInstance(InstanceContext instanceContext, Message message)
  {
   object instance;

   if (YieldsSingleton)
   {
    lock (_singletonLockTarget)
    {
     if (SingletonService == null)
     {
      SingletonService = InstantiateService();
     }

     instance = SingletonService;
    }
   }
   else
   {
    instance = InstantiateService();
   }

   return instance;
  }

  public object GetInstance(InstanceContext instanceContext)
  {
   return GetInstance(instanceContext, null);
  }

  public void ReleaseInstance(InstanceContext instanceContext, object instance)
  {
   if (instance is IDisposable)
   {
    (instance as IDisposable).Dispose();
   }
  }

  #endregion
 }
}

The most important method is the InstantiateService method that's called by the GetInstance method.  It serves two key purposes:

• Looks for all InterceptionBehaviorAttribute instances that have been applied to the ServiceType.  This extensibility point is reading the metadata applied to the service class via a lined of code that I have yet to explain.  That is the [InterceptionBehavior(typeof(TrackMethodPerformanceBehavior))] attribute.  This is another of my custom attributes and is not WCF related per se.  It's basically hooking up the TrackMethodPerformanceBehavior class, which does the work of creating and using the performance counters that this whole exercise was really all about.
• Creates an instance of the service class and calls the Unity method Intercept.ThroughProxy.  The proxied service instance is returned instead of the raw service instance.  This is "the main thing".

The base interface, IUnityIntstanceProvider looks like this:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.ServiceModel;
using System.ServiceModel.Description;
using System.ServiceModel.Dispatcher;

namespace SampleFramework
{
 public interface IUnityInstanceProvider
  : IInstanceProvider
 {
  DispatchRuntime DispatchRuntime
  {
   get;
   set;
  }

  bool YieldsSingleton
  {
   get;
   set;
  }
 }
}

This inteface is defined so that I can access the properties defined in non-generic code on instances of UnityInstanceProvider<,>.

Here's what the InterceptionBehaviorAttribute looks like.  It's dead simple:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Diagnostics.Contracts;

using Microsoft.Practices.Unity.InterceptionExtension;

namespace SampleFramework
{
 [AttributeUsage(AttributeTargets.Class, AllowMultiple = true, Inherited = true)]
 public class InterceptionBehaviorAttribute
  : Attribute
 {
  #region Constructors

  private InterceptionBehaviorAttribute()
  {
  }

  public InterceptionBehaviorAttribute(Type behaviorType)
  {
   Contract.Assume(behaviorType != null);
   Contract.Assume(typeof(IInterceptionBehavior).IsAssignableFrom(behaviorType));

   BehaviorType = behaviorType;
  }

  #endregion

  #region Properties

  public Type BehaviorType
  {
   get;
   private set;
  }

  #endregion
 }
} 

Next, I'll share the code for the TrackMethodPerformanceBehavior which implements the Unity IInterceptionBehavior interface so as to add the needed functionality before and after the method invocation:

 using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.ServiceModel;
using System.ServiceModel.Description;
using System.ServiceModel.Dispatcher;
using System.ServiceModel.Channels;
using System.Diagnostics.Contracts;
using System.Threading;

using Microsoft.Practices.Unity;
using Microsoft.Practices.Unity.ObjectBuilder;
using Microsoft.Practices.Unity.StaticFactory;
using Microsoft.Practices.Unity.Utility;
using Microsoft.Practices.Unity.InterceptionExtension;

using Yeast;
using Yeast.Reflection;
using Yeast.Threading;
using Yeast.Collections.Generic;

namespace SampleFramework.Diagnostics
{
 public class TrackMethodPerformanceBehavior
  : IInterceptionBehavior
 {
  #region Fields

  private static ReaderWriterLockSlim _dictionaryLock = new ReaderWriterLockSlim(LockRecursionPolicy.SupportsRecursion);

  #endregion

  #region Constructors

  public TrackMethodPerformanceBehavior()
  {
   TypeCategoryMap = new Dictionary<Type, IServiceContractPerformanceCategory>();
  }

  #endregion

  #region IInterceptionBehavior Members

  public IEnumerable<Type> GetRequiredInterfaces()
  {
   return new List<Type>();
  }

  public IMethodReturn Invoke(IMethodInvocation input, GetNextInterceptionBehaviorDelegate getNext)
  {
   IMethodReturn returnValue;

   var beforeTime = DateTime.Now;

   returnValue = getNext().Invoke(input, getNext);

   var afterTime = DateTime.Now;
   var millisecondDelta = afterTime.Subtract(beforeTime).TotalMilliseconds;
   var category = this[input.Target.GetType()];
   var counter = category.MethodCounterMap[input.MethodBase];
   counter.RawValue = (int)millisecondDelta;

   return returnValue;
  }

  public bool WillExecute
  {
   get
   {
    return true;
   }
  }

  public static Dictionary<Type, IServiceContractPerformanceCategory> TypeCategoryMap
  {
   get;
   private set;
  }

  public IServiceContractPerformanceCategory this[Type serviceType]
  {
   get
   {
    IServiceContractPerformanceCategory category;

    using (var readerLock = new UpgradeableReaderLockContext(_dictionaryLock))
    {
     if (!TypeCategoryMap.TryGetValue(serviceType, out category))
     {
      using (var writerLock = readerLock.GetWriterContext())
      {
       if (!TypeCategoryMap.TryGetValue(serviceType, out category))
       {
        category = ServiceContractPerformanceCategoryFactory.GetForContract(serviceType);
        TypeCategoryMap.Add(serviceType, category);
       }
      }
     }
    }

    return category;
   }
  }

  #endregion
 }
}

I've highlighted the important code in bold red.  This method illustrates the raw power of unity to intercept method calls transparently so as to add code before and after method execution or even totally reimplement a method call.  Tangentially, I'm using a Yeast class called UpgradeableReaderLockContext that I'll post a blog article about later.  It provides an IDisposable-based wrapper around the ReaderWriterLockSlim class at makes for clean syntax.

Finally, I'll present the code that backs the expression ServiceContractPerformanceCategoryFactory.GetForContract(serviceType);.  It's an ultra simple method that just does some reflection to create the constructed type that does the fun work:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;

using Yeast.Reflection;

namespace SampleFramework.Diagnostics
{
 public static class ServiceContractPerformanceCategoryFactory
 {
  public static IServiceContractPerformanceCategory GetForContract(Type contractType)
  {
   var genericCategory = typeof(ServiceContractPerformanceCategory<>);
   var constructedCategory = genericCategory.MakeGenericType(contractType);
   var category = TypeFactory.CreateInstance(constructedCategory);

   return category as IServiceContractPerformanceCategory;
  }
 }
}

The ServiceContractPerformanceCategory<> generic class is constructed using the WCF service type, instantiated, and returned.  It does the bulk of the work.  Note that, at this point in this blog article, I've already really demonstrated how to do the main point of the article which was to use Unity with WCF.  The rest of this is interesting only insofar as you care about the code I used to dynamically create and use custom performance counters to instrument service methods.  Here's the source for ServiceContractPerformanceCategory:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.ServiceModel;
using System.ServiceModel.Description;
using System.Reflection;
using System.Diagnostics.Contracts;
using System.Diagnostics;

using Yeast.Reflection;

namespace SampleFramework.Diagnostics
{
 public class ServiceContractPerformanceCategory<ServiceType>
  : IServiceContractPerformanceCategory
  where ServiceType : class
 {
  #region Fields

  private static Type _serviceTypeInstance;
  private static Type _contractTypeInstance;
  private static List<MethodInfo> _operations;
  private static Dictionary<MethodBase, PerformanceCounter> _methodCounterMap;
  private static PerformanceCounterCategory _category;

  #endregion

  #region Constructors & Setup

  static ServiceContractPerformanceCategory()
  {
   _serviceTypeInstance = typeof(ServiceType);

   var serviceContractAttribute = _serviceTypeInstance.GetAttribute<ServiceContractAttribute>(false);

   if (serviceContractAttribute == null)
   {
    var serviceInterface =
     (from type in _serviceTypeInstance.GetInterfaces()
      let attribute = type.GetAttribute<ServiceContractAttribute>(false)
      where attribute != null
      select new { Type = type, Attribute = attribute }).FirstOrDefault();

    if (serviceInterface == null)
    {
     throw new Exception(String.Format("Could not find an interface implemented by {0} that is decorated with {1}.", _serviceTypeInstance, typeof(ServiceContractAttribute)));
    }

    serviceContractAttribute = serviceInterface.Attribute;
    _contractTypeInstance = serviceInterface.Type;
   }
   else
   {
    _contractTypeInstance = _serviceTypeInstance;
   }

   _operations =
    (from operation in _contractTypeInstance.GetMethods(BindingFlags.Instance | BindingFlags.Public)
     let operationContractAttribute = operation.GetAttribute<OperationContractAttribute>(false)
     where operationContractAttribute != null
     select operation).ToList();

   EnsureCreated();

   _methodCounterMap = new Dictionary<MethodBase, PerformanceCounter>();
   foreach (var operation in _operations)
   {
    _methodCounterMap.Add(operation, new PerformanceCounter(PerformanceCategoryName, operation.Name, false));
   }
  }

  public ServiceContractPerformanceCategory()
   : this(true)
  {
  }

  public ServiceContractPerformanceCategory(bool shouldRecreate)
  {
   if (shouldRecreate && PerformanceCounterCategory.Exists(PerformanceCategoryName))
   {
    PerformanceCounterCategory.Delete(PerformanceCategoryName);
   }

   EnsureCreated();
  }

  private static void EnsureCreated()
  {
   try
   {
    if (!PerformanceCounterCategory.Exists(PerformanceCategoryName))
    {
     var creationData = new CounterCreationDataCollection();
     foreach (var operation in _operations)
     {
      creationData.Add(new CounterCreationData()
      {
       CounterType = PerformanceCounterType.NumberOfItems32,
       CounterName = operation.Name
      });
     }

     _category = PerformanceCounterCategory.Create(
      PerformanceCategoryName
      , null
      , PerformanceCounterCategoryType.SingleInstance
      , creationData);
    }
    else
    {
     _category = new PerformanceCounterCategory(PerformanceCategoryName);
    }
   }
   catch (Exception exception)
   {
    throw new Exception(String.Format("Failed to create performance category {0}.", PerformanceCategoryName), exception);
   }
  }

  #endregion

  #region Properties

  public static string PerformanceCategoryName
  {
   get
   {
    return _contractTypeInstance.Name;
   }
  }

  public Type ServiceTypeInstance
  {
   get
   {
    return _serviceTypeInstance;
   }
  }

  public Type ContractTypeInstance
  {
   get
   {
    return _contractTypeInstance;
   }
  }

  public List<MethodInfo> Operations
  {
   get
   {
    return _operations;
   }
  }

  public Dictionary<MethodBase, PerformanceCounter> MethodCounterMap
  {
   get
   {
    return _methodCounterMap;
   }
  }

  #endregion
 }
} 

Problem

When running the sample application in a Windows Azure web role, I got an exception due to the fact that the service account that's used to run the application pool does not have sufficient permissions to write to the registry so as to create the performance counters.  I now need to investigate how to configure things in Azure so that it will work properly there as well.  My initial thought process is that I may need to use a startup task, but we'll see.  Please provide any ideas you have there.

Goals

• Expose a URL that an end user can click on that'll initiate a file download of a file stored in Azure blob storage
• Ensure that any amount of security requirements can be enforced
• Enable running custom code before, during, and after the file download to include writing progress information to a data store like SQL Azure or table storage
• Support everything from tiny up to multi-gigabyte files
• Provide performance on par with standard blob downloads

Approach

I’ve been examining four distinct download strategies and their performance characteristics across Tiny (> 1MB), Small (~10MB), Medium (~200MB), and Gigantic file (~2.5GB) file sizes, implying four file size magnitudes.  Each download strategy represents a different way of downloading a file from Azure blob storage.  The strategies are:
 
1.       Direct Download – Uses the direct Azure storage service URL and an ultra-simple HTTP file download mechanism
    a.    Consider this a good baseline measure
    b.      Provides no ability to run code before, after, or during a file download
2.       API Download – Uses the Azure storage APIs to download a file.  This is fundamentally similar to a direct download call but is using owner-level permissions to access Azure storage
    a.       Similar to a direct download, there is no ability to run server-side code before, after or during a file download
3.       Proxy Download – Leverages a web role that internally retrieves the blob stream from the blob storage service and proxies it through to the HTTP response stream.
    a.       This makes all access to the storage service opaque and encapsulated into a custom HTTP handler that controls the entire download process
    b.      Allows running code before, after, or during a file download
4.       Pass-through Download – Takes advantage of the Azure Shared Access Signature technology and essentially redirects the requestor to a URL that includes authentication information to ensure all downloads are secured
    a.       Likely the best single solution to your storage problem.  This solution minimizes overhead while retaining control over security
    b.      Allows running code before *only* a blob download.  This is because the HTTP response is an HTTP 302 redirection wherein the server relinquishes control once the download URL is transmitted
    c.       This approach will tend to yield the greatest amount of consistency across file sizes
 

To test how each of these perform with respect to one another, I created Visual Studio unit tests along with a set of Visual Studio load tests to see how they perform over time.  The good news is that the Proxy strategy out performs other download strategies in 75% or more cases, where the pass-through strategy came generally in second (good news).  Here are my statistics on average download times:

Problem

There remains an issue that only occurs with supremely large files.  That problem is that, since we are proxying the downloaded file through an Azure web role, there appears to be a timeout set somewhere (maybe on the load balancer???) that constrains downloads *of this strategy only* to about 30 minutes.  As a result, files in excess of about 2GB will download in large part but have the potential to get aborted closer to the end of the download.  This limitation, again, appears to be a network infrastructure issue and not specific to the download strategy.  Can anyone verify the root cause of this for me?  Are there any solutions in site?  If you can solve this, you help me solve an important customer problem.

The Guts

The main chunk of code that I used to implement the Proxy approach is a very simple HTTP handler that asynchronously reads buffers from blob storage.  The code appears below.  At first, I tried just using

blob.DownloadToStream(context.Response.OutputStream);, but this would end out failing with anything other than small files since no content was showing up on the response stream for a long period of time (until the entire download was complete).  That approach also yielded problems with causing the file download dialog to pop up.  This approach of chunking the download works well.  I even made the buffer size configurable via the query string for the purposes of this SAMPLE code.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Web;
using System.Text;
using System.Text.RegularExpressions;
using System.IO;
using System.Threading;

using Microsoft.WindowsAzure;
using Microsoft.WindowsAzure.StorageClient;
using Microsoft.WindowsAzure.ServiceRuntime;

using Yeast;
using Yeast.Collections.Generic;
using Yeast.IO;

namespace BlobProxy.Web
{
 /// <summary>
 /// Summary description for Download
 /// </summary>
 public class Download
  : IHttpHandler
 {
  public void ProcessRequest(HttpContext context)
  {
   try
   {
    var accountName = context.Request["accountName"];
    var path = context.Request["path"];
    int bufferSize;
    if (!Int32.TryParse(context.Request["bufferSize"], out bufferSize))
    {
     bufferSize = 100 * 1024;
    }

    if (accountName != Configuration.BlobProxySection.Instance.AccountName)
    {
     context.Response.ContentType = "text/plain";
     context.Response.Write(String.Format("I don't understand how to work with account name '{0}'.", accountName));
    }
    else
    {
     var account = CloudStorageAccount.Parse(String.Format("DefaultEndpointsProtocol=https;AccountName={0};AccountKey={1}", accountName, Configuration.BlobProxySection.Instance.AccountKey));
     var client = account.CreateCloudBlobClient();
     var pathExpression = new Regex("/(?<containerName>[^/]+)/(?<blobPath>.+)", RegexOptions.IgnoreCase | RegexOptions.Singleline | RegexOptions.ExplicitCapture);
     var pathMatch = pathExpression.Match(path);
     var containerName = pathMatch.Groups["containerName"].Value;
     var blobPath = pathMatch.Groups["blobPath"].Value;
     var container = client.GetContainerReference(containerName);
     var blob = container.GetBlobReference(blobPath);

     Exception downloadException = null;
     bool isDataComplete = false;
     var dataQueue = new BlockingQueue<Tuple<int, byte[]>>();
     var bufferQueue = new BlockingQueue<byte[]>();

     ThreadPool.QueueUserWorkItem(new WaitCallback(o =>
     {
      try
      {
       var blobStream = blob.OpenRead();
       int blockLength = 0;

       do
       {
        byte[] blockBuffer = null;
        if (bufferQueue.Count > 0)
        {
         blockBuffer = bufferQueue.Dequeue();
        }
        else
        {
         blockBuffer = new byte[bufferSize];
        }

        blockLength = blobStream.Read(blockBuffer, 0, blockBuffer.Length);

        if (blockLength > 0)
        {
         dataQueue.Enqueue(new Tuple<int, byte[]>(blockLength, blockBuffer));
        }
       } while (blockLength > 0);
      }
      catch (Exception exception)
      {
       downloadException = exception;
      }
      finally
      {
       isDataComplete = true;
      }
     }));

     blob.FetchAttributes();
     context.Response.ContentType = "application/x-msdownload"; // blob.Properties.ContentType;
     context.Response.Buffer = false;
     context.Response.AddHeader("content-disposition", "attachment; filename=" + blobPath);

     while (context.Response.IsClientConnected && (!isDataComplete || dataQueue.Count > 0))
     {
      if (downloadException != null)
      {
       throw new Exception("The internal blob download failed.", downloadException);
      }

      var data = dataQueue.Dequeue();
      context.Response.OutputStream.Write(data.Item2, 0, data.Item1);
      context.Response.Flush();
      //Put the processed buffer back into the queue so as to minimize memory consumption
      bufferQueue.Enqueue(data.Item2);
     }
    }
   }
   catch (Exception exception)
   {
    context.Response.Write(String.Format("Failed to download file due to the exception:{0}{1}", Environment.NewLine, exception));
   }
  }

  public bool IsReusable
  {
   get
   {
    return true;
   }
  }
 }
}

Now, the common question: "Why the heck is the framework called Yeast?"  It's a metaphor for adding yeast to bread dough so that it'll rise.  So, by adding Yeast to your software, you'll make it rise too.  Nobody wants a flat app, now do they.  Also, I do my very best not to compete with pre-existing or new solutions to problems and welcome having my Yeast code become obsolete because Microsoft has introduced a solution.  As an example, the original version of this post introduced a class I called BlockingQueue<T> that implemented a thread safe producer-consumer queue.  Thanks to that original post version Stephen Toub turned me on to BlockingCollection<T> which gives me exactly what I need.  I'm ripping out BlockingQueue<T> from Yeast as we speak.  For perpetuity sake, this is what BlockingQueue looks like.

using System;
using System.Collections;
using System.Collections.Generic;
using System.Threading;

namespace Yeast.Collections.Generic
{
 // Summary:
 //     Implements a queue which blocks all consumer threads if the queue is empty,
 //     and signals any waiting consumers once data becomes available.
 public class BlockingQueue<T> : IEnumerable<T>, IEnumerable
 {
  // Summary:
  //     THe object used to ensure that the methods herein are thread safe.
  public readonly object SyncRoot;

  public BlockingQueue();
  //
  //
  // Parameters:
  //   capacity:
  //     The maximum capacity of the underlying queue.
  //
  // Exceptions:
  //   System.ArgumentException:
  //     If the capacity specified is less than one, this constructor will fail.
  public BlockingQueue(int capacity);

  // Summary:
  //     Gets the capacity of this instance as specified in the constructor.
  public int Capacity { get; }
  //
  // Summary:
  //     Gets the number of elements currently in the queue.
  public int Count { get; }
  //
  // Summary:
  //     Set when the queue has no elements in it.
  public WaitHandle Empty { get; }
  //
  // Summary:
  //     Set when the queue has reached it's Capacity.
  public WaitHandle Full { get; }

  // Summary:
  //     Removes an element from the queue and returns it. This method will block
  //     if the queue is empty and will unblock once an item has been enqueued.
  //
  // Returns:
  //     An item of type T that is at the front of the queue.
  public T Dequeue();
  //
  // Summary:
  //     Adds an element to the back of the queue.
  //
  // Parameters:
  //   element:
  //     The element to be added to the queue.
  //
  // Remarks:
  //     If the queue is empty prior to this operation, the Empty event will be set.
  //      If the queue reaches its capacity as a result of this operation, the Full
  //     event will be set.
  public void Enqueue(T element);
  //
  // Summary:
  //     Implements an enumerator that internally calls Yeast.Collections.Generic.BlockingQueue<T>.Dequeue()
  //     so as to yield the next elements in the queue.  The resulting enumerator
  //     will behave just as if Dequeue were called in a loop.
  //
  // Returns:
  //     An enumerator that internally dequeues elements from the queue infinitely.
  public IEnumerator<T> GetEnumerator();
 }
}

As an adult, I've been oft been called everything from a Microsoft fan boy to a Microsoft zealot, depending on how much you like me =^).  I'll own those labels too.  Hopefully I'll live up to them.

Now, getting down to business: My plan is to blog about technology from Microsoft, my own inventions and discoveries, as well as the excellent and awful from the tech universe at large.  Obviously, I spend a lot of my time focused on the Cloud, but I'll stray from there as it suits me to discuss what impassions me at the moment.

I look forward to your interest, ideas, and passion.

~Cheers

Sean