So if you are trying to use the REST API, here are a couple of findings that I have run into that I hope will help you.

When creating a virtual machine you have two APIs that you can use.  The first is the Add Role and the second is the Create Virtual Machine Deployment. 

Which one should you utilize?   If you are creating only one virtual machine you can create it with either API. 

However, the issues start to arise when you need to create multiple virtual machines in the same script.  If you use the Add Role API you will be able to create the first virtual machine but when the second virtual machine in the script is encountered you will receive a 400 Bad Request error (keep in mind that there are a number of 400 error codes that can be returned so make sure to look at the text and not just the HTTP status code).   You can find all the Error codes here.

When you have multiple virtual machines to create in the script then you will need to start with the Create Virtual Machine Deployment API to create the first virtual machine and then use the Add Role API to create all subsequent virtual machines in your deployment.

You will notice when you look at the XML payload you need to include in the API call that the Create Virtual Machine Deployment contains a <Role> element within a <Deployment> element.    When you look at the Add Role XML everything falls under the <PersistVMRole> element.  So you need to run the API the does the <Deployment> first before running the <PersistVMRole> elements.

NOTE: It is very important that you have all the XML nodes in the correct order. 

This is important because if the elements are not in the correct order, your VM can still be created (usually) but things like the Availability Set or EndPoints won't get created.  What I have found is that when everything is in the right order (based on the examples in the documentation) then everything worked and all the virtual machines were created as expected. 

In addition, if simple things like the <Protocol> node is before the <LoadBalancerProbe> element (in the documentation it is the only element after the <LoadBalancerProbe> element) then the endpoints will be created but the Load Balancer won't and you will see that the Load Balancer Set Name column in the portal will show an error stating Missing Probe.

The BizTalk Summit 2012 is a 2 day pre-release event to prepare for the launch of the next version of BizTalk Server. Activities at the summit will include a Keynote by Scott Guthrie, product roadmap, and technical sessions by the Engineering Team, MCS and Partners.

I, along with my co-presenter Michael Muta, will be doing a presentation on Notes from the Field and will include case studies of projects and their architecture (on-premise and hybrid with Azure) along with best practices and how to get your applications ready for the cloud.

This two day event is being held at the Microsoft campus December 10th and 11th.  If you are planning on attending check out the session and stop by and say hello.

The Shared Access Signature feature provides a means for you to provide permissions, policies and a time window for those permissions and policies to be valid.  You can also modify or revoke the permissions as necessary.  This is important because Azure storage works by providing a password for your storage account.  Anyone who has that password has access/ownership of your storage account.  That is why you can’t give out your shared key.  The only exception to this is of course if you mark your container public.  The Shared Access Signature features allows you to give permissions but still keep control over security.

So, the Shared Access Signatures provides code that lets us separate the code that signs the request from the execution.  It is implemented as a query string parameter that proves that the creator of the URL is authorized to perform the operations.   The following MSDN article shows how to manually create the signature string.  This is here just so that you can see what is provided and how the string is make up.  In addition, there is another MSDN article that shows how to create a shared access signature and provides the following graphic to show the part that make up the query string.

How can we code the blob access (upload and download) to include the Shared Access Signature? 

The following code shows how to upload a file to blob storage using a Shared Access Signature:

        public string UploadBlobToAzure(string fileName)
        {
            if (fileName == null)
            {
                throw new ArgumentNullException("fileName");
            }

            try
            {
                // create storage account
                string storageConnnectString = string.Format("DefaultEndpointsProtocol={0}; AccountName={1}; 
                       AccountKey={2}", "http", ConfigurationManager.AppSettings["storageAccountName"], 
                       ConfigurationManager.AppSettings["storageAccountKey"]);
                var account = CloudStorageAccount.Parse(storageConnnectString);
               
                // create blob client
                CloudBlobClient blobStorage = account.CreateCloudBlobClient();
                
                //Get the container.  This is what we will attach the signature to.
                CloudBlobContainer container = blobStorage.GetContainerReference("<storagecontainername>");
                container.CreateIfNotExist();

                //Create the shared access permissions and policy
                BlobContainerPermissions containerPermissions = new BlobContainerPermissions();
                string sas = container.GetSharedAccessSignature(new SharedAccessPolicy()
                    {
                        SharedAccessExpiryTime = DateTime.UtcNow.AddMinutes(30), Permissions = 
                            SharedAccessPermissions.Write | SharedAccessPermissions.Read
                    });
                //Turn off public access
                containerPermissions.PublicAccess = BlobContainerPublicAccessType.Off;
                container.SetPermissions(containerPermissions);

                string uniqueBlobName = string.Format(@"<storagecontainername>/<filename>_{0}.config", 
                       Guid.NewGuid().ToString());

                //assign the shared access policy
                CloudBlobClient blobClient = new CloudBlobClient(account.BlobEndpoint, new 
                       StorageCredentialsSharedAccessSignature(sas));

                CloudBlob blb = blobClient.GetBlobReference(uniqueBlobName);

                //upload the file
                blb.UploadFile(fileName);

                System.Console.WriteLine("File successfully uploaded to " + blb.Uri);
                return "File successfully uploaded to " + blb.Uri;
            }
            catch (StorageClientException e)
            {
                return "Blob Storage error encountered: " + e.Message;
            }
            catch (Exception e)
            {
                return "Error encountered: " + e.Message;
            }
            finally
            {
            }
        }

Once you upload the file then you can use the following code to access that file and download it locally:

        public string DownloadBlobfromAzure(string fileName)
        {
            if (fileName == null)
            {
                throw new ArgumentNullException("fileName");
            }

            try
            {
                // create storage account
                string storageConnnectString = string.Format("DefaultEndpointsProtocol={0}; AccountName={1}; 
                          AccountKey={2}", "http", ConfigurationManager.AppSettings["storageAccountName"], 
                          ConfigurationManager.AppSettings["storageAccountKey"]);
                var account = CloudStorageAccount.Parse(storageConnnectString);;
                // create blob client
                CloudBlobClient blobStorage = account.CreateCloudBlobClient();

                CloudBlobContainer container = blobStorage.GetContainerReference("<storagecontainername>");
                container.CreateIfNotExist();

                //Create the shared access permission and policy
                BlobContainerPermissions containerPermissions = new BlobContainerPermissions();
                string sas = container.GetSharedAccessSignature(new SharedAccessPolicy()
                {
                    SharedAccessExpiryTime = DateTime.UtcNow.AddMinutes(30),
                    Permissions = SharedAccessPermissions.Write | SharedAccessPermissions.Read
                });
                //Turn off public access
                containerPermissions.PublicAccess = BlobContainerPublicAccessType.Off;
                container.SetPermissions(containerPermissions);

                string uniqueBlobName = string.Format(fileName);

                //Assign the shared access policy
                CloudBlobClient blobClient = new CloudBlobClient(account.BlobEndpoint, new 
                      StorageCredentialsSharedAccessSignature(sas));

                //Get a point to the blob object
                CloudBlob blb = blobClient.GetBlobReference(uniqueBlobName);

                string fileLocation = @"C:\Documents\";
                int pos = fileName.IndexOf('/');
                string fname = fileName.Substring(pos + 1, fileName.Length - pos - 1 );

                //Download the blob object
                blb.DownloadToFile(fileLocation + fname);

                System.Console.WriteLine("File successfully downloaded to " + fileLocation + fname);

                return "File successfully downloaded to " + fileLocation + fname;
            }
            catch (StorageClientException e)
            {
                return "Blob Storage error encountered: " + e.Message;
            }
            catch (Exception e)
            {
                return "Error encountered: " + e.Message;
            }
            finally
            {
            }
        }

You may have noticed that when we created the policy we didn’t specify a SharedAccessStartTime attribute.  We can specify a start time that policy will be good for or we can leave it off in which case the policy will go into effect immediately.

NOTE:  The files names are case sensitive.  If you attempt to find a blob and are not using lower case for the file name you will receive a return error

Outer Exception

Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.

Inner Exception Message

{"The remote server returned an error: (403) Forbidden."}

Content Based Router, Message Filter, Recipient List, Splitter, Aggregator, Resequencer, Composed Message Processor, Scatter-Gather, Process Manager and Message Broker.

So, lets dive right in to each of these patterns and see how these can be implemented.

  Content-Based Router

This pattern requires the the router system inspect each message and direct that message to different channels based on data elements contained in the message.  The routing should be configurable to route based on different fields, the existence or non-existence of a specific value and the routing rules should be easily maintained as there are typically frequent changes

This pattern can be implement in Azure using a Service Bus Topic.  There are two parts that need to be implemented to put this pattern into use.  The first is that subscriptions need to be setup and the second is that a message needs to be created with the specific properties populated to match the subscription which follows the diagram below:

The following code shows to create the subscriptions as well as the submitted message.  Let’s first start by creating the subscriptions

public static void CBRPatternExampleWithTopic(String topicName)
 {
 String serviceNamespace = ....;
 String issuer = ....;
 String key = ....;
 
 SharedSecretCredential credential =.... 

 ....

 ServiceBusNamespaceClient namespaceClient = new ServiceBusNamespaceClient(serviceBusUri, credential);
 Topic topic = namespaceClient.GetTopic(topicName);

 //using the AddSubscription(String, FilterExpression) method overload 

 topic.AddSubscription(“PremiumMemberType”,new SqlFilter("MemberType = Premium"));

 topic.AddSubscription("OtherMemberTypes", new SqlFilter("MemberType <> Premium"));
 }

Then we will create the message with the property populated that the subscriptions will key off of.

public static void SubmitMessageToTopic()
 {
 String issuer = ....;
 String key = ....;

 TokenProvider tp = TokenProvider.CreateSharedSecretTokenProvider(issuer, key); 
 Uri uri = ServiceBusEnvironment.CreateServiceUri(.., .., ..);

 MessagingFactory factory = MessagingFactory.Create(uri, tp);
 MessageSender myTopic = factory.CreateMessageSender("<TopicName>");

 BrokeredMessage message = new BrokeredMessage(); 

 //This will create a message that will be routed to the first subscription above 

 message.Properties.Add("MemberType", Premium); 

 //OR add the following property instead to have it routed to the other subscription above

message.Properties.Add("MemberType", Gold);

myTopic.Send(message);

  Message Filter

This pattern requires that any message that doesn’t match the criteria of a subscription will be discarded.  This pattern is used to eliminate any messages that aren’t routed to a subscriber such as those setup using the Content Based Router pattern.  If the message doesn’t fall into a filter used by the subscriber then the message should be routed to a filter channel. 

When using an Azure Service Bus Topic you can utilize the RuleDescription object along with the FilterAction and FilterExpression objects to setup the subscriber filters.  You can explicitly utilize the MatchNoneFilterExpression object to subscribe to messages that haven’t been routed to any other subscriber.  Once routed to this subscriber you can either log that the message wasn’t picked up and you can either process it or discard the message

The following code shows how to add a subscriber to the Topic for a MatchNoneFilterExpression

public static void AddFilterPatternSubscriptionToTopic(String topicName)
 {
 String serviceNamespace = ....;
 String issuer = ....;
 String key = ....;
 
 SharedSecretCredential credential =.... 

 ....

 ServiceBusNamespaceClient namespaceClient = new ServiceBusNamespaceClient(serviceBusUri, credential);
 Topic topic = namespaceClient.GetTopic(topicName);

 <place other subscriber code here>

 RuleDescription matchNoneRule = new RuleDescription()
 {
     FilterAction = new SqlFilterAction(“set defer = ‘yes’;”),
     FilterExpression = new MatchNoneFilterExpression()
 };
 

 //using the AddSubscription(String, RuleDescription) method overload

 Subscription matchNoneFilterSubscription =
 topic.AddSubscription(“matchNoneFilterSubscription”, matchNoneRule);  

 }

  Dynamic Router

This pattern provides for the scenario in that each recipient would register and list the conditions or rules for which messages or types of messages it can handle.  The conditions or rules are stored so that when a message arrives, the rules run and the message parameters are evaluated against the rules and then the message is routed accordingly to the correct receiver.  The benefit of this is that you can achieve predictive routing, with eased maintenance and less dependency on each receiver.

This pattern is implemented in Azure by default in both the Topic and the Queues.  When you create a subscription and a receiver for a topic you are registering with the service and as part of that registration process you also provide rules or conditions for the types of messages or the value of properties that you want to filter for. 

But what if we want to take this a step further and provide even more dynamic functionality.  What if we could implement the functionality that is available in BizTalk called Promoted Properties but do it completely within the functionality of Azure.  Promoted Properties is a way of taking data contained in the body of an object or message and put that in a properties collection that can then be used to route messages.  When working with Azure, the BrokeredMessage object contains a properties collection that can be used to route however, you manually have to set these properties at design time.  Our goal is to take the values of properties in the object body itself and promote those to the BrokeredMessage properties collection so that they can be assigned and routed at run time.

The subscribers on the Topic still need to be created and registered but this functionality allows the data in the message to direct the flow of the message and provides further dynamic routing scenarios. 

I have been working with Michael Muta (MCS Consultant) and he came up with a great implementation of this pattern.  Lets look at how this can be accomplished. 

First, we would create a new class that will be serialized as the message body of the BrokeredMessage.  In this case we will create a class for membership.

    [Serializable]
    public class Member
    {
        public Member()
        {
            
        }

        public bool NewMember { get; set; }
        public string Name { get; set; }
        public string Address { get; set; }

        [PromotedProperty("ReceiveEMail", PropertyDirection.ReadAndWrite)]   
        public string ReceiveEmailAds { get; set; }

        [PromotedProperty("Licensed", PropertyDirection.ReadOnly)]
        public bool Licensed
        {
            get { return (....); }
        }
    }

As you can see we have decorated a number of the properties with custom attributes.  We decorate every property that we will want to add to the BrokeredMessage properties collection.  In addition, we can set the PropertyDirection to either ReadOnly or ReadAndWrite.  ReadOnly certainly makes sense – we just need to take a value of the property and promote that into the collection.  However, we can take this beyond just static promotion.  With the ReadAndWrite we can provide the functionality to update the property as the message is received from the Topic.  With the Subscriptions on our Topics we have the ability to perform an action in the rules.  In that action we can set the value of a property, and in this case, we can push that modification all that way back to the internal property on the object.  We can then use that for other downstream actions in our processing of the message.  This give us a lot more flexibility and functionality in our solution. 

After we have our object decorated we can put together the code to send the message.  We can take the same method that we used earlier to submit a message but we will do a couple of different things. 

As you will notice, the code to submit the message is pretty much the same – we create the factory, create the message sender, create an instance of our member object and create a BrokeredMessage.  In order to get our promoted properties code to work as an extension to the BrokereMessage we needed to utilize the Extension Methods functionality in C#.  We needed to do this because the the BrokeredMessage class is sealed.  One of the drawbacks of the Extension Methods functionality is that it doesn’t work on the objects constructor.  Therefore, we need to call the PromoteProperties method that is placed on the BrokeredMessage class.  By doing this, this method will interrogate all of decorated properties on your object and promote them. 

public static void SubmitMessageToTopic()
 {
 String issuer = ....;
 String key = ....;

 TokenProvider tp = TokenProvider.CreateSharedSecretTokenProvider(issuer, key); 
 Uri uri = ServiceBusEnvironment.CreateServiceUri(.., .., ..);

 MessagingFactory factory = MessagingFactory.Create(uri, tp);
 MessageSender myTopic = factory.CreateMessageSender("<TopicName>");

 //Create our message body object 

 Member memberItem = new Member();

 BrokeredMessage message = new BrokeredMessage(memberItem); 

 //Now that we have created the message and passed in our object we need to do one more thing 

 message.PromoteProperties(memberItem); 

 //even thought we are promoting properties, we can still add properties at design time as usual as shown below

 //message.Properties.Add("MemberType", Gold);

 myTopic.Send(message);

}

One other item I want to mention in the code above is that even though you promote properties decorated in your object you can still add properties to the BrokeredMessage properties collection at design time as usual.

So, we now have our object decorated and we have the code to submit the message to the Topic.  Lets look at the extension method that promotes the properties.

public static void PromoteProperties(this BrokeredMessage msg, object serializableObject)
        {
            PropertyInfo[] properties = serializableObject.GetType().GetProperties();

            foreach (PropertyInfo prop in properties)
            {
                foreach (PromotedProperty promotedProp in prop.GetCustomAttributes(typeof(PromotedProperty), true))
                {
                    object value = prop.GetValue(serializableObject, null);
                    msg.Properties.Add(promotedProp.Name, value);
                }
            }
        }

  Recipient List

This pattern is related to the Content Based Router pattern in that the CBR pattern routes messages based on the message content.  The Recipient List pattern provides a means for the message to contain a list of one or more recipients to receive the message.  This differs from CBR in that we may want to submit a message to multiple order fulfillment suppliers in order to receive back prices thus allowing for the comparison of responses and the selection of the cheapest supplier as shown in the diagram below:

This pattern can be implemented in the Azure Service Bus Topic very similarly to that of the Content Based Router example.   The difference will be that we will create the subscriptions using the SQLFilter but will utilize the LIKE keyword.  This way the sender can populate the property with a comma separated list and the subscriptions will pick it up and route to N number of receivers.

So, if we created three subscriptions

topic.AddSubscription("OtherMemberTypes", new SqlFilter("MemberType LIKE ‘%Premium%’"));

topic.AddSubscription("OtherMemberTypes", new SqlFilter("MemberType LIKE ‘%Gold%’"));
topic.AddSubscription("OtherMemberTypes", new SqlFilter("MemberType LIKE ‘%Silver%’"));

and we wanted to create a message that would be routed to Premium and Gold we would create a BrokeredMessage and set the following property:

message.Properties.Add("MemberType", “Premium,Gold”);

This would allow us to set, at the client,  the recipients that we wanted to send the message to.

  Splitter

This pattern provides the ability to break apart a composite message into its separate message parts and process the related message parts separately.  This pattern is useful in scenarios where you might have an order that has many order items and each of the order items need to be sent to different receivers or sent to different inventory systems.  

This pattern can be implemented in Azure using the Service Bus Topic or Queue by utilizing the sessions feature.  To setup a queue or topic for sessions we need to set the RequiresSession property in the QueueDescription or SubscriptionDescription.  Then each message needs a SessionId in order to relate the individual messages.  The receiver will read a message and accept the session.  Once the session is accepted, the session is locked and other receivers won’t be able accept it.  This ensures that there are no competing receivers.  One thing to note about sessions and Azure is that session ordering is guaranteed even if the receiver crashes.  Another thing to think about is this pattern comes in handy when you need to provide support for sending messages larger than the allowed message size.  To send larger message, you can chunk the message into smaller sizes and use the session id in order to receive all the messages in the same session and then reassemble them.

There is a great example of this that Alan Smith has put together so I won’t duplicate the effort.  Take a look at his blog for an example of implementing this pattern.

  Aggregator

This pattern is related to the splitter pattern.  Whereas the splitter processes separate messages, the aggregator is used to bring all the separate messages back together.

This pattern can be implemented in Azure using the Service Bus Topic and correlation.  When setting up the subscription you can use the CorrelationFilterExpression just as we did with the FilterExpressions in the code samples shown above in previous patterns.  So the submitter would set the CorrelationId property of the BrokeredMessage object to an identifier.  Then this same identifier would be used in the CorrelationFilter object that was configured when creating the subscription.  Also, you will want to know when the processing is finished for this correlation set.  You could easily create a custom property on the BrokeredMessage object (as we did previously with the MemberType property) that lists the total number of messages that are in the set or you may just decide to create a flag that states that this is the last message in the set.  Once the total count has been received or the message containing the flag is received then you can perform whatever task is required to aggregate all the messages together.

You might ask, why use Correlation over Session.  The CorrelationId property provides an exact match, the CorrelationId is saved in a hash table and the CorrelationId processing is better optimized for matching performance than the SQL expression filters.

There is a great example of this that Alan Smith has put together that is in the same article as his Splitter implementation so I won’t duplicate the effort. Take a look at his blog for an example of implementing this pattern.

  Resequencer

The Resequencer pattern provides guidance around putting messages back in order.  Individual messages may be processed by a downstream system and returned faster than other downstream systems can process their messages and thus leading to messages that are out of order.  This pattern uses a buffer to store messages so that they can be resequenced.  Typically the Resequencer does not modify the message in any way.  Once in the required order, the messages can be delivered as shown in the following diagram. 

This pattern can be implemented in Azure through a Service Bus Queue.  The queue becomes the buffer to not only store the messages that are submitted but also to store the messages that are out of order.  There are a number of steps that will be required to put this pattern in practice.  First, there is no change to the code that is used to submit the messages to the queue however you will need to populate the MessageId property of the BrokeredMessage object.  While there is nothing out of the ordinary to submit, there is a bit of work to be done by the receiver.  Next, we are going to take advantage of the Defer() method which will allow us to put a message back on to the queue that we detect is in the wrong sequence.  Then we can dequeue it later when we need that message by calling the receive message and passing in the message sequence.  This means that we need to keep a list of message sequence numbers in our code.  One thing to note is that once we defer a message and put it back on the queue the only way that we can retrieve it again is to call the receive and pass in the sequence number.  Another thing to note is that the Service Bus Sequencing may be different than the applications sequence numbering.  Therefore you may need to store both in either session state or an internal list.

Lets look at the code to put this pattern into place. 

First, let’s look at what is needed to send the message to the queue.

 string issuer = "....";
 string key = "....";

 //create the token provider and URI that are needed to send to the MessagingFactory
 TokenProvider tp = TokenProvider.CreateSharedSecretTokenProvider(issuer, key); 
 Uri uri = ServiceBusEnvironment.CreateServiceUri("sb", "<ServiceNamespace>", string.Empty);

 //retrieve the MessageSender for the SeqQueue within the namespace
 MessagingFactory factory = MessagingFactory.Create(uri, tp);
 MessageSender SeqQueue = factory.CreateMessageSender("SeqQueue");

 //create a message and use a string for the body.  We will put the sequence here as well as in a property.
 BrokeredMessage message1 = new BrokeredMessage("MessageSequence 1”);

 //set some additional custom app-specific properties
 message.Properties["MessageNumber"] = "1";
 message.Properties["TotalMsgsInSequence"] = "3";   
 //send the first message to the queue
 SeqQueue.Send(message1);

//create the next message in the sequence. This message will be out of order.
 BrokeredMessage message3 = new BrokeredMessage("MessageSequence 3”);
 message.Properties["MessageNumber"] = "3";
 message.Properties["TotalMsgsInSequence"] = "3";   
 //send the first message to the queue
 SeqQueue.Send(message3);

//create the last message in the sequence.
 BrokeredMessage message2 = new BrokeredMessage("MessageSequence 2”);
 message.Properties["MessageNumber"] = "2";
 message.Properties["TotalMsgsInSequence"] = "3";   
 //send the first message to the queue
 SeqQueue.Send(message2);

Now we have three messages in our queue that are out of order.  Next, let’s look at what is needed to receive the message and process them in order.  By the way, I will be utilizing the QueueClient object as this supports more advanced features (including sessions).

private static void ReadMessagesFromQueue()

{

.... 

// Read messages from queue until queue is empty: 
 
MessageReceiver Qreceiver = queueClient.CreateReceiver(); 
List<MessageReceipt> MessageReceiptsDeferred = new List<MessageReceipt>(); 
 
BrokeredMessage receivedMessage; 
while (Qreceiver.TryReceive(TimeSpan.FromSeconds(10), out receivedMessage)) 
{ 
     

  msgCount++;
  //check the received message and determine if it is in the sequence we need 
  if (receivedMessage.Properties["MessageNumber"].ToString() == msgCount) 
  { 
     
     ProcessEachMessage(receivedMessage); 
  } 
  else 
  { 
     receivedMessage.Defer();

     //store the message receipt for later 
     MessageReceiptsDeferred.Add(receivedMessage.MessageReceipt); 
  } 
} 
 
 
//we have now processed messages 1 and 2.  We need to go back and get message 3. 
foreach (MessageReceipt receipt in MessageReceiptsDeferred) 
{ 
      ProcessEachMessage(Qreceiver.Receive(receipt)); 
} 
 
 
Qreceiver.Close(); 
 
 

}

private static void ProcessEachMessage(BrokeredMessage message) 
{ 
         //This is where we do the work we need to on the messages (i.e., reassemble, send off to a LOB system, etc.)

         //Do your work here

       
         message.Complete(); 
} 
 

This example shows you how we can put this pattern into code.  As with all code examples, there is still work that needs to be done to this code if you are going to use this as a sample. 

As you think about putting this pattern into practice, it could be useful for situations in which you want to process messages that have high and low priority all on one queue.  You may not know what order the messages are submitted but you can receive all the messages, process the ones marked as high priority and defer the other submitted messages and then go back and process those messages after all the high priority messages have been drained.

  Composed Msg. Processor

This pattern provides guidance for message processing where there are a number of sub messages (i.e., order line items) that need to be routed to individual destinations, processed, returned and aggregated back into a single message.  This pattern is composed of a number of other patterns that include the Splitter, Content Based Router and Aggregator.

Because this pattern utilizes patterns already covered in this article I won’t duplicate the code as that can be found in each patterns implementation.  However, I will include a diagram that shows how these patterns can come together to create this complex pattern.

Scatter-Gather

This pattern outlines how a single message can be sent to many receivers, or a message may be split and its parts sent to many receivers so that one of many external providers may process the message.  Once the message has been processed and returned then the messages need to be aggregated and brought back into one message that will be processed.  Think of an order that could be fulfilled by a number of suppliers.  You can scatter the messages to those suppliers and gather all the responses and decide if you are going to take the first message returned and continue processing or if you analyze all the responses and take the lowest cost supplier.  Either way, once you have received all the messages you will take the message and send it on to the next step in the process.

To put this pattern in practice we can take implementations of other patterns.  The scatter portion can be implemented through the use of the Splitter pattern whereas the Gather portion can be implemented through the use of the Aggregator pattern.  Both of which are covered above.

  Process Manager

The process manager pattern outlines how you send a message through a number of processing steps that aren’t known at design time. These steps may not even be sequential.  In addition, the sequence of the processing steps may also be modified based on the results of the previously executed processing step.

To put this pattern into practice you can utilized Windows Workflow.  There are a number of blog articles showing how you can implement the currently released version of WF 4 in your code and deploy it to Azure.  In addition, there was a CTP of WF in Azure that was released last summer.  Lastly, there is also the Workflow Activity Pack for Windows Azure that allows you to run WF on-premises and interact with artifacts in Azure.       

However, there is very little that shows how these patterns can be implemented using the Windows Azure Service Bus.  This two part series will take a patterns first  look at how these different patterns can be implemented. 

The patterns that will be covered in this post will focus on the Messaging Channel groupings.  The patterns in that fall in the Messaging Channel group contain:

Point to Point, Publish/Subscribe, Datatype Channel, Invalid Message Channel, Dead Letter Channel, Guaranteed Delivery, Channel Adapter, Messaging Bridge and Message Bus.  

In the next post we will focus on the patterns that fall in the Message Routing group, which contain:

Content Based Router, Message Filter, Recipient List, Splitter, Aggregator, Resequencer, Composed Message Processor, Scatter-Gather, Process Manager and Message Broker.

There is also the Messaging Systems group of patterns but those are being addressed with the Azure Integration Components (that is currently in CTP).  As those components get closer to release I will cover those patterns as well.

Lets start out by looking at each of the patterns in the Messaging Channel Group including a description of the pattern and then how it can be implemented.

Point-to-Point Channel requires that only one receiver gets the message.  If there are multiple receivers, then only one must still get the message.  This functionality can be done directly through WCF (the Service Bus can also be included by hosting the WCF service).  This pattern can also be accomplished through the use of the Service Bus Queue and controlling what services are listening.

Publish-Subscribe Channel requires that the message or event sent to the channel will be delivered to each and every receiver.  Once the receiver obtains a copy of the message it is removed from the channel.  This can be implemented through the use of Service Topics. 

A Service Bus Topic can have multiple subscriptions with each subscription being able to be individually managed.  The subscriber receive all messages, based on filters, that are sent once the subscriber is created.  If a message satisfies the filter criteria for more than one subscriber the message will be delivered to each subscriber and then onto the receiver.  When a message needs to be delivered to more than one subscriber, the message is not duplicated.  The meta data for the filter/subscriptions is stored and is tracked based on which subscriber/receiver has received the message.  In terms of removing it from the channel, think of the concept of ref counts.  One other thing to keep in mind is that if the Receiver is not online to receive the message, then it is stored until the receiver comes on-line.

Datatype Channel requires that each channel will contain the same type of data.  Thereby when the receiver obtains a message from the channel it will always be the same type.  This can be accomplished with directly with WCF (the Service Bus can also be included by hosting the WCF service).  

In addition, an Azure Queue can also be utilized.  You can create a new queue for each type of message that will flow through the queue. 

Invalid Message Channel requires that a message that is deemed to be invalid be moved to a Invalid Message Channel where they will not be processed by the normal receiver.

The Azure Queue implements the dead letter queue concept and uses the $DeadLetterQueue subqueue to store invalid messages.  Messages are placed in the dead letter subqueue:

When a message expires and deadlettering for expired messages is set to true in a queue or subscription.
When the max delivery count for a message is exceeded on a queue or subscription.
When a filter evaluation exception occurs in a subscription and deadlettering is enabled on filter evaluation exceptions

To get to the dead letter queue you would use:

the same behavior is available with Topics.  However, to get to the dead letter queue for a topic you would use:

string subscriptionDeadletterPath = SubscriptionClient.FormatDeadLetterPath("<Your Topic Name>", "<Subscription Name>");

Dead Letter Channel requires that if the system can’t deliver a message there is a mechanism to move the message to a dead letter channel.  This capability is handled like the Invalid Message Channel above.  However, check out this link from MSDN on the QueuePolicy.PoisonMessageDrop property.  It points to the prerelease documentation but it is interesting to read what is being planned. 

Guaranteed Delivery requires that the system persists the message so that it is not lost if there is a crash or outage.  In using the Service Bus Queues, you are ensured of guaranteed delivery as the queue uses SQL Azure as the backing store.

Channel Adapter requires that there is an interface so that any application can connect and be integrated with other applications.  The Service Bus Queues and Topics both provide a full API that can be utilized by a number of different technologies.

Messaging Bridge provides a connection between different messaging systems.  Essentially to transmit message that was intended for another messaging system.  This can be done through either the Service Bus Queues or Topics with the receiver getting the message and forwarding it to endpoints provided by the other messaging system.  In addition, the receiver can get the message and forward it on to on-premise messaging systems through WCF.

Message Bus provides a means for separate application to work together in a decoupled manner with the ability for the applications to be added or removed with no affect on any other applications.  The ability to be decoupled but still allowing the application to work together, share data and operate in a unified manner is the basis for this pattern.  This pattern can be easily implemented through the Azure Service Bus and WCF endpoints.  We will come back to this pattern again when we spend more time covering the integration components.

In this post I have talked about the Service Bus Queue, but there is still some confusion between the Windows Azure Queue and the Service Bus Queue.  Yes, there are two Queues and they are different and have different uses and purposes.  Everything I have been referring to here is in reference to the Service Bus Queue.  You can find more information on the differences between Azure Queues and Azure Service Bus Queues in this article - Windows Azure Queues and Windows Azure Service Bus Queues - Compared and Contrasted.

Lastly, in talking about Gregor Hohpe and his patterns book, there are a number of diagrams that are utilized that represent the patterns.  These diagrams can be downloaded and used in Visio which you can find at https://www.eaipatterns.com/downloads.html

In my case, I saw this error, in the screen shot below, when I clicked on the URL for my service in the AppFabric portal.

Then I saw that I had the following error in the Event Log.

I knew why the error in the Event Log was showing up.  I have Windows Server AppFabric installed and had configured my service to Auto-Start.  What I didn’t know was why I was getting the underlying error of not being able to reach watchdog.servicebus.windows.net.  After doing a lot of searching on-line I couldn’t find anything that could solve the problem.

So, after talking to a member of the dev team who created the wizard functionality we found a fix.  It turns out you will get this error when the service can not communicate with Azure.  Since I had a proxy server in the mix, the fix was to add an entry in the config file in order for communications to flow through the proxy server.  When the wizard completed it created a set of artifacts in IIS including a config file.  Open that config file and we will add the entry below and place it just before the system.web section.

...
<connectionString />

<system.net>
   <defaultProxy useDefaultCredentials="true">
      <proxy proxyaddress="[your proxy address here]" />
   </defaultProxy>
</system.net>

<system.web>
...

Enter your proxy address and you should see this error go away.

So, as you build your composite applications, or just look to extend your current application, you may want to take advantage of the benefits offered by extending your application to Windows Azure.  This will let you leverage your existing on-premise artifacts, and LOB systems, in order to expose them and extend their reach to external clients in a secure manner.

Microsoft released the BizTalk Server 2010 AppFabric Connect for Services as a new feature for BizTalk 2010 which provides functionality to expose your BizTalk artifacts (schemas and Orchestrations) as WCF Services exposed through the Windows Azure cloud.  This new functionality is exposed through changes to the BizTalk WCF Service Publishing Wizard.  When you run this wizard you will see a number of new screens in addition to the screens that you are already familiar with.

To expose BizTalk services to Windows Azure, specifically Windows Azure ServiceBus, you will need to have a few prerequisite components installed. 

First, of course, you must have BizTalk 2010 and the BizTalk development tools installed on the machine where the AppFabric Connect for Service feature will be used.

Next, you must have the Windows Azure AppFabric SDK (1.0 or later) installed.

Lastly, there are benefits of having Windows Server AppFabric installed.  While this is not a requirement, it makes configuration setting easier (such as Auto-Start).

After all the pre-requisites have been installed, you will need a Windows Azure account and you will need to create a namespace for the Service Bus in the Azure Portal. 

Once this is done then you can start the wizard and walk through each of the screens.  The wizard walks you through selecting the type of endpoint (service or metadata) – as it did previously.  The next screen will be a new screen which will fork between the functionality of the old wizard (exposing web services on-premise only) or to sprinkle in the new screens to allow you to expose the services to the cloud.  This screen presents a single checkbox asking if you want to add a Service Bus endpoint.  It also reminds you that you need to have a ServiceBus service namespace.  Select the checkbox (otherwise there is really no need to keep reading this post).  The next five screens are the same as the screen in the original wizard (select either schemas or Orchestrations, select the assembly, select the public ports, assign the target namespace (and remember to check allow anonymous access)).  One thing that I found very helpful is that the wizard will take all of the selected ports and merge them into a single WCF service.  The last two screens are specific to the configuration of the Service Bus. 

The first is the Endpoint Configuration.

There are a number of options that will effect the output of the wizard.  The first is the Relay Binding option and you have three choices – BasicHTTPRelayBinding, NetTcpRelayBinding, and the WS2007HttpRelayBinding.  The Service Namespace is the namespace that you created in the Azure portal.  I am sure that this goes without saying but what you enter here must match with the name you created in the Portal. 

The Enable Discover checkbox lets the services you are exposing be publicly discoverable through the ATOM feed page created by Azure for your Azure Service Bus account. 

Lastly, the Enable metadata exchange on cloud tells the wizard to create the metadata endpoint in the cloud so that external clients can generate a proxy for your service.

The next screen takes the authentication information that you can find on the portal site. 

You will need to enter the Issuer Name and Key.  When you created the namespace in the Azure ServiceBus the issuer name defaults to owner.  Enter both the Name and Key into the text boxes.  The first checkbox controls whether the consumer needs to authenticate using to the relay service to consume the endpoint.  The second checkbox controls the same but for the metadata endpoint.  Since you are charged per access to your endpoints you may want to set these to prevent unauthorized access.

Click Next and Finish to complete the wizard.  Once it is finished you will have a new Receive Port/Location and a site in IIS.  However, there are still a number of important steps that need to be done in order for your service to work.

If you created a service based on an Orchestration then you need to bind the port to the orchestration in the BizTalk Administrator.  You will also need to Enable the Receive Location and Enlist the Orchestration.

Next we move to IIS.  We need to have an Application Pool that is configured to use the .NET Framework 4.0 with the Pipeline set to Integrated.  Set the site to use the Application Pool by right clicking on the site that was created.  In the popup menu, select Manage Application and then click on Advanced Settings.  When the Advanced settings dialog box appears change the Application Pool to use the one you just created.

Lastly, we need to start our service.  If you are running IIS 7.5 (Windows 7 or Windows Server 2008 R2) then you can configure Auto-Start.  If you are running a different OS version then you will need to start the services using the on-premise endpoint with a local client.  Once the service is started it will make contact with Windows Azure and register the service.  One thing to note is that this is the first time in the entire process that your service has communicated with Azure.

If you have Windows Server AppFabric installed then configuring Auto-Start if very simple.  To do this, right click on your site and select Manage WCF and WF Services in the popup menu, then click the configure item in the fly out menu.  When the dialog box appears, select Auto-Start from the left hand side and then click the Enabled radio button.

You are now ready to test your endpoint.  Point your browser to .servicebus.windows.net">.servicebus.windows.net">https://<servicebusnamespace>.servicebus.windows.net.  You should see the ATOM feed page and your service should be listed.  You can click on the service and you should see a new page with your service and MEX endpoint.  You can now consume this endpoint the same is if it were on-premise.

The answer is yes it can be done.  Movement from Hyper-V to VirtualPC is not as easy (and many times not possible) but there are a number of blog posts on this topic already so I won’t cover that here.  One of the reasons that it is easier to move from VirtualPC to Hyper-V is that VirtualPC is only 32 bit. 

There are a number of steps that must be occur for a successful move.

First, start by uninstalling the integration components while the virtual machine is running in VirtualPC.  You can do this through the Add/Remove programs feature in Windows in the Virtual Machine.  The Hyper-V drivers and additions will not install over the VirtualPC additions and that is why you must remove them first.

Next, move the vhd file to a location where it can be accessed by your Hyper-V instance.  Walk through the wizard to create a new virtual machine but when prompted to create a new drive or select and existing drive, pick select an existing drive and point it to your .vhd file.

Finally, once you have the virtual machine configured in your Hyper-V instance then start the machine.  Go through the Settings Menu and install the Hyper-V additions.  Once you do this, the Hyper-V additions installs a new HAL as well as new drivers for network, video and sound devices.  The process of installing the new HAL is one of the reasons that a Hyper-V image is no longer portable back to VirtualPC.

However, at this point, you might think that everything is done and you are ready to use the virtual machine.  Most of the time this is correct, however, there are situations that require additional steps.  You will know that you have additional steps if your integration components aren’t working – you can tell really quickly if your mouse doesn’t move outside of the virtual machine.

You are more likely to have this occur if your virtual machine is running versions of Windows prior to Vista or if you are running Windows Server 2008 as these do not have the ability to dynamically detect the HAL at boot time.

So, to fix this run MSConfig.exe – by clicking the Start menu, selecting Run and typing msconfig.  Once the utility launches, click on the Boot tab and click the Advanced Options Button.  When the BOOT Advanced Options dialog appears, click the Detect HAL checkbox and hit ok.  Restart the virtual machine and you should be good to go!

One of the first questions that people ask is what is a composite app? 

Composite apps can take on many forms and use many differing technologies.  Many times people will talk about composite Apps and will only be referring to the UI tier.  While the concept of a UI composite can be compared to a mashup (a UI that combines publically available web based services to create a meaningful combination of information to a user) a UI composite is built to use business sources – in many scenarios these sources (assets) already exist within the organization from previous application development efforts. 

A composite app can also refer to just the middle tier as well.  Many companies have spend a lot of time and money over the past many years building out SOA services.  A composite app can consist of orchestration services that draw on the functionality of these different services.  One thing to keep in mind however, is that a composite app is not make a SOA architecture.  Instead they they will build on top of the work that has already been done to create a SOA architecture. 

The idea of a composite app is to build upon existing software assets to quickly bring data to consumers in new way. 

Many of the articles and blogs on this subject talk a lot around the UI tier.  While that is a very important part of a composite application, I am going to spend time over the next few posts talking about composites at the middle tier.

There is a very good all-up article titled What Are Composite Applications from Atanu Banerjee where he walks through the full composite application stack.  In this article he outlines three steps that a solution architect must do to design a composite application.  Those things are to:

• Choose a composition stack. Pick one or more containers from each tier, and a set of component types that are deployable into those containers.
• Choose components. Define the repository of assets that must be built from this set of component types, based on business needs.
• Specify the composite application. Define the ways in which those assets will be connected, to provide a particular cross-functional process. The platform should enable these connections to be loosely coupled.

Each of these things are required so that the consumer can get the most benefit.  Over and above this, the organization should start to benefit by realizing faster time-to-benefit turnaround as well as faster time-to-deployment.  These two, along with exposing existing LOB data are the three driving factors that are making the idea of composite applications so popular.

In my next several posts, I will cover some of the technical choices, how to bring bridge between services that live in BizTalk and services that live in AppFabric, and some of the lessons learned.

The meeting takes place at 6:00 p.m. at the Microsoft offices at 8300 Norman Center Drive, Bloomington, MN 55437.

Andy Morrison will be presenting on a Lap Around BizTalk 2010 and the ESB Toolkit 2.1