Let me first start with BizTalk Server. The diagram below tries to show BizTalk Server in terms of its components / capabilities. Not everything is shown here – just what is relevant to this post.

This is how BizTalk Server has been since 2004. The capabilities / components have been enhanced, but the overall architecture has remained the same.

Since then, there have been changes in other parts of the Microsoft ecosystem.

The first change was introduced with .NET 3.0. That was – the introduction of WCF. WCF was the new communication framework – if you need to implement a component to abstract your transport protocol, then WCF was the way to do it. BizTalk Server had its own Adapter Framework for developing Adapters, and the obvious limitation was that they could work only with BizTalk. WCF allowed you to write “Adapters” that could run in any .NET application. With BizTalk 2006 R2, we tried to bridge the difference between the frameworks, with the introduction of the WCF Adapter in BizTalk 2006 R2, and the WCF LOB Adapter SDK to make it simpler to write transports for LOBs using WCF.

The next change – Workflow – was also introduced with .NET 3.0, and was then revamped in .NET 4.0. Recently (post .NET 4.5) we shipped the Workflow Manager, which is a host for running Workflows (though the workflows that run here have some differences with respect to what they can do compared to if you use the execution options that ship with .NET (i.e., WorkflowInvoker, WorkflowApplication, and WorkflowServiceHost). BizTalk Server had its own framework for designing workflows for modeling the message flow across endpoints – orchestrations. But again, the biggest limitation is that this was tightly coupled with BizTalk (and can also be said that the other big limitation is that it hasn’t evolved much since its original incarnation). While, .NET Workflows can be used in any .NET application, and they’ve been evolving with every .NET release.

The next change was introduced somewhere around … I think the 2007 timeframe. You may remember it first being known as BizTalk .NET Services. This was essentially the Relay capability which is there in Service Bus today. Soon after was introduced the concept of a MessageBuffer, which was some sort of a “queue in the cloud”. All this ultimately evolved to what we know today as Service Bus, which comprises of Relay, Queues and Topics. A Message Bus. BizTalk Server had its own version of a pub-sub system – the MessageBox – but not only was this tightly coupled with BizTalk (just like Adapters and Orchestrations), but this wasn’t even something which developers/customers could directly use – you only worked with it indirectly when working with Send Ports (filters), Messaging (promoted properties), Orchestrations (filters). You knew it was there, but it wasn’t meant to be used directly from your code.

So, let’s see how the components listed above fit in.

Let’s now take a look at Windows Azure BizTalk Services. Here, we’re attempting to bring Integration to the cloud, a small set of components / features / capabilities at a time. We’ve release version 1.0 just recently (we announced the public preview just a few days ago), and hope to announce GA soon.

The above picture captures the components unique to BizTalk Services. One thing you’ll notice is that the position of the box that represented the “Message Bus” has changed. The reason for this is – Bridges in Windows Azure BizTalk Services aren’t built on top of a “Message Bus”. BizTalk Services can of course interact with the other components in Azure – so, we can interact with the “Message Bus” component in Azure, which is – Service Bus. Keeping with the current component pictures, we now arrive at this:

Over the course of the above pictures, it should now be clear how each of the technologies mentioned here is meant to be used. Depending on the capabilities that you need, the appropriate technology/product can be picked.

It’s also easy to hypothesize how BizTalk Services will evolve:

Assume you have 5 ordered send ports (SP1 through SP5) running in a host H1, and you have messages for each of these send ports available to be dequeued. In prior versions of BizTalk, the behavior you may have observed is that if there is a message M1 available for SP1, and another for SP2, then SP2 begins processing M2 only after SP1 has completed processing M1. The workaround we would propose to customers was to place these 5 send ports into 5 different hosts. This was feasible only if the number of ordered send ports was small.

In BizTalk 2010 R2, we've fixed this problem. No matter how many ordered send ports you have in a host, each will process its messages without blocking other ordered (or non-ordered) send ports. We use a separate thread for each ordered send port, and if you have (say) 500 ordered send ports in a single host, then you'll need to increase the Messaging Engine Threads setting so that we can spin up the necessary number of threads for you.

Please let us know if you have any suggestions / feature requests for adding more platform-level features in the adapter. When adding your comments, if you leave behind a way to contact you, we can get in touch with you when we begin to explore and think about your suggestion.

Thanks.

This post is to inform you all that the workaround has been incorporated into the WCF Adapter product code. Henceforth, it will be possible to set up a 1-way Send Port which uses a binding such as BasicHttp (or the WCF-BasicHttp adapter) and invoke a 1-way operation on an external service. This feature will be present in the next version of BizTalk scheduled for release in the first half of 2013 (a CTP of which will be available within the next few weeks).

Here’s what I had:

Workflow:

1. Has a number of activities executing in sequence. None of the activities are of type AsyncCodeActivity, and none of the activities create Bookmarks.

2. The Workflow starts off with a Receive, and has no SendReply.

3. Workflow hosted in a Workflow Service Host.

Endpoint:

1. The binding used was a custom binding, using the NamedPipesTransportBindingElement. The TransferMode property was set to Buffered.

Client:

1. Client creates an IDuplexSessionChannel.

2. Client invokes channel.Open().

3. Client invokes channel.Send(Message).

4. Client invokes channel.Close().

Expected Behavior:

Client:

1. Channel.Close() should return immediately (since when channel.Send() has returned, it should imply that the message was dispatched to the service).

Observed Behavior:

Client:

1. Channel.Close() gets blocked until the Workflow completes.

Interpretation:

Since a session is used, when the client invokes channel.Close(), the session needs to be closed. This involves a call being made to the WCF Service (the workflow in this case), and the Service needs to somehow “respond”. Since the workflow was busy executing synchronous activities, it wasn’t able to respond, and thus Close() got blocked until the Workflow Runtime became “available”.

Workarounds (each of these is individually a workaround):

Workflow:

1. Add a Delay activity with Timespan of at least 1 millisecond. This causes a bookmark to get created, and the workflow runtime is free to do something else until the bookmark resumes. (Of course, the assumption here is that the client’s channel.Close() call reaches the workflow service before the bookmark got resumed).

2. Change one of the activities (preferably the one just after the Receive) to be an AsyncCodeActivity (assuming it does not complete synchronously). Again, this frees up the workflow runtime.

Endpoint:

1. Use a Binding which does not require sessions (e.g. one of the Http Bindings).

2. Change the TransferMode setting to Streamed on NamedPipesTransportBindingElement. You can now use an IRequestChannel instead of IDuplexSessionChannel. Since your workflow doesn’t have a SendReply, the response Message returned when you invoke channel.Request(Message) will be null.

Client:

1. Use Channel.BeginClose() instead of Channel.Close(). This way, your main thread doesn’t get blocked.

Before your read further – please note that the technique mentioned here for scaling out dynamic send ports is UNSUPPORTED by the Product Team. I’ve gotten this to work based on my understanding of the BizTalk Management Database and Message Box Database – however, my understanding could be wrong leading to issues in your scenario. Before executing these steps in a Production Environment, you should make sure that you have thoroughly tested this out in your test environment.

As far as Dynamic Send Ports are concerned, BizTalk has one big limitation – all the Dynamic Ports configured only execute on the Default Hosts for each Adapter. That is, if you have 15 Dynamic Send Ports, and on each Port you send messages for, say, the SQL, FILE and FTP Adapters, then all messages targeted for the FILE Adapter run on the same Host (whichever the default Send Handler is for the FILE Adapter), all the messages targeted for the SQL Adapter run on whichever host is the default Send Handler for SQL, and similarly for the FTP and all other Adapters. For Scale-Out, what you really want is to be able to say – for Dynamic Send Port 1 for the SQL Adapter, use Host 1, while for Dynamic Send Port 2 for the SQL Adapter, use Host 2.

I’ve written a small utility (.cs as well as .exe included in the .zip file) that allows you to do this. Here’s how you use the tool:

1. Create your Dynamic Send Port. Enlist it, but do not start it (let it remain in the Stopped State)
2. Run the tool, specify the name of the Port, the Adapter Type, and the new Host which you want to use.
3. Start your Dynamic Send Port.

For Dynamic Send Ports which already exist on your system:

1. Ensure that there are no active/queued/suspended/etc messages for this Port – in short, this Port should not be currently in use.
2. Stop the port (let it remain in the stopped state) – do not un-enlist it.
3. Run the tool.
4. Start the Port.

The tool accomplishes its objective by modifying the Subscription set up for the Dynamic Send Port, to point to the specified Host instead of the Default Host for that Adapter.

NOTE – If you ever delete a Handler / Host which was associated with a Dynamic Send Port using the utility above, then you will need to re-run the utility to specify a new Handler / Host for the specific Adapter for this Port. To help you with this, I’ve also written another tool which will inspect all the Dynamic Send Ports, and let you know if any port has been associated with an incorrect Host for a specific Adapter Type (incorrect meaning either the Host no longer exists, or there is no handler configured for that Adapter + Host combination).

You can download the utility here.

1. You have an adapter on a Receive Location, which submits a message in Format A into BizTalk.
2. The Pipeline configured on this Receive Location converts the message into Format B.
3. Assume that there is no map configured on the Receive Port.
4. You have the “Route Failed Messages” option enabled on the Receive Port.
5. You have set up an orchestration (or Send Port) which subscribes to the Error Reports (i.e., the failed messages which are routed due to the “Route Failed Messages” feature) generated by this Receive Port 1.

Problem: When a message is received on this Receive Location, and there are no subscribers for it, you expect that the message in Format B will be received by your error-handling orchestration (or Send Port). However, in some scenarios (depending on the pipeline, pipeline configuration, etc), you see that the message received by the error-handling orchestration (or Send Port) is in Format A. How come?

Here’s what is most probably happening:

1. Adapter submits a message in Format A.
2. The pipeline executes, and the message is converted to Format B.
3. The Messaging Engine attempts to publish the message, but finds that there are no subscribers for this message. It thus returns an error back to the Adapter.
4. The Adapter attempts to suspend the message. NOTE – here, it attempts to suspend the message which it knows about (which it received from the back-end) – this message is in Format A.
5. The Messaging Engine sees this:
1. someone (the adapter in this case) is suspending a message (message in Format A, in this case)
2. “Route Failed Messages” is enabled on the port
6. The Messaging Engine, instead of suspending, decides to “route” an Error Report corresponding to the message being suspended. This is the message in Format A, in this case.
7. Your error-handling orchestration (or send port) thus ended up receiving the message in Format A.

Question: What do you do if you want to have the message received by your error-handling orchestration (or send port) to be in Format B?

Answer: The output message from the pipeline must have the context property “SuspendMessageOnRoutingFailure” present, with a value of true (boolean).

How does that work? Well, it changes the sequence above to be as follows:

1. Adapter submits a message in Format A.
2. The pipeline executes, and the message is converted to Format B.
3. The Messaging Engine attempts to publish the message, but finds that there are no subscribers for this message.
4. The Messaging Engine sees the “SuspendMessageOnRoutingFailure” message context property is set to true, and instead of returning an error back to the Adapter, attempts to suspend the message. NOTE that this message is in Format B.
5. At the same time, the Messaging Engine also sees this:
1. someone (the Messaging Engine itself, in this case) is suspending a message (message in Format B)
2. “Route Failed Messages” is enabled on the port
6. The Messaging Engine, instead of suspending, decides to “route” an Error Report corresponding to the message being suspended. This is the message in Format B.
7. Your error-handling orchestration (or send port) now receives the message in Format B.

The out-of-box XML Disassembler component, has an option named “Recoverable Interchange”. When this option is set to true, the XML Disassembler internally translates this to mean “SuspendMessageOnRoutingFailure”=true, and sets this context property to the message.

• If an out-of-box binding was used, then most of those bindings have a Transactions property. We would look at that to determine whether the property value was true or false.
• If a custom binding was used (either WCF's CustomBinding, or your own binding), we would get the binding elements from the stack, and see if certain binding elements were present, which we know control transaction flow. Based on the configuration of those binding elements, we would determine whether a transaction was required or not.
• If we couldn't figure out what to do, we would just play safe and use a TransactionScope.

NOTE that in the above three steps, step 2 is expensive for dynamic send ports, since for dynamic send ports, we would execute that step every time, and calling CreateBindingElements() multiple times can add up.

In BizTalk 2009, we went ahead and added two properties on the port - "EnableTransaction" and "IsolationLevel". If we find the EnableTransaction property set on the message, then we know whether to use a TransactionScope or not, and hence we don't need to execute those three steps above. However, if we don't find this message context property, then we fall back to the previous behavior.

Hence, for performance, when using dynamic send ports with the WCF-Custom adapter, when setting the context properties for BindingName and BindingType (amongst others), also set the EnableTransaction and IsolationLevel message context properties.

In this post, I’m going to port over a MSDN sample which uses the Ws2007FederationHttpBinding to BizTalk. You will need Microosft BizTalk Server 2009 R2 in order to get things to work correctly.

Firstly, you need the WCF samples from MSDN. You can get them from here. The MSDN documentation for these is available here. This is the sample which we will be using. Make sure that everything is working fine in the sample, and that you’re able to use the client WinForms application to successfully browse the books list, and are able to buy the 2nd and 3rd books (if you attempt to buy the 1st book, you should get back an error message complaining about the price being too high).

We’ll now attempt to bring BizTalk in the picture. First, we’ll make BizTalk act as a client and send a request to the server for purchasing a book. Next, we’ll make BizTalk act as the server and then use the same WinForms client application (present in the MSDN sample) to connect to BizTalk and purchase a book. NOTE that I won’t act BizTalk act as a client and send a request to the server for browsing the books list; since that uses the WsHttpBinding, and it is pretty straightforward to get that part working once you work on the second part of making BizTalk act as a client and purchase a book.

NOTE – when going forward, the assumption is that you are familiar with the MSDN sample – the code, and how it functions.

BizTalk as the Client – purchasing a book

We need to do two things before we start with the BizTalk side of things.

1. Run “certmgr”. Find the certificate issued to "Root Agency" (under Intermediate Certification Authorities\Certificates). Right Click, choose Cut. Paste it under Trusted Root Certificatioin Authorities\Certificates. (You can always revert this later).

The reason we have to do this – when you set up the MSDN sample, three certificates are created for use with the sample. These certificates are issued by “Root Agency”, which is a dummy CA for testing purposes, and it is not trusted. The WCF Adapter in BizTalk only accepts certificates whose CA is trusted; hence without this step, you’d get an Invalid Certificate exception.

2. In the zip file, you’ll find an endpoint behavior. You need to register it in machine.config, and add the assembly to the GAC. For registration in machine.config, add the XML below to the <system.serviceModel>/<extensions>/<behaviorExtensions> node.

<add name="federationSampleEndpointBehavior" type="FederationSample_EndpointBehavior.FS_EndpointBehavior_ExtensionElement, FederationSample_EndpointBehavior, Version=1.0.0.0, Culture=neutral, PublicKeyToken=fe278db62e2dcf8b" />

The reason we need this endpoint behavior: the way the WCF sample is written, the book which you want to buy – the name of that book must appear within the header of the message. The way the sample client does this is it sets the header on the endpoint – thus, all messages flowing out of that client endpoint will have the header applied to them. If you want to do this the same way in BizTalk, a custom endpoint behavior fits perfectly, allowing you to set the header on the endpoint which BTS will use.

NOTE – the code in the WCF sample client is as follows:

EndpointAddressBuilder myEndpointAddressBuilder = new EndpointAddressBuilder(myBuyBookClient.Endpoint.Address);

myEndpointAddressBuilder.Headers.Add(AddressHeader.CreateAddressHeader(Constants.BookNameHeaderName, Constants.BookNameHeaderNamespace, bookName));

myBuyBookClient.Endpoint.Address = myEndpointAddressBuilder.ToEndpointAddress();

and the custom endpoint behavior does the same thing.

Creating the Send Port in BizTalk

• Create an Application in BizTalk (from the BizTalk Server Administration console) named "FederationSampleApplication".
• Create a 2-way Send Port named "BuyBook_SendPort", choose WCF-Custom as the adapter. Click "Configure"
• In the Import/Export tab, choose Import, and browse to the app.config for the client WinForms application (for example, you would import the BookStoreClient.exe.config file from "C:\WCF_WF_Samples\WCF\Scenario\Federation\CS\BookStoreClient\bin" if you installed the MSDN samples to C:\WCF_WF_Samples).
• In the dialog for choosing an endpoint, choose WSFederationHttpBinding_IBuyBook.
• In the General tab, set the Action to https://tempuri.org/IBuyBook/BuyBook.
• In the Behavior tab, right click on BuyBookClientBehavior, and choose Add Extension. Add the federationSampleEndpointBehavior. Set the BookName property to "Book Title Two". (We’ll attempt to purchase this book).
• The port is now configured.
• Create a 1-way Receive Port named "FileIn", add a 1-way FILE Receive Location. Assume that the input folder is C:\IN.
• Create a 1-way Send Port named "FileOut" with the FILE Adapter. Assume that the output folder is C:\OUT.
• Setup a filter at the BuyBook_SendPort send port, with the condition being BTS.ReceivePortName == FileIn
• Setup a filter at the FileOut send port, with the condition being BTS.SPName == BuyBook_SendPort
• Start BizTalk and start/enlist/enable the artifacts. Drop the file BuyBook_Input.xml (this is present in the zip file) in the folder pointed to by FileIn; you should see the output in the folder pointed to by FileOut

One thing you must have noticed is that all messages sent out via the Send Port can only purchase a single book. In the MSDN sample, you were able to specify which book you wanted; and it worked there because for every request you made, you were recreating the client endpoint, and so setting the header on the endpoint worked. In BizTalk on the other hand, we don’t re-create the client endpoint for every call.

So, how do you work around it? Here’s one way – create your own schema containing three values: the emailAddress, the shipToAddress, and the bookName. Drop in an input message conforming to this schema. Use a map within an orchestration to transform it to another message containing only the emailAddress and the shipToAddress – since the sample service only expects these two to be present in the message body. Then, you can use an expression shape in the orchestration to set the value of the WCF.OutboundCustomHeaders context property to any custom header which you want to set – the bookName in this case. And of course, if you did this, you wouldn’t require the custom endpoint behavior anymore which we added earlier to the Send Port.

BizTalk as the Service

You must have seen that the sample consists of 3 services – the BookStore Service (which hosts the Browse and Buy endpoints), the BookStore STS, and the HomeRealm STS. I’m only going to host the Buy endpoint of the BookStore service in BizTalk, the Browse endpoint as well as the token services will yet continue to be hosted in IIS. Once you go through the rest of this port, it should be easy to figure out how the remaining endpoints / services can be “migrated” to BizTalk.

The .zip file contains a class library which houses two things – the code for the authorization manager, and a service behavior. The authorization manager code is more or less identical to what was there in the MSDN sample. But here’s why we also need another behavior: if you look at the BuyBook implementation in the sample, the code attempts to extract a specific claim from the incoming message and then use that for further processing. In BizTalk, if you want to access the WCF Message directly, you’d have to use a MessageInspector; and so we’re inserting a MessageInspector into the WCF Dispatch Runtime via a service behavior. You’ll need to register this service behavior extension in machine.config, have a look at the comment in the “FederationSample_ServiceBehavior.cs” file - this contains the machine.config entry. Compile and GAC this dll.

Create a 2-way Receive Port + Receive Location in BTS. In the Receive Location, import the web.config from the BookStoreService IIS virtual directory (C:\Inetpub\wwwroot\FederationSample\BookStoreService in my case). Select the Buy endpoint as the endpoint to import. Then, in the behaviors tab, you need to make the following changes:

• Change the authorization manager type to “FederationSampleServiceHelper.BuyAuthorizationManager, FederationSampleServiceHelper, Version=1.0.0.0, Culture=neutral, PublicKeyToken=fe278db62e2dcf8b”. (this is from our dll which we GACed above).
• Set ServiceCredentials -> IssuedTokenAuthentication -> CertificateValidation to PeerOrChainTrust (in the MSDN sample, this is being done in code in the BookStoreServiceHost constructor)
• Add the federationSampleServiceBehavior extension (so that our MessageInspector gets injected at runtime).

In the General tab, you also need to set a valid URI (by default, it gets set to “buy” which was the relative address used in the endpoint configuration when hosting in IIS).

Also, build the orchestration and deploy it. In the MSDN sample, three things were used from the incoming message to construct the response: the BookName header (present in the WCF headers), the name of the caller (present as a claim), and the email address (present in the WCF Message body). In my orchestration, I’m using a construct message shape to extract the first two (the last one is pretty straightforward – we just need to use XmlDocument to read the message body and then use an XPATH to get the email address out). The headers present in an incoming WCF Message are present in the InboundHeaders context property (the WCF Adapter sets this), and so the BookName header is readily obtained. Also, our message inspector code (when it ran during the message receive) did this – searched the claims in the message for the caller’s name (using the same code as there in the MSDN sample in the contract implementation) and put that claim in the message header; and so we are now able to extract this also from the InboundHeaders context property. The code in the expression box looks like:

inboundHeaders = Message_1(WCF.InboundHeaders);

xmlDom = new System.Xml.XmlDocument();

xmlDom.LoadXml(inboundHeaders);

nsmgr = new System.Xml.XmlNamespaceManager(xmlDom.NameTable);

nsmgr.AddNamespace("ab", "https://tempuri.org/");

n = xmlDom.SelectSingleNode("/headers/ab:CallerName", nsmgr);

callerName = n.InnerXml;

n = xmlDom.SelectSingleNode("/headers/ab:BookName", nsmgr);

bookName = n.InnerXml;

str = System.String.Format("Hello {0}, we will ship {1} to you.", callerName, bookName);

str = "<BuyBookResponse xmlns='https://tempuri.org/'><BuyBookResult>" + str + "</BuyBookResult></BuyBookResponse>";

xmlDom = new System.Xml.XmlDocument();

xmlDom.LoadXml(str);

Message_2 = xmlDom;

The last portion above is constructing the response message which the client will see. I’ve intentionally made the text different from what the MSDN sample returned so that when we test it, we can be sure that the response is coming from BTS.

To test it out, use the BookStoreClient in the MSDN sample. You just need to change the endpoint address for the “WSFederationHttpBinding_IBuyBook” endpoint to the address specified when configuring the RL.

FederationSampleWithBizTalk.zip

Thanks.