Implementing Message Broker with BizTalk Server 2004

 

Retired Content
This content is outdated and is no longer being maintained. It is provided as a courtesy for individuals who are still using these technologies. This page may contain URLs that were valid when originally published, but now link to sites or pages that no longer exist.

Integration Patterns

Contents

Context

Background

Implementation Strategy

Example

Resulting Context

Testing Considerations

Security Considerations

Operational Considerations

Variants

Related Patterns

Acknowledgments

Context

Your application is interacting with multiple other applications through a Message Broker so that each application only needs to communicate with the Message Broker, rather than all the other applications. The Message Broker is implemented by using Microsoft BizTalk Server 2004.

Background

Global Bank is implementing an Execute Scheduled Payment business process that customers can use to schedule payments. Each payment is transacted through a payment authority. The payment authority is determined when that payee is first configured. Currently, there are three different payment authorities:

• There is an internal payment authority that Global Bank uses to handle payments itself.
• There is an external payment authority where payment requests are sent to an external bill payment authority.
• There is a manual payment authority where payment requests are handled manually.

Bank growth, acquisitions, and mergers may increase the number of payment authorities.

To implement the Execute Scheduled Payment business process, Global Bank chose to use Microsoft BizTalk Server 2004 as an implementation of the Message Broker pattern. As Figure 1 shows, requests for payment are sent directly to the message broker, and the message broker then sends the request to the correct payment authority.

Figure 1. BizTalk Server 2004 as Message Broker in Global Bank

Apart from sending any message received to the appropriate recipient, the Message Broker implementation also must do the following:

• The implementation must parse the message to determine where to send it.
• The implementation must convert the message from the source format to the destination format.
• The implementation must allow receivers (applications) to register with the broker so that the broker can send messages to them.

Implementation Strategy

BizTalk Server 2004 uses a Publish/Subscribe messaging architecture that can easily be used to implement Message Broker. BizTalk Server 2004 uses Pipes and Filters for the message transformation.

Figure 2. BizTalk Server 2004 internal publish-subscribe architecture

Figure 2 shows how BizTalk Server 2004 receives, processes, routes, and sends messages. A client application (the publisher) sends a message to a receive port through a predefined protocol. For example, the publisher may send a message to the message broker by writing a file into a directory on the server. The message is received by the receive adapter. The following two types of adapters are available:

• Transport adapters. Transport adapters receive messages by using standard transport mechanisms. BizTalk Server 2004 includes many standard transport adapters such as HTTP, Simple Mail Transfer Protocol (SMTP), File, and File Transfer Protocol (FTP).
• Application adapters. Application adapters connect with common business applications. You usually purchase these adapters from a vendor, or you develop them yourself.

The receive adapter passes the message to a receive pipeline, and the receive pipeline then parses and converts the incoming message to XML (except when using the pass-through pipeline). XML is the format that BizTalk Server 2004 uses internally. As the receive adapter receives the message and as the receive pipeline processes it, additional data about the message is collected. This message metadata includes information about the transport the message was received on, such as the original name of the file in the case of a file transport. It also includes information from each of the components in the pipeline. For example, the Party Resolution component stores information it receives from a digital certificate attached to the message. This metadata is known as the context properties.

The receive pipeline uses an XML schema that the developer created. This schema disassembles and converts the message to XML and then validates the message against the schema specification. The schema describes the type, the cardinality, and the other attributes of every record and every field that is contained in the message. As the receive pipeline parses the message, it collects data from within the message itself. To do this, the developer creates a second schema that is known as a property schema. The property schema allows specific portions of the message to be promoted as additional metadata. This metadata is known as the promoted properties. For an example of how promotion occurs, see the "Example" section later in this pattern. After preprocessing in the receive pipeline, the message is submitted to the MessageBox database complete with all additional metadata.

The MessageBox database sends the message to the message recipients by using a message subscription. To implement a message subscription, the developer creates a send port that represents the message recipient. A send port usually consists of a send pipeline for post-processing of the message, a send adapter that transmits the message to its destination, and a filter expression that defines whether the message will be sent through this send port. The filter expression is a query against the context properties of messages in the MessageBox database. A copy of the message is only sent to this send port if there is a match between the filter expression and the context properties. For example, the filter expression on the send port might state, "send all documents that are of type ExecutePayment.Payment, where the fulfillmentTypeId is equal to 1. "

The MessageBox database sends a copy of the messages to each send port that has a filter expression that matches the context properties. The message is received by the send port (the subscriber process), processed by the send pipeline, and then transmitted through the send adapter. The send adapter can be a transport adapter such as File, FTP, HTTP, BizTalk Message Queuing (MSMQT), SMTP, SOAP, or MQSeries. Or, the send adapter can be an application adapter that connects directly to the destination application.

To implement the Message Broker, the Global Bank developer must do the following:

1. The developer must define the format of the message sent by the source application to the Message Broker.
2. The developer must define the formats of the messages sent from the Message Broker to each of the external systems and specify how the source message is translated to the formats that are required by the external systems.
3. The developer must define how messages are sent from the Message Broker to each external system.

Example

Figure 3 shows the Global Bank Message Broker implementation based on BizTalk Server 2004. The numbers mark the areas that the following implementation steps focus on.

Figure 3. Global Bank Message Broker implementation with BizTalk Server 2004

Step 1: Create the Receive Port and Define Message Schemas

The first step in the implementation of the Message Broker is to create the receive port in BizTalk Explorer. This receive port defines the location that the client application will send messages to so that the messages are published to the message broker. Virtually any transport can be used to send messages from the client to the message broker. This example uses the BizTalk Message Queuing (MSMQT) adapter because MSMQT provides guaranteed delivery.

To create the receive port for the message broker

1. Create a receive port called Payment Broker RP.
2. Create a receive location on this receive port called Payment Broker RL.
3. Set the transport for the receive location to MSMQT with a queue named Payments.
4. Specify the BizTalkServerApplication as the receive handler, and specify XML Receive as the receive pipeline.
5. After following the remaining steps to implement the message broker, enable the receive location.

This completes the creation of the receive port.

Next, the payment messages sent by Global Bank and the messages going to each of the three payment channels have to be defined. The payment messages are marked 1a in Figure 3, and the payment channels are marked 1b. BizTalk Server uses these definitions (XML schemas) to parse and to convert messages into XML if they are not already in XML. A disassembler component in the receive pipeline parses and converts these messages.

For the Execute Scheduled Payment implementation, the schema is actually generated automatically by the SQL Transport Schema Generation Wizard (see Implementing Process Integration with BizTalk Server 2004). Figure 4 shows the generated schema in the BizTalk Server Schema Editor.

Figure 4. The Payment schema in the BizTalk Server 2000 Schema Editor

You can also create a special second schema that is known as a property schema. This property schema defines one or more fields that are linked to specific fields in the message schema. Linking fields in this way is known as promoting properties. This link between the message schema and the property schema is established in the Promote Properties dialog box in the BizTalk Server Schema Editor. In Figure 4, the unique symbol next to the fulfillmentTypeId field denotes a promoted property field.

Figure 5 shows the Promote Properties dialog box where you promote the fulfillmentTypeId field that is in the payment message. The value of the fulfillmentTypeId field is used later to determine the payment authority that the message is sent to.

Figure 5. Creating a promoted property field (Click the image to enlarge it)

The format of the message that is sent to the message broker is specified by the XML schema shown in Figure 4. The format of the messages for each of the payment authorities is likely to be different from the source format. The format of the messages is also likely to vary between different payment authorities. Therefore, schema definitions for each of the outgoing message formats also have to be built. For brevity, these schema definitions are not presented here.

Step 2: Define Maps to Convert Between Message Formats

For each of the destination message formats specified in the previous section, an XSLT map must be developed. This XSLT map specifies how the source message format is translated to the destination message format (number 2 in Figure 3). These message format translations may be merely structural. For example, converting an XML message to a comma-separated-values (CSV) message is structural. Or, the translation may require the generation of new message content. For example, the source message may not contain any information about the time or date that the message was sent, but this information may be required in the destination format. New message content also has to be generated when a field in the source message is used as a key to a specific row in a database table, and fields in the destination message are filled from fields in this database row. This is a common requirement.

Figure 6. Payment authority map in the BizTalk Mapper (Click the image to enlarge it)

You can use the BizTalk Mapper to create the XSLT maps. The BizTalk Mapper provides a graphical view of source and destination schemas and of the conversion from one to the other. The output from the BizTalk Mapper is the XSLT map. Figure 6 shows the development of a payment authority map from the source XML schema in the BizTalk Mapper. Three separate maps are created, one for each of the payment authorities:

• PaymentRequest2InternalPayee.btm is created for the internal payee.
• PaymentRequest2ExternalPayee.btm is created for the external payee.
• PaymentRequest2ManualPayee.btm is created for the manual payee.

Step 3: Create Subscriptions to Messages

To configure the BizTalk Server message broker for the Execute Scheduled Payment scenario, three send ports must be configured. Each send port is configured with its own specific transport type and transport address, send pipeline, and BizTalk map (number 3a in Figure 3). In addition, the port filter has to be specified appropriately on each send port to set up the correct subscription for this payment authority to the MessageBox database (number 3b in Figure 3).

To configure one of the send ports (Internal Payment)

1. Create a new send port that is called InternalPayee_SP.

2. Select SOAP (Web Services) for the transport type. Use the following as the transport address (the Uniform Resource Identifier [URI]):

   http://EAIPayee/InternalPayee/Payee.asmx

   Depending on the transport chosen, different transport address information can be specified.

3. Specify XML Pipeline as the send pipeline. The send pipeline defines the post-processing that is performed on the message just before the message is sent, such as attachment of a digital signature to the message.

4. Specify PaymentRequest2InternalPayee.btm as the XSLT map to be applied to the source messages.

5. Specify the filters for this send port. These filters define the subscription for this send port in the MessageBox database, and they control the criteria that the MessageBox database uses to determine the messages that are sent to this port. For the Execute Scheduled Payment scenario, the filter specifies the following two criteria as shown in Figure 7:

   • All messages of type http://GlobalBank#payment.
   • Messages that have a promoted fulfillmentTypeId property that is equal to 1 (internal fulfillment).

   

   Figure 7. Specifying the filter (subscription) for a send port

   With this configuration, all messages of type http://GlobalBank#payment that have a fulfillmentTypeId property that is set to 1 (internal fulfillment) are sent to this send port. This send port is configured to transmit the payment authorization to the internal payment application by using a SOAP call.

6. Configure the remaining ports in a similar way. Each port has its own unique transport address and its own unique XSLT map. In addition, the filters on these two ports specify a different value for the fulfillmentTypeId property**.** A value of 2 denotes external fulfillment, and a value of 3 denotes manual fulfillment.

7. Finally, enlist and then start each of the send ports.

Resulting Context

The implementation of the Message Broker pattern with BizTalk Server 2004 results in the following benefits and liabilities:

Benefits

• Message routing is determined at run time. Destination applications can be added or deleted from the system at run time without affecting the source application or other destination applications.
• Message tracking and archiving. All messages transmitted through the message broker can be automatically tracked and archived. The tracking, archiving, and reporting functionalities are implemented only once, and they are shared by all applications that use the message broker. In BizTalk Server 2004, these functionalities are provided by the BizTalk Server Health and Activity Tracker (HAT).
• Centralized management. The messaging for an application is configured and managed from a central location and is not coded into the applications.

Liabilities

• Additional message processing steps. This Message Broker implementation adds extra steps in the processing of a message. These extra steps could prevent some fully synchronous messaging scenarios from being implemented, and they could lead to higher latency than a straight-through processing solution. BizTalk Server 2004 has an alternate message processing mechanism that is known as a request-response port. The request-response port is specifically designed to solve problems caused by these synchronous messaging scenarios.
• Routing rules are not centralized. Decentralized routing rules are an inherent weakness in this Message Broker solution. The routing rules are spread between all the send port configurations in the filter rules that are specified. Therefore, configuration errors are harder to debug, and the configuration is more complex to manage.

Testing Considerations

You can use several strategies to test the Message Broker implementation:

• Create a mock destination application (in other words, set up a send port) that subscribes to all messages of a specific type sent to the message broker. These messages are written to a test directory. This process allows the message schemas and message maps to be validated.
• Enable or disable send ports to test individual subscribers in isolation.
• Use the HAT to track all incoming and outgoing messages, to analyze the message flow through the MessageBox database, to report exception conditions, and to analyze message throughput.
• Use the Microsoft Visual Studio .NET environment to test schemas and maps by validating sample documents, or use the schemas and maps to generate sample message instances.

Security Considerations

The following security considerations apply when implementing Message Broker:

• Controlling the visibility of messages in the message broker. If many different groups in an organization are using the message broker, it may be necessary to implement access security to control who can view messages that are sent to the message broker and to control who can set up subscriptions to messages. Remember that these messages may all be tracked and held in the Microsoft SQL Server database. BizTalk Server 2004 provides a very detailed security model that allows this type of access security to be enforced. For example, message traffic can be partitioned across multiple host instances, where each host instance runs by using different security credentials.
• Passing digitally signed messages. In a point-to-point processing scenario, a message originator may digitally sign and encrypt a message by using a certificate issued by the receiver of the message. However, if this message is transmitted through a message broker, the message broker usually cannot decrypt the message. To overcome this problem, the encrypted message can be wrapped in a clear text envelope. The envelope contains metadata about the message, such as the address. The metadata allows the message broker to pass the message to the intended recipient.
• Validating senders and receivers. Under some circumstances, the message broker may need to validate the publishers or the subscribers. This means the message broker must be able to validate a message that has been digitally signed and then attach a digital signature to a message. BizTalk Server 2004 provides these capabilities.

Operational Considerations

The following operational considerations apply when implementing Message Broker:

• Dynamic configuration. One of the big benefits of the Message Broker pattern is that it is easy to configure dynamically at run time. New publishers can easily be added at run time, and subscribers can be added or removed easily.
• Load-balancing and fault tolerance. As Message Broker becomes central to an organization's integration solution, it is essential for the message broker to be fault-tolerant and for the message processing workload to be spread across multiple nodes. BizTalk Server 2004 works in conjunction with Microsoft SQL Server 2000 so that all the configuration information, the messages, and the tracking data are held within SQL Server 2000. BizTalk Server 2004 also works in conjunction with Microsoft SQL Server 2000 so that all processing is distributed across all the servers running BizTalk Server 2004 that are part of the same BizTalk Server 2004 group. (A BizTalk Server 2004 group is a group of servers that share the same set of SQL Server tables).
• Store and forward. Rather than send a message immediately, some message brokers can hold onto a message until a subscriber requests it. This is useful in situations where subscribers are not connected all the time, or where network issues cause problems. BizTalk Server 2004 does not provide this functionality directly, although it can easily be implemented by using database staging tables or Microsoft Message Queuing (MSMQ).

Variants

Using promoted properties in the message and using filters on the send port is the simplest way to implement a message broker by using Microsoft BizTalk Server 2004. However, other more complex implementations may be preferable in certain circumstances.

Business Rule Engine

In some situations, the logic used to determine where the message is sent is more sophisticated than the simple example used here, or the logic is subject to frequent change. For example, a business may use a number of suppliers. The business may determine which supplier to send a purchase order to based on the current relationship of the business with that supplier. For example, the business might select the supplier based on the level of discounts. This cannot be calculated by using information in the message, so business logic has to be written to determine this. This logic could be coded in a Microsoft .NET Framework assembly and referenced from a map, but if the business rule is subject to frequent change, each time the rules change a new version of the business component must be deployed.

BizTalk Server 2004 solves this issue by providing a Business Rules Engine. The Business Rules Engine enables a user to define a vocabulary that is based on information contained in the message and in a database, and that is held in .NET Framework components. This vocabulary may then be used to define a series of business rules. These rules can implement sophisticated business logic, and this business logic can then be changed and deployed easily.

Roles and Parties

As the number of publishers and subscribers increases, it becomes increasingly difficult to manage and to maintain the receive ports, the send ports, and the subscriptions. In a scenario where there are thousands of potential subscribers for a message, BizTalk Server 2004 uses roles and parties. For example, you may have configured thousands of subscribers (suppliers) to send purchase orders to in your order processing system.

For this example, the Supplier role is created, and a port type is associated with this role. In this case, that is the PurchaseOrderToSupplierType port type. The outgoing message is then sent to an abstract PurchaseOrderToSupplierType port type.

Independently of this, parties are created to represent individual suppliers. After parties are created, a party can be enlisted in a role. For example, a party called Northwind Traders may be created and then enlisted in the Supplier role. When this is done, a physical port of type PurchaseOrderToSupplier must be configured to send the document to the actual supplier.

The roles and parties implementation is more complex and requires the message broker to use BizTalk Orchestration instead of using BizTalk Messaging. However, when many potential message recipients are involved, it provides a much simpler mechanism for maintaining the recipients in the system.

Related Patterns

Of particular note among related patterns, Enterprise Integration Patterns [Hohpe04] examines the difference between the Publish/Subscribe implementation (reactive filtering) and a content-based router. A content-based router examines the message and sends it to a recipient by using a process that is known as predictive routing. For more information, see the following patterns:

• Broker [Buschmann96]. In a distributed setting, the Broker pattern tries to make distribution transparent. Client-side and server-side proxies mediate between the broker and the clients and server.
• Content-Based Router [Hohpe04]. The Content-Based Router pattern is a Broker variant that specializes in routing communications to different applications based on the communication's content.
• Mediator [Gamma95]. The Mediator pattern separates objects so that they are only aware of the mediator but not each other. The Broker deals with similar concerns, but it can only be used in the context of enterprise applications.

Acknowledgments

[Buschmann96] Buschmann, Frank; Regine Meunier, Hans Rohnert, Peter Sommerlad, and Michael Stal. Pattern-Oriented Software Architecture, Volume 1: A System of Patterns. John Wiley & Sons Ltd, 1996.

[Gamma95] Gamma, Erich; Richard Helm, Ralph Johnson, and John Vlissides. Design Patterns: Elements of Reusable Object-Oriented Software. Addison-Wesley, 1995.

[Hohpe04] Hohpe, Gregor, and Bobby Woolf, Enterprise Integration Patterns: Designing, Building, and Deploying Messaging Solutions. Addison-Wesley, 2004.

Start | Previous | Next

Retired Content
This content is outdated and is no longer being maintained. It is provided as a courtesy for individuals who are still using these technologies. This page may contain URLs that were valid when originally published, but now link to sites or pages that no longer exist.

© Microsoft Corporation. All rights reserved.