Integration and Patterns

 

Retired Content
This content is outdated and is no longer being maintained. It is provided as a courtesy for individuals who are still using these technologies. This page may contain URLs that were valid when originally published, but now link to sites or pages that no longer exist.

Integration Patterns

May 2004

Summary: This chapter introduces the Global Bank scenario that is used throughout this guide and briefly discusses how patterns can help development teams find workable answers to integration challenges.

Contents

The Problem of Integration

The Global Bank Scenario

Patterns

Patterns at Global Bank

Next Chapter

"The significant problems we face cannot be solved at the same level of thinking we were at when we created them."—Albert Einstein

Few enterprise applications exist in isolation. Most are connected to other applications and services by data feeds and common reference data. Others are connected through elaborate integration networks. If you look above the level of single applications and focus on an enterprise's whole software portfolio, you often see a complex collection of silo applications, heterogeneous platforms, and islands of sometimes duplicated data and services that are interconnected by messages, objects, file transfers, batch feeds, and human interactions.

At the same time, businesses consider information technology (IT) to be a key element of both operational efficiency and competitive advantage. There are high expectations of technical investments despite rapidly changing business conditions and operating environments. Compounding this problem is the rate of change in technology, as innovations such as Web services emerge. Although adopting this new technology promises a new level of interoperability between systems and enterprises, it also demands that practitioners devise an integrated, enterprise-level approach to building applications and services.

Given today's complex technical and business environment, how do you create an integrated portfolio of applications and services for your enterprise?

This guide discusses known good ways to integrate systems, and it uses patterns to describe them. To ground the discussion in something tangible, the guide:

• Describes a representative scenario in detail
• Builds out a performance-tested, baseline architecture to validate the approach
• Uses the vocabulary of patterns to describe important design tradeoffs
• Traces the patterns to an implementation that uses the Microsoft® platform

The guide does not:

• Describe a feature-complete or fully secure implementation
• Assert that there is only one right answer when it comes to design
• Promote patterns as a silver bullet for solving all design problems

The Problem of Integration

Many enterprises create overly complex integration architectures in very predictable ways. Business units within the enterprise often have a strong business case for an IT capability. They fund and staff projects to provide this capability, while tracking primarily the delivered functionality. However, they often have little regard for the technical architecture underneath. Assuming the business case is sound, this is often in the best interest of the business—at least in the short run.

In the long run, however, building business capabilities without careful consideration of an enterprise-wide technical architecture can lead to a high cost for IT operations, an inflexible portfolio of applications and services, and a high cost for new application development. Even worse, the enterprise will be at a distinct disadvantage with respect to other competitors that have built well-factored, agile, and well-integrated applications and services. This is especially true in industries where information has a high economic value and new business models emerge quickly, posing real economic threats.

The balance between these business and technology forces is delicate. Moving too fast to enable business capabilities can result in a glut of architecturally incompatible applications, which likely will need to be rationalized and integrated later at a high cost to the enterprise. On the other hand, unchecked indulgence of the natural engineering tendency to study the problem deeply before acting can lead to long and costly enterprise architecture engagements. Not only do these efforts take significant time to execute (at a high opportunity cost), but, if not carefully managed, they risk producing little more than a set of binders that sit unused on a shelf.

Integration Architecture

An enterprise's integration architecture balances the requirements of the business and the requirements of individual applications. Inside this integration architecture, you often find an overwhelming maze of systems, connections, and channels. If you study enough of these, you see common combinations of integrated systems such as portals, networks of connections such as message brokers, buses, and point-to-point connections, and numerous individual connections and channels. To understand the maze, it is helpful to understand how many of these integration architectures evolve—one application at a time.

Many developers and architects start by designing and building stand-alone applications. They then progress to more complex enterprise applications. As applications require connections to shared enterprise resources, it is natural to create abstractions and wrappers that encapsulate these resources from an application-centric point of view. After all, it is just one more connection to the enterprise resource. Further enterprise-level work is often out of scope for the application project.

Although this approach works well from the perspective of a single application, connecting all applications in this way is unlikely to produce a well-ordered set of applications. Instead, you need a logical design at the integration level, just like you need a logical design at the application level. To think clearly about an integrated portfolio of applications and services at the enterprise level, you must invert your viewpoint. You must first consider the needs of the enterprise as an integrated whole and then consider how to expose shared functionality through networked applications. This kind of thinking is quite different from traditional monolithic application development or n-tier development. It begs the question: what is an application anyway?

Applications

Most software-related definitions describe applications as "any part of a software system used to deliver end-user functionality" [Firesmith95] or "a computer program designed to help people perform a certain type of work" [Microsoft02-3]. If you think of design from a traditional application-centric point of view, you usually expect to encapsulate functionality into one or more executable files and then deploy them to necessary servers. You do not expect to use existing services to any large degree. However, if you approach this same problem from an integration architecture perspective, the ideal application is a thin layer of presentation that consumes shared functionality or data at the enterprise level. Ideally, much of this functionality already exists and is accessible at a level of granularity that is meaningful to the business. And if new functionality must be built, it is designed not to stand alone, but to be shared with other enterprise applications and services.

To show how this kind of thinking might be practically applied, the remainder of this guide uses some of these concepts in an interesting, yet challenging, online bill payment scenario called Global Bank. This scenario introduces enough complexity to illustrate the design tradeoffs without introducing too many details.

The Global Bank Scenario

Although talking about architecture and design at a conceptual level helps to set guiding principles, there is nothing like building out an actual system against requirements to gain common understanding at a more technical level. That is why the authors of this guide have developed an executable baseline architecture against a concrete scenario: Global Bank. Later chapters of this guide describe the design and implementation details of the solution, but first, let's look at some of the context and requirements of this scenario.

Context

Global Bank is a midsize, traditional bank that has acquired a complete range of financial services capabilities through a series of acquisitions. It has a limited online banking presence that is fragmented across its various divisions. As part of its strategy to expand with the limited cash it has available, Global Bank has decided to innovate in the online banking market by providing a host of value-added services in addition to a fully integrated financial management capability.

Note: This chapter contains an intentionally brief overview of Global Bank's business context and approach to building integration architecture. For more detailed information, see Chapter 7, "Project Notebook."

The chief executive officer (CEO) decided the first step was to immediately add an electronic bill payment capability to the current online banking system. This would allow customers to schedule electronic payments online from their checking accounts—a high demand feature providing greater customer convenience. The CEO believed this added convenience would have an immediate impact upon customer satisfaction and loyalty, while demonstrating tangible progress to his board of directors. To initiate this effort, the CEO brought in his chief technical officer (CTO) and the vice president for consumer banking and asked them to deliver this capability before the end of the fiscal year. He expected rough-order-of-magnitude (ROM) cost and schedule estimates within six weeks.

Requirements

The CTO immediately involved a senior program manager to create a project around this initiative. The program manager formed a team to build a high-level project plan and to start gathering requirements. Unlike many projects, the CTO expected to not only gather requirements from the consumer banking division, but to also negotiate requirements with the consumer banking division based on the overall needs of the business.

As he reflected on the overall initiative, the CTO felt confident that the business would continue to invest in additional financial services for its customer base and that additional acquisitions were likely to follow. This was clearly not an isolated initiative; rather, it reflected a longer-term strategy for the company. He realized it was important to have a well-conceived technical architecture at the enterprise level that would smoothly support these corporate goals.

Beyond the functional requirements that would emerge, he wanted a solid technical foundation that would allow him to meet operational requirements as well. He pulled together an architecture team and asked them to create a baseline architecture that would support this initiative and future initiatives. As a first approximation, he started with the following high-level requirements and constraints:

• Build a baseline architecture for a Web-based online banking portal that allows customers to pay bills online from their checking accounts.
• All account-related transactions will use the current system, which resides on an IBM mainframe (OS390) using Customer Information Control System (CICS) based transactions.
• The online bank system will reside in the corporate data center in Seattle, Washington. It will be connected to an acquired bank's data center in Los Angeles, California though a private leased line.
• Loan information will be pulled from the acquired bank's loan systems, which reside on systems that are based on IBM WebSphere J2EE.
• All customer profile information will use the current Customer Relationship Management (CRM) system.
• Domestic electronic payments will use the current payment system, and international electronic payments will use SWIFT-based transactions through an external payment gateway. Payees that cannot receive electronic payments will be paid using electronic transactions to a manual fulfillment center, which will then make the payments manually through the U.S. mail.
• Except for the systems previously identified, the system will be based on the Microsoft platform.
• The system's overall transaction rates, concurrent users, and response time must meet the first year's projected usage plus an engineering safety factor of 3x (or three times the first year's projected usage) to handle burst load.
• The system must meet or exceed the service level agreement (SLA) for our current online system.

Next Steps

If you were part of this architecture team, how would you proceed? If you were fortunate, someone on this team would have built a system like this before and would apply those experiences and lessons learned to this effort. This would be optimal, but is not probable. It is more likely that members of your team are very proficient with a set of technologies that might solve part of this problem. For example, they might be proficient with object-oriented design, message-oriented middleware, integration servers, or distributed object systems. Naturally, team members want to apply the tools they have used before to solve future problems, but how do you know which technology is appropriate for which area of the design and when? When the problem and the technology align, you can move quickly and effectively to build the solution. However, we have all seen familiar technology applied in unfamiliar areas for which it is suboptimal.

Wouldn't it be great to be able to break this problem down into relatively atomic decision points and understand the design alternatives available to you at each point? For each alternative, wouldn't you want to know how others have implemented similar choices and what the resulting advantages and disadvantages were? Although you may not have the luxury of an experienced person to discuss this with, the next best alternative is a catalog of best practices that are documented as patterns. Before continuing with the Global Bank scenario, let's discuss the concept of patterns at a very high level and how they might apply to software development.

Note   Rather than repeat the introductory material from Enterprise Solution Patterns Using Microsoft .NET or from a formal pattern description found in an introductory patterns book, this chapter relaxes the formal pattern description and provides some examples from everyday life. This is an effort to make the pattern idea more approachable. The chapter then shows the results of applying pattern-based thinking to an integration scenario. Later chapters explain specific patterns in more detail.

Patterns

People think in patterns. It is the way we naturally communicate ideas related to complex subject areas such as music, science, medicine, chess, and software design. Patterns are not new. We all use them intuitively as part of the learning process without really thinking about it. And because our minds naturally use patterns to perform complex tasks, you can find patterns nearly everywhere.

Patterns in Sports

Consider what happens during a soccer game or an American football game.

Figure 1. Patterns in soccer

Figure 2. Patterns in American football

Individuals who are acting according to predetermined patterns move quickly and decisively against targeted opponents. Each individual's pattern of movement is also part of a larger pattern of orchestration where each player has clear responsibilities and scope. In addition, the entire team is in a binary state—either offense or defense. Without patterns in sports, the games would not be as rich and interesting. Can you image how long the huddle would be in an American football game without the language of plays (patterns)?

Note   Software patterns are significantly more complex than these simple examples. The examples are intended to make the notion of software patterns more approachable at the expense of being less technically rigorous. For more rigorous introductions to patterns, see the bibliography section.

If you look closer at patterns, you will find relationships between them. In sports, for example, teams have certain plays for offense and certain plays for defense; the patterns that describe two players' actions must fit into a larger pattern that the team is following. In this sense, patterns can be described in terms of hierarchies.

Patterns in Music

Another example of how people think in patterns is the patterns found in music, such as rock and roll. In rock and roll, a rhythm guitar player usually repeats a pattern of chords in a specific key. Against this backdrop, a lead guitarist plays a freeform series of notes from a candidate pattern of notes that correspond to the chord progression being played. Figure 3 shows a pattern chart that lead guitarists use to learn the correct finger positions on a guitar neck.

Figure 3. Pentatonic scale patterns in the key of A

The root note in Figure 3 indicates the key that the song is in. Within the song's key, the lead guitar player is free to improvise, although most of the notes he or she plays will correspond to the pattern chart in Figure 3. The order and sequence of the notes may vary according to artist, style, and song, but the pattern of actual notes played remains. If the key changes, the scale pattern moves to a different place on the guitar neck that corresponds to the song's new key. Interestingly enough, this notion of one layer of patterns constraining another is exactly what happens when you apply pattern-based design methods. This is just as true in software design as it is in other design disciplines.

Pattern Structure

Patterns have a natural relationship with each other. Perhaps the most often used example is the interplay between patterns for designing towns, which in turn, contain patterns for designing clusters of buildings and roads. The building cluster and road patterns, in turn, contain patterns for designing buildings. Figure 4 shows these relationships.

Figure 4. Hierarchy of patterns

Pattern-Based Design

While pattern-based design is relatively new in the field of software development, industrial technology has used pattern-based design for decades, perhaps even centuries. Catalogs of mechanisms and standard configurations provide design elements that are used to engineer automobiles, aircraft, machine tools, and robots. Applying pattern-based design to software development promises the same benefits to software as it does to industrial technology: predictability, risk mitigation, and increased productivity.

Experience is Key

Of course, pattern-based design alone is no guarantee of success in either software design or industrial technology. Known good mechanisms can be used to build planes that do not fly, cars that do not handle well, and applications that do not scale. There is simply no substitute for the skill and experience of designers and engineers in any subject area, and software is no exception. Although patterns help by offering manageable portions of design knowledge, they are not complete solutions by themselves. They still require your skill and experience to tailor them to your specific requirements.

Applying Patterns

Applying patterns to a specific scenario usually involves an iterative design process. As a guiding principle, you want to keep your design as "simple as possible and no simpler," as Albert Einstein once said. Although you can use patterns to solve design problems, make sure that you have a legitimate problem first before applying a pattern. Do not use patterns just for the sake of using them.

Although design guidelines and process are related topics (and worthy of dedicated works), this book focuses on the tangible outputs of the design process. It focuses in particular on the role of patterns as they are applied to problems. To examine the concrete artifacts produced by a pattern-based design process, let's go back to Global Bank and see what came out of the design sessions as the team worked on the baseline architecture.

Patterns at Global Bank

The architecture team analyzed the high-level requirements and constraints provided by the CTO and reviewed existing technical architecture models of the enterprise. The architecture team also designated several members of the team to do a build-versus-buy analysis of related commercial off-the-shelf software (COTS) packages that might meet the requirements.

Based on the build-versus-buy analysis, the team decided to build a custom extensible portal by using commercial platform infrastructure components such as Web servers and database servers, but not to use packaged portal applications. Figure 5 shows their initial approximation of the server types in a network diagram.

Figure 5. Initial network diagram with server types

For each key use case, the team determined the sequence of system interactions that must occur to fulfill the stated requirements. They described these interactions in terms of server types and message sequences. Figure 6 shows the View Scheduled Payments use case realization in the form of a collaboration diagram.

Figure 6. View Scheduled Payments collaboration diagram

The flow of the use case in Figure 6 is:

1. A customer navigates to the online bill payment application.
2. The Web server prompts the customer for a user name and password.
3. The Web server authenticates the customer by using information retrieved from the directory server.
4. The Web server sends an asynchronous request to the integration server asking for related loans.
5. The Web server retrieves the customer's mainframe account number from the payment server.
6. The Web server retrieves customer profile information from the CRM server.
7. The Web server retrieves account balance information from the mainframe.
8. The Web server retrieves a list of scheduled payments from the payment server.
9. The Web server checks the integration server to see whether any loan information has been retrieved.
10. The Web server builds the presentation, which displays account balance, scheduled payments, and customer profile information.
11. If loan information is available, it appends this optional information onto the presentation.
12. The Web server returns the presentation code back to the browser.
13. The browser renders the view.

This use case realization is a representative sample of the bill payment application's significant use cases. The team took a similar approach to analyze other use cases, identify server types, and design message interactions. To create these use case realizations, the team conducted a series of iterations, each beginning with a design session using class-responsibility-collaboration (CRC) style techniques. Although similar in nature to CRC sessions, these sessions were not limited to class-level abstractions. Often these sessions involved subsystems, server types, processes, and channels as well.

The team's goal, as they considered the necessary collaborations between elements, was to design the simplest system that would satisfy all current requirements and account for system constraints. While working through the alternatives, they relied on the language of patterns to provide a common vocabulary for the team. Patterns were also useful as a concise way to communicate the context, forces, and tradeoffs involved in each design decision. At times, they realized that certain patterns only added complexity to the design, so they eliminated those patterns.

As they completed each iteration, they created a pattern model of the system to record their decisions. The model from the last iteration is shown in Figure 7. This pattern model represented the simplest system that realized the target use cases and constraints. To keep their models simple, they represented patterns as circles and added other high-level design elements to the model to communicate the overall design.

Figure 7. Patterns and design element model

The next chapter captures some of the pattern-based discussion that occurred during the design process. For now, just notice how the patterns connect key design elements such as the customer and the mainframe system.

Note   Because architects and developers are the primary audience for this guide, this discussion moved quickly into applying patterns to a specific scenario. For a more detailed discussion of pattern-based design, see Chapter 7, Project Notebook.

The next step was to map these patterns to an implementation technology and to iterate again. In this case, many of the platform decisions were already made, and these platform decisions constrained the design choices. Sometimes, the implementation constraints forced the team to reconsider their design, and they adjusted accordingly. When they finished, they produced the platform-level implementation diagram shown in Figure 8.

Figure 8. Pattern diagram mapped to implementation technology

Figure 8 shows that the Global Bank integration architecture is composed of numerous pattern-based design elements implemented on the Microsoft platform. To trace the implementation of these elements down to running bits, refer to the appropriate implementation pattern later in this guide. For example, to understand how to implement the gateway to the mainframe, refer to Implementing Gateway with Host Integration Server 2004. This pattern includes the details associated with connecting Global Bank's .NET Framework portal application with their existing COBOL-based CICS transactions.

Next Chapter

This chapter introduced the Global Bank scenario that is used throughout this guide and briefly discussed how patterns can help development teams find workable answers to integration challenges. The next chapter uses the language of patterns to explore the decisions and tradeoffs that the Global Bank architecture team made while designing and implementing their bill payment system.

Start | Previous | Next

Retired Content
This content is outdated and is no longer being maintained. It is provided as a courtesy for individuals who are still using these technologies. This page may contain URLs that were valid when originally published, but now link to sites or pages that no longer exist.

© Microsoft Corporation. All rights reserved.