Complex Event Processing (Middleware)---a Technical Reference Guide for Designing Mission-Critical Middleware Solutions

Want more guides like this one? Go to Technical Reference Guides for Designing Mission-Critical Solutions.

Complex Event Processing (CEP) is characterized by temporal analysis of massive streams of business events resulting in the identification of meaningful patterns in real-time. That analysis is performed in an effort to identify correlated events that happen over a span of time whereby the culmination of those events is significant, such that a complex event is derived. To quote Mark Simms, Microsoft Senior Program Manager, the goal of CEP is "to find the needle in the stack of needles."

For example, among millions of incoming events, a CEP system may receive the following three, from one or multiple sources, over the course of an hour:

  • A sales person receives an email with the word congratulations in the body.

  • An opportunity owned by that sales person is closed in a CRM system.

  • The accounting department receives a purchase order from the company for which the opportunity was closed.

From these events the CEP system may infer the complex event—we have a new customer.

Best Practices

By leveraging an event-driven architecture, organizations are enabled to be more proactive in identifying and responding to business events. StreamInsight is a unique Microsoft product capable of legitimately processing hundreds of thousands of data points per second, and executing complex filtering or incremental aggregations. However, even if your business does not have event producers that send out massive amounts of events, you can still benefit from having an event processing engine like StreamInsight that maintains standing queries, and drives your organization to look for new ways to improve efficiency and make smarter decisions faster.



  • Solutions that require throughput in excess of 5000 events per second and/or latency tolerance that is less than 5 seconds should select Premium Edition whereas solutions with less robust requirements should select Standard Edition. The Choosing StreamInsight Edition article explains which edition of SQL Server ships with which edition of StreamInsight.

  • StreamInsight supports two deployment models, including hosted and stand-alone. This Deployment Model article discusses the scenarios in which each model would be a good fit.

  • StreamInsight has a somewhat limited hosting model for load balancing and failover, but there are a few options for achieving failover that are described in this discussion thread.

Management and Administration

  • Although there is no user interface for managing or monitoring StreamInsight, it is possible to create and access diagnostic views for Monitoring the StreamInsight Server and Queries.

  • The only graphical tool that StreamInsight provides is the StreamInsight Event Flow Debugger. The debugger provides the abillity to start and stop queries, view existing applications and show queries included in applications.

  • What is not available graphically is the ability to create applications, delete applications and delete queries, but Richard Seroter has created an application that let’s you do this called the StreamInsight Server Manager.

Case Studies and References

Case Study

 EdgeNet StreamInsight Case Study


Questions and Considerations

This section provides questions and issues to consider when working with your customers.

  • Recognize that events may come from various source systems and may be destined for diverse target systems. Understand what those sources and sinks are and whether or not adapters already exist for those systems.

  • Determine if events are typed (known and defined at design time) or untyped (dynamically generated at runtime).

  • Understand if events will happen at a point in time, over a predetermined interval of time, or over an interval of time where the start time is known but the duration is not.

  • Understand if those source or sink systems implement the IObservable or IObserver interfaces.

  • Determine if you need to filter events on one or more conditions.

  • Determine if events will need to be aggregated to arrive at averages, sums, counts, or the like.

  • Determine if patterns or situations will need to be detected from source events.

  • Determine what windows of time are monitored, and at what intervals results will be computed.

  • Recognize that StreamInsight is a nascent and emerging Microsoft technology and that documentation and resources may seem somewhat limited relative to other, more mature products.


  • Determine throughput or event rate—the number of events that must be processed per second.

  • Determine the latency requirements—the amount of time in which events must be consumed in to produce the required results.

  • Identify the memory footprint constraints of your solution.

  • Understand availability and reliability requirements.

Management and Administration

There is currently no user interface for managing or monitoring StreamInsight.


< Full URLS for Hyperlinked Text>