• Cloud scenarios
• Design and architecture questions
• Troubleshooting and diagnostics
• Sharing best practices from customer deployments 

Look for us daily in the Windows Azure area in The Big Room, daily from 9am – 6pm and on Friday till 3pm.  

Let’s start with our “working” top category query.

What is actually happening here in terms of the event flow is described in the diagram below.  The input stream, weather, is broken up into substreams by the station code, which then have Tumbling windows applied over top of them.  This calculates the number of events in each category, which is then passed to the snapshot window (and then the order by and take).

This works so long as the input events to the snapshot window do not overlap.  If they do, older “calculations” can be counted against the new ranking.   For example, if a Hopping window is used, such as this query:

The following effect is observed – at points in the event stream the results of older Tumbling windows are passed to the snapshot window for ordering.  This can produce incorrect results, as seen in this diagram.

In order to prevent this from happening, we need to stop the output of the hopping windows from overlapping in the snapshot window.  There are a number of ways to do this, but the easiest is to convert the output of the Hopping window back into point events via the ToPointEventStream method.

Which will produce the desired results:

Note that the diagrams don’t entirely line up; I’ll polish this up a bit and add some information on tie-breaking before posting to the AppFabric CAT blog site next week.

Working with LINQPad

• StreamInsight- Creating a LINQPad data context (the short version)
• StreamInsight- Creating a custom data context for LINQPad, chock full of weathery goodness (the not so short version)
• StreamInsight- Reading from other data contexts into StreamInsight with LINQPad
• StreamInsight- Understanding dynamic query composition
• StreamInsight- Where did that query result come from (using the Event Flow Debugger in LINQPad)

Query Patterns

• StreamInsight Query Patterns- Fun with process data – calculating the wind chill factor
• StreamInsight Query Pattern- Find the top category (using Order By, Take and ApplyWithUnion)

Minor, yet annoying error messages

• StreamInsight- Obscure LINQ error - Stream other than apply input stream is cannot be referenced
• StreamInsight- Obscure LINQ error - The field 'Key' of type 'IKey`1' either contains a nested me

Some recent favorites from around the web

• Permanent link to 5 Quick Steps For Trying Out StreamInsight with LINQPad
• StreamInsight Sequence Integration- Five Easy Pieces
• LINQ Macros

Working up my pipeline of interesting things to write about in the next month or two.  Other than finishing (almost there!) the rest of the “101 LINQ Samples” for StreamInsight, wondering if there’s any suggestions out there on interesting topics.

1. Store a large number of events (> 1,000,000) in a human readable format (by human readable, meaning being able to visually review the data through commonly available tools).  This generally means text, nothing being friendlier to sharing, converting, manipulating, etc.
2. Provide a sufficiently small and portable data size (MB, not GB) to be easily downloaded and moved across the network (i.e. the data must be self-contained)
3. Be read performant – provide the ability to retrieve at least 60k events per second on my desktop (as writing the encoded data sets is a 1-time activity, not concerned about optimizing performance).

The intent for this system was to be able to store and replay “canned” data sets for StreamInsight applications.  The performance metrics, in order of priority were:

1. File size
2. Read throughput (target is 100k+ events / second)
3. Generic applicability – needs to be relatively non-specific to any conformant data set (see below)
4. CPU utilization
5. Memory consumption

The “tricks” (constraints, really) I was able to take advantage of were:

• Forward only access to data (i.e. did not need random access to specific data elements).
• Only flat data record schema (StreamInsight doesn’t support nested types in streams).

The reference data set consisted of 733 MB of data downloaded from Environment Canada’s National Climate Archives web site, in XML format.  My first pass through performance tuning consisted of evaulating the following techniques:

1. Raw XML (to establish a baseline for performance tuning)
2. Optimized XML (remove extraneous fields from source XML)
3. SQL Compact
4. CSV files
5. Compressed CSV files (via the most excellent GZipStream class, introduced in .NET 4.0)

In order to perform the testing I needed a benchmark framework which could execute a deterministic test set and capture associated performance data.  This framework needed to:

• Provide a pluggable interface for inserting the read/write code which would implement each store/format.
• Measure the key performance metrics

Download the test frameworks here.

Stage 1 - Evaluating File Size

First pass with the full test set were pretty clear – compressed CSV and XML files look to be the clear winner for size (I threw in compressed XML just to be fair).  We execute the tests by opening up and executing the PerformanceTests project.

Or, if you prefer the raw data:

Based on the first of our ranked performance criteria (file size), we can now winnow down the field to two clear front runners – compressed XML and CSV format.  Now we move on to reviewing read throughput for these two techniques.

Stage 2 - Evaluating Read Throughput

Now that we’ve winnowed down the field to two candidates, let’s go and test them for read performance.  We’ll run four tests here:

• XML Serializer – Compressed
• XML Serializer - Uncompressed
• CSV reader – Compressed
• CSV reader – Uncompressed

Note that we expect the XmlSerializer read performance to suck, as it is designed (and intended) to pull the entire file into memory and provide random access as an in-memory object.  Remember one of the constraints we can take advantage of  - Forward only access to data – we don’t have to pay the price to load the entire data set into memory for random access.  I could have written a forward-parser using XmlTextReader or XElement, but to be honest, making it generic was more work than I wanted to bother with (remember – this has to be readily applicable to all forms of StreamInsight compatible data).  

Or, if you prefer the raw data:

Note that we only lose ~ 13% of our read throughput by leveraging the GZipStream class.  That’s pretty awesome

The numbers for our CSV reader weren’t actually all that good in comparison to the supposedly sucky XmlSerializer.  Keep in mind that XmlSerializer is as optimized as it gets, and we haven’t done any optimization work on the CSV reader.  Let’s dig a little deeper in to that code to see if we can’t get some solid performance improvements. It’s not measured in the test framework, but there’s a huge memory usage difference between the two (which makes sense, as XmlSerializer loads the whole thing into memory), Using the Red Gate Memory Profiler, companion tool to the Performance Profiler which I use extensively in the next section, the peak values are: XmlSerializer 94 MB CSV reader 267 kB Looks like with some performance optimization, our CSV reader will be the winner!

Stage 3 – Optimizing the Compressed CSV reader

We’re now pretty close to our goals, but have plenty of room to push the performance envelope.  Up to this point we’ve been using relatively simple measures and metrics to quantify and evaluate performance (file sizes via the FileInfo class, and elapsed time via the Stopwatch class).  Now that we’re diving a bit deeper into performance tuning, it’s time to pull out the big guns.  There are a number of excellent performance and profiling tools available for .NET (including the Visual Studio Profiler).  My personal favorite is Red Gate’s excellent, ANTS Performance Profiler, part of their line of .NET Development tools. 

One of the reasons this is my go-to application for performance investigations is that when I’m working with customers whom don’t have (or can’t provide) an instrumented build, or cannot install Visual Studio on a live box – it’s a great lightweight experience for understanding how your application is burning CPU and memory.

To follow along through this next section, download an evaluation copy of the ANTS Performance Profiler.

The first thing we’ll do is turn off the other tests in the framework so we can focus on the Compressed CSV path, and disable the file writing (i.e. we just want to execute the file reading code path).  To streamline this, we fork the framework into a a separate project, PerformanceTests_Profile.

Next, we run the test framework in the ANTS Performance Profiler:

• Start a copy of the ANTS Performance Profiler by clicking on Start, All Programs, Red Gate, ANTS Performance Profiler.
• If the profiler configuration dialog (as per the image below) does not pop up, click on File, New Profiling Session.
• In the .NET executable selection, type in the name of the executable to test (or use the ellipsis to bring up a file browse dialog).
• Select the profiling mode Line-level and method-level timings; all methods (most detailed).
• Select the SQL and file I/O mode Do not record SQL and file I/O performance.
• Click on Start Profiling.

The profiler will then start up the framework application, and read back the compressed CSV file.  Wait until the profiler completes to get a full data set (note that this will take longer than normal due to the overhead of profiling). 

Let’s start by looking at the Wall time view – ensure that the 3rd drop down box is set to wall-clock time.  This shows two hot paths:

• Waiting for Synchronization.  This is busy-wait time while the process waits for the I/O infrastructure (as we’re using synchronous I/O).
• ISequenceRunner.ReadEventsBack.  This is the interesting path right now – where are we spending time.

To focus on just our code paths, switch back to CPU time.  Waiting for synchronization disappears, and we see MoveNext() – our enumerator method – popping to the top with 97% of time with children (remember that the .Count() method forces the enumerator to run through the entire file).  This is where we’ll focus our efforts.

Download the profile trace to walk through this section.  We see the performance culprits being threefold: [35.8%] RuntimePropertyInfo.SetValue() [19.6%] System.Convert.ChangeType() [14.5%] DateTime.Parse() In order to get our performance up to where it needs to be, let’s focus on each of these in order (biggest bang for the buck first).

Performance fix 1 – Removing reflection / PropertyInfo.SetValue() and Convert.ChangeType()

Our first approach will be to remove the use of reflection and change type for assigning values.  Since we need to maintain a flexibility, we need another “generic” method of assigning values to variables.  Since functional programming is cool, let’s approach this problem by defining a function delegate Action<string[], T> that is responsible for assigning values to an object of type T given an array of strings. 

This involves updating the signature for ReadEvents<T> as below (the signature change on lines 1-2, and invoking the function delegate on line 25).

The function definition is fairly straightforward:

This preserves our need for the Csv reading code to be generic (as the parsing code is externally defined and passed in), as well as keeping complexity reasonable.  Let’s have a look at our updated performance results.

Before: 33920 events / second After: 65804 events / second Much better!  Download the profile trace to walk through this section.  We see the performance culprits being threefold: [29.8%] Assign function (of which 14.9% is DateTime.Parse) [15.0%] DateTime.Parse [ 7.6%] Double.Parse()

Based on our profiler results, parsing DateTime values account for 30% of our current CPU time.  We have a number of potential performance fixes we can apply here:

• Convert the DateTime storage format to a more easily parsed value (such as raw ticks)
• Parse the date time value using a more optimal parsing routine (such as ParseExact)

Performance fix 2 – Better DateTime parsing

Let’s start by investigating the use of a more optimized DateTime parsing routine – that provided by the XmlConvert class.  As XML serialization has been heavily optimized as .NET has evolved, it stands to reason that there may be some optimal parsing algorithms buried in there.  Let’s find out

In order to use XmlConvert.ToDateTime, we also need to change the format in which we encode date time values by altering the WriteEvents method and recreating the data file:

Note the use of XmlConvert on line 3 (using ToString(DateTimeOffset)) and on line 11 (using ToString(DateTime, XmlDateTimeSerializationMode)).

Next, we need to update our parsing methods in ReadEvents<T>

And in the assignment function:

Let’s go ahead and run this and observe the results.

Before: 65804 events / second After: 87744 events / second Now we’re getting there! Download the profile trace to walk through this section.  We see the performance culprits being twofold: [21.8%] XmlConvert.DateTime for retrieving the event start time value. [21.2%] Assign function (10% of which is Double.Parse(), and only 2.215% of which is XmlConvert.ToDateTime). Weird.  Apparently the format in which we’re storing the StartTime values is more expensive to parse (it stands to reason they’re different – StartTime value is being parsed from a DateTimeOffset record.  Wait – we wrote out a DateTimeOffset, but are reading in a DateTime.  Could that make a difference?

Performance fix 3 – Parsing StartTime - DateTimeOffsets not DateTimes 

Going back to our change in ReadEvents<T>(), let’s modify it to use XmlConvert.ToDateTimeOffset instead.

Before: 87744 events / second After: 116392 events / second Wow.  Huge difference!  Download the profile trace to walk through this section.  We now see the performance culprits being centered in the assign function: [13.1%] Double.Parse()  [3.3%] Int32.Parse(). Almost there.  Let’s make a few small changes to finish this out – let’s switch in the XmlConvert functions for Double.Parse() and Int32.Parse().

Performance fix 4 – Parsing numbers with XmlConvert

Ok, let’s modify the assignment method to use XmlConvert instead of Double.Parse:

Before: 116392 events / second After: 112083 events / second Uh-oh – performance regression!  Looks like the Double.Parse() is the better function  Download the profile trace to walk through this section.  Looks like our version 3 is the most performant reader.  Having achieved all of our performance goals – let’s call it done!

Stage 4 – Review

Now that we’ve finished our performance exploration, let’s run the full test suite and review our results and key learnings.

Or, for the raw data:

The key learnings are:

• Know what to measure.  Understand what the key performance metrics are, and their relative priority.  If you don’t understand where you’re going, you won’t know when you’re there.
• Know how to measure how you got there.  Having the right measurement framework in place (matching up with your performance metrics), and the right tooling is critical! 
  • In this case we were measuring for performance, so a performance profiler was the critical tool that let us achieve a 4x performance gain with about half an hour’s worth of work. 
  • Having this detailed, repeatable insight is not an option – if you want to be able to tackle performance issues with confidence (and deliver on your projects) you MUST have the right tools, and be able to understand how to profile and tune your code.
• Know when to stop.  Performance tuning can become a never ending death spiral if you let it.  Understand your performance goals, work towards them, but realize when you start encountering regressions past your performance goals.
• It’s a balance between conflicting priorities. 

For more reading on Red Gate’s ANTS Performance profiler check out:

• ANTS Performance Profiler 6.1
• Learning ANTS Performance Profiler - 6.1 - Worked example: profiling the performance of an application

AD485A SQLCAT: Next Generation Real Time Data Processing using StreamInsight

Thursday, November 11 4:00 PM - 5:15 PM

StreamInsight ships with SQL Server 2008 R2, and provides complex event processing for tackling high volume, low latency analytics and data processing challenges. This talk will explain the business goals and target scenarios for StreamInsight, its architecture, and how to leverage StreamInsight to complement extant data processing and BI capabilities by exploring a series of customer implementations (including Sunday Night Football, and the 2010 Winter Olympics) for large scale real time web analytics.

AD386A SQLCAT: Optimizing application workload & performance by leveraging AppFabric Cache along with SQL Server

Tuesday, November 9 3:00 PM - 4:15 PM

Today, more and more data is being exposed via interactive web applications such as social networking, news, sports, video sharing, etc. Such web sites can now be accessed via different devices such as mobile phones, netbooks and easily exposed to 1000s of users. Hence, application architectures must leverage distributed application caching along with the database tier for building highly scalable, high throughput and highly available systems. In this session, we will cover some of the lessons learned based on customer engagements for AppFabric Cache, Microsoft's distributed caching solution, with some performance data.

AD211CT AppFabric Cache and SQL Server 2008

Wednesday, November 10 1:30 PM - 2:10 PM

With all the hype about NoSQL going around in the SQL community its about time we looked at a Microsoft's distributed cache-aside solution, AppFabric Cache. In this session we will briefly go over what AppFabric Cache is, what it can and cannot do, as well as some of the compelling features that make it a complimentary product to SQL Server. We will go over some real-world examples on how AppFabric Cache can co-exist with SQL Server and increase performance. In addition, we will compare the speed of the AppFabric Cache in comparison to SQL Server 2008 in both queries and inserts/updates. This chalk-talk will go more in-depth into some of the information presented in the SQLCAT Optimizing application workload & performance by leveraging AppFabric Cache along with SQL Server session.

In addition, don’t forget to check out all of the great SQLCAT presentations!

• DAT301 - Introduction to Complex Event Processing with SQL Server 2008 R2 StreamInsight

• DAT301-LNC - Building Operational Intelligence Solutions with Microsoft SQL Server and StreamInsight

• DAT302 - Advanced SQL Server 2008 R2 StreamInsight Query Design Techniques

I’ll be at SQL PASS 2010 with the rest of my CAT (SQL and AppFabric) folks, hanging out in the SQL clinic and discussing interesting StreamInsight scenarios and solutions.  I’ll be presenting a StreamInsight session on Thursday about some of the cool stuff we’ve been doing regarding real-time web analytics and telemetry (that I’ve been alluding to in a few recent blog posts).

Hope to see folks there!

• StreamInsight: Obscure LINQ error - The field 'Key' of type 'IKey`1' either contains a nested member, or is a type that is not supported as event type field.  A walkthrough of resolving a little issue I ran into while building out a project this week, related to proper use and definition of grouping keys.

• StreamInsight: Getting started with using the (Event Flow) debugger, viewing diagnostics, and exposing the management service.  Everything you ever wanted to know about ensuring you can connect to the StreamInsight management service with the Event Flow Debugger.

• Monitoring process control event streams for values that exceed a given threshold (for example, a valve’s pressure exceeding a certain safety threshold).
• Enriching utility usage information in a smart metering scenario with user and geography information (for example, to allow visualization of power usage by geography).

This reference data is commonly held in some form of repository, such as SQL Server, a process historian’s metadata store (e.g. OSIsoft’s PI Asset Framework), which then needs to be integrated into a StreamInsight query.

In my previous post, I showed how to use the TimeImport capability on streams to synchronize a slow moving reference stream with a fast moving data stream.  I did not, however, walk through the process of creating a reference stream from a relatively static source (such as a database table or file).  I intend to now correct that minor, and somewhat deliberate oversight

The project referenced in this blog post, along with the source data files is available here.

Reference data can be integrated into StreamInsight in a couple of different ways, each with their own pros and cons:

Let’s look at an example of each technique, step by step. 

The Scenario

For this illustration, let’s consider the following scenario:

• Stream of data events, each containing a user ID, an activity code and a status (user 7, logged in, success).
• Set of user metadata, contained in a file (for the purposes of illustration – a database would be a more likely source).  This metadata contains information such as user ID, user name, etc.
• Want to annotate the data events with the user metadata.

We’ll use the following data sets:

Data Events

Or, in the .csv format that we’ll use:

StartTime,EndTime,UserId,ActivityCode,Status
6/25/2009 0:00:00,,1,Logon,Success
6/25/2009 0:00:01,,2,Logon,Fail
6/25/2009 0:00:05,,1,Browse,Success
6/25/2009 0:00:06,,2,Logon,Success

Reference Events

Or, in the .csv format that we’ll use:

StartTime,EndTime,UserId,UserName,Location
6/24/2009 0:00:00,,1,Fred Jones,Seattle
6/24/2009 0:00:01,,2,Bob Murphy,Portland

User-Defined Lookup Function

With a user defined lookup function, we need to load the list of metadata into an associative array (i.e. a hashtable), and create a lookup function that allows us to grab user metadata from the dictionary on demand.

Note: Why did I use the hard coded dictionary rather than reading from a file?  I wanted to illustrate the technique of a truly static data source.  If you want to see the code that parses a .csv file into a dictionary object suitable for using in a lookup function, this is covered in the appendix section of the blog).

Note: Why didn’t I simply join the dictionary with the data stream?  StreamInsight needs to join streams both relationally and temporally.  Since a dictionary isn’t a stream (and has no concept of time, it’s not possible to directly join.  Converting a reference source into a stream and joining is the other technique covered in the next section.

Upon running this function with the supplied data files, we observe the merged output:

Press <enter> to close application
REF,Point,12:00:00.000
        Activity = Logon
        Location = Seattle
        Name = Fred Jones
        Status = Success
        UserId = 1
REF,Point,12:00:01.000
        Activity = Logon
        Location = Portland
        Name = Bob Murphy
        Status = Fail
        UserId = 2
REF,Point,12:00:05.000
        Activity = Browse
        Location = Seattle
        Name = Fred Jones
        Status = Success
        UserId = 1
REF,Point,12:00:06.000
        Activity = Logon
        Location = Portland
        Name = Bob Murphy
        Status = Success
        UserId = 2

Reference Data Stream (static data)

Using the more robust reference data stream approach, we:

• Use a TextFileInputFactory to import the contents of the text file
• Convert the series of point events into “infinite” signals.
• Join the reference stream with the data stream.

Note: this addresses conditions where the reference data DOES NOT CHANGE.  This is not a very real-world scenario – we’ll cover the additions required to incorporate changing data in the next blog post, showing how to pull data from SQL Server. 

Note: I don’t use the CTI import technique in this example.  This is due to the text file input adapter sending an “infinite time” CTI when it reaches the end of the file.  If we were using a continuous or changing reference data source (such as SQL Server), I’d need to employ that technique.

Upon running this query with the supplied data sets, we observe:

STREAM,Point,12:00:00.000
        Activity = Logon
        Location = Seattle
        Name = Fred Jones
        Status = Success
        UserId = 1
STREAM,Point,12:00:01.000
        Activity = Logon
        Location = Portland
        Name = Bob Murphy
        Status = Fail
        UserId = 2
STREAM,Point,12:00:05.000
        Activity = Browse
        Location = Seattle
        Name = Fred Jones
        Status = Success
        UserId = 1
STREAM,Point,12:00:06.000
        Activity = Logon
        Location = Portland
        Name = Bob Murphy
        Status = Success
        UserId = 2

Compare and Contrast

For sake of clarity, let’s put the two techniques side by side.

In this diagram we see two streams (data and reference) joined together to create a joined stream, with a LINQ syntax such as:

CepStream<SensorReading> dataStream = ...;
CepStream<SensorMetadata> metadataStream = ...;

var joinedQuery = from e1 in dataStream
                    join e2 in metadataStream
                    on e1.SensorId equals e2.SensorId
                    select e1;

From the diagram above, we see that we’ll only get two output events for (1) and (2).  This is as a result of how the StreamInsight engine produces output – based on the application time (which is updated by the CTIs).  In this case the engine sees:

• I have a time of T5 on the data stream
• I have a time of T0 (ish) on the reference stream
• Since I have joined these two streams together I can only produce output when I have received all of the events for a given time period.  Since the reference stream lags behind the data stream, I can only output events at the speed of the reference stream.

In general, this is absolutely the correct behavior.  However, we know something about the reference stream that the StreamInsight engine does not – that the reference stream is slowly changing, and we don’t want to wait for it to catch up.  That is to say, we want to produce output at the pace of the data stream.  How do we do this?

By importing the CTIs from the data stream into the reference stream:

This operation is performed by one of the overloads on the CepStream<>.Create() method, as per the syntax below (reading from this csv file as the sample data source, and this csv file as the sample reference source).  The entire project can be downloaded from here (look in the SimpleJoin.cs class).

////////////////////////////////////////////////////////////////////
// Create a time import settings definition that states that a stream
// will import its CTI settings from dataStream 
var timeImportSettings = new AdvanceTimeSettings(null,
    new AdvanceTimeImportSettings("dataStream"),
    AdvanceTimePolicy.Adjust);

////////////////////////////////////////////////////////////////////
// Create a reference data stream from the refStream.csv file; use
// the CTIs from the dataStream as defined by the timeImportSettings
// object
CepStream<SensorMetadata> metadataStream = CepStream<SensorMetadata>
    .Create("refStream", typeof(TextFileReaderFactory),
        new TextFileReaderConfig()
        {
                CtiFrequency = 1,
                CultureName = CultureInfo.CurrentCulture.Name,
                Delimiter = ',',
                InputFileName = "refStream.csv"
        }, EventShape.Point, timeImportSettings);

Note that this syntax assumes the presence of a “dataStream” stream.  At this point, we should be able to join the two streams and see a steady flow of output.  However, if we simply want to look at the raw output of metadata stream as per:

var rawData = metadataStream.ToQuery(cepApp, "MetadataStream", "",
    typeof(TracerFactory), traceConfig, EventShape.Interval,
    StreamEventOrder.FullyOrdered);

Here is the error that is returned:

Error in query: Microsoft.ComplexEventProcessing.ManagementException: Advance time import stream 'dataStream' does not exist. ---> 

Microsoft.ComplexEventProcessing.Compiler.CompilerException: Advance time import stream 'dataStream' does not exist.

Huh?  What do you mean the time import stream does not exist?  I already defined it!  What’s actually happening here is that the import stream has not yet been physically joined with the stream.  In order to resolve this issue, you need to join the two streams before binding an output adapter (i.e. creating a query).

////////////////////////////////////////////////////////////////////
// Create a join of the two streams, and bind the output to the 
// console
var joinedQuery = from e1 in dataStream
                    join e2 in metadataStream
                    on e1.SensorId equals e2.SensorId
                    select e1;

var query = joinedQuery.ToQuery(cepApp, "JoinedOutput", "",
    typeof(TracerFactory), traceConfig, EventShape.Interval,
    StreamEventOrder.FullyOrdered);

Ok, now let’s look at the output:

REF,Interval from 06/25/2009 00:00:00 +00:00 to 06/25/2009 00:00:00 +00:00:,MySensor_1001,1001,14
REF: CTI at 06/25/2009 00:00:00 +00:00
REF: CTI at 06/25/2009 00:00:09 +00:00
REF: CTI at 12/31/9999 23:59:59 +00:00

Huh?  Where’s all of my output?  Look at the data sources in question – the reference data is a sequence of point events.  If we want to truly use it as a reference stream, we need to convert the series of point events into edge events (i.e. this value is good until replaced by a new value).  In order to do this, we need to apply an AlterEventDuration and a Clip operator:

// Convert the point events from the reference stream into edge events
var edgeEvents = from e in metadataStream
                        .AlterEventDuration(e => TimeSpan.MaxValue)
                        .ClipEventDuration(metadataStream, (e1, e2) => (e1.SensorId == e2.SensorId))
                    select e;

What this bit of code does is:

• Stretch the point events out to infinity (i.e. these reference values are good “forever”)
• Clip each point event by any arriving event with the same sensor ID.  For example, given a value of (1001, SensorId_1001), if another event later arrives with the values (1001, MySensor), the initial event will be clipped off and the new value will be MySensor

Then, putting the whole thing together:

// Convert the point events from the reference stream into edge events
var edgeEvents = from e in metadataStream
                        .AlterEventDuration(e => TimeSpan.MaxValue)
                        .ClipEventDuration(metadataStream, (e1, e2) => (e1.SensorId == e2.SensorId))
                    select e;

    ////////////////////////////////////////////////////////////////////
    // Create a join of the two streams, and bind the output to the 
    // console
var joinedQuery = from e1 in dataStream
                    join e2 in edgeEvents
                    on e1.SensorId equals e2.SensorId
                    select new
                    {
                        SensorId = e1.SensorId,
                        Name = e2.Name,
                        Value = e1.Value
                    };

Results in:

REF,Interval,12:00:00.000,12:00:00.000,,MySensor_1001,1001,14
REF: CTI at 06/25/2009 00:00:00 +00:00
REF,Interval,12:00:01.000,12:00:01.000,,MySensor_1001,1001,4
REF,Interval,12:00:02.000,12:00:02.000,,MySensor_1001,1001,77
REF,Interval,12:00:03.000,12:00:03.000,,MySensor_1001,1001,44
REF,Interval,12:00:04.000,12:00:04.000,,MySensor_1001,1001,22
REF,Interval,12:00:05.000,12:00:05.000,,MySensor_1001,1001,51
REF,Interval,12:00:06.000,12:00:06.000,,MySensor_1001,1001,46
REF,Interval,12:00:07.000,12:00:07.000,,MySensor_1001,1001,71
REF,Interval,12:00:08.000,12:00:08.000,,MySensor_1001,1001,37
REF,Interval,12:00:09.000,12:00:09.000,,MySensor_1001,1001,45
REF: CTI at 12/31/9999 23:59:59 +00:00

Following in the great tradition of other learn through example LINQ example pages, such as 101 LINQ Samples, here’s the first in a series of posts on StreamInsight query examples.  This will first show up on this blog, then get migrated over to a centralized site on the StreamInsight developer center.  We’ve got a fairly hefty queue we’re working through, but if there are specific examples or query patterns of interest, please post them in the comments.

Each query pattern will consist of a description of the input set (.csv file samples for most of these, for ease of use), the full query and the output result.  For more complicated query structures the query pattern will also be described visually, using a Visio stencil (to be described in future posts). 

The full application framework that I use to run all of these queries, associate with the data sources, etc, is available here.  Illustration of the techniques in this codebase will be left for a future post. 

In this post:

• Filtering
  • How do I filter a stream, keeping only specific events?
  • How do I filter on multiple conditions?
• Windows and Aggregates
  • How can I calculate aggregates, such as averages, over a set of events?
  • How can I calculate the average of all events in the last 5 seconds?
  • How can I calculate the average of all events in the past 5 seconds, every 2 seconds?
  • How can I calculate the average of all events in the past 5 seconds, every time a new event arrives?
  • How can I calculate the average of the past 5 events?
  • How do I measure the rate of arriving events every two seconds?
  • How can I calculate the average of each group of events in the last 10 seconds? 
  • How can I calculate the average of each group of events in the last 20 seconds, every 10 seconds? 
  • How can I calculate the average of each group of events in the past 10 seconds, every time a new event in that group arrives?
  • How can I find the events with the largest value every 10 seconds?
  • How can I find the events with the two smallest values in the last 10 seconds, every 5 seconds?
  • How can I find the group of events with the highest average value in the last 10 seconds?
  • How can I find the group of events with the highest maximum value in the last 10 seconds, every 5 seconds?
  • How can I detect if two events with the same ID value are received within a specific time frame, such as a 5 minute window?

[Where] - How do I filter a stream, keeping only specific events?

var filterQuery = from e in inputStream
    where e.Value > 20
    select e;

[Input] Result:

SimpleFilter,Point,12:00:02.000,1001,77
SimpleFilter,Point,12:00:03.000,1001,44
SimpleFilter,Point,12:00:04.000,1001,22
SimpleFilter,Point,12:00:05.000,1001,51
SimpleFilter,Point,12:00:06.000,1001,46
SimpleFilter,Point,12:00:07.000,1001,71
SimpleFilter,Point,12:00:08.000,1001,37
SimpleFilter,Point,12:00:09.000,1001,45

How do I filter on multiple conditions?

var filterQuery = from e in inputStream
where e.Value > 70 && e.SensorId == 1001
select e;

[Input] Result:

MediumFilter,Point,12:00:01.001,1001,77
MediumFilter,Point,12:00:03.010,1001,72
MediumFilter,Point,12:00:12.082,1001,73
MediumFilter,Point,12:00:14.145,1001,75
MediumFilter,Point,12:00:14.154,1001,71
MediumFilter,Point,12:00:15.163,1001,74
MediumFilter,Point,12:00:15.172,1001,73

How can I calculate aggregates, such as averages, over a set of events?

All aggregates, such as averages, min, max, etc are calculated over a set of events.  Sets of events in StreamInsight are defined as those events falling into a window.  Windows operate over a period of time (hopping, tumbling and sliding), or over a specific number of events (count).

How can I calculate the average of all events in the last 5 seconds?

var query = from window in inputStream.TumblingWindow(
    TimeSpan.FromSeconds(5),
    HoppingWindowOutputPolicy.ClipToWindowEnd)
select new 
{
    Average = window.Avg(e => e.Value)
};

[Input] Result:

SimpleTumblingWindow,Point,12:00:00.000,32.2
SimpleTumblingWindow,Point,12:00:05.000,50

How can I calculate the average of all events in the past 5 seconds, every 2 seconds?

var query = from window in inputStream.HoppingWindow(
                TimeSpan.FromSeconds(5), TimeSpan.FromSeconds(2), 
                HoppingWindowOutputPolicy.ClipToWindowEnd)
            select new
            {
                Average = window.Avg(e => e.Value)
            };

[Input] Result:

SimpleHoppingWindow,Point,11:59:56.000,14
SimpleHoppingWindow,Point,11:59:58.000,31.66666
SimpleHoppingWindow,Point,12:00:00.000,32.2
SimpleHoppingWindow,Point,12:00:02.000,48
SimpleHoppingWindow,Point,12:00:04.000,45.4
SimpleHoppingWindow,Point,12:00:06.000,49.75
SimpleHoppingWindow,Point,12:00:08.000,41

How can I calculate the average of all events in the past 5 seconds, every time a new event arrives?

var query = from window in inputStream
    .AlterEventDuration(e => TimeSpan.FromSeconds(5))
    .SnapshotWindow(SnapshotWindowOutputPolicy.Clip)
select new
{
    Average = window.Avg(e => e.Value)
};

[Input] Result:

SimpleSlidingWindow,Point,12:00:00.000,14
SimpleSlidingWindow,Point,12:00:01.000,9
SimpleSlidingWindow,Point,12:00:02.000,31.66
SimpleSlidingWindow,Point,12:00:03.000,34.75
SimpleSlidingWindow,Point,12:00:04.000,32.2
SimpleSlidingWindow,Point,12:00:05.000,39.6
SimpleSlidingWindow,Point,12:00:06.000,48
SimpleSlidingWindow,Point,12:00:07.000,46.8
SimpleSlidingWindow,Point,12:00:08.000,45.4
SimpleSlidingWindow,Point,12:00:09.000,50
SimpleSlidingWindow,Point,12:00:10.000,49.75
SimpleSlidingWindow,Point,12:00:11.000,51
SimpleSlidingWindow,Point,12:00:12.000,41
SimpleSlidingWindow,Point,12:00:13.000,45

How can I calculate the average of the past 5 events?

Aggregates over a set number of events are produced using Count windows.  As of StreamInsight v1, Count Windows cannot use the built-in aggregate functions (Count, Avg, Min, Max), but must instead use user-defined aggregates.  The code sample below shows the use of a simple averaging function with a count window with the user defined aggregate.

var query = from window in inputStream.CountByStartTimeWindow(
            5, CountWindowOutputPolicy.PointAlignToWindowEnd)
        select new
        {
            Average = window.UserDefinedAverage(e => e.Value)
        };

 public class SimpleAggregate : CepAggregate<double, double>
{
    public override double GenerateOutput(IEnumerable<double> payloads)
    {
        return payloads.Average();
    }
}

public static class MyExtensions
{
    [CepUserDefinedAggregate(typeof(SimpleAggregate))]
    public static double UserDefinedAverage<InputT>(this CepWindow<InputT> window, 
        Expression<Func<InputT, double>> map)
    {
        throw CepUtility.DoNotCall();
    }

}

[Input] Result:

SimpleCountWindow,Point,12:00:04.000,32.2
SimpleCountWindow,Point,12:00:05.000,39.6
SimpleCountWindow,Point,12:00:06.000,48
SimpleCountWindow,Point,12:00:07.000,46.8
SimpleCountWindow,Point,12:00:08.000,45.4
SimpleCountWindow,Point,12:00:09.000,50

How do I measure the rate of arriving events every two seconds?

var query = from window in inputStream.TumblingWindow(
    TimeSpan.FromSeconds(2),
    HoppingWindowOutputPolicy.ClipToWindowEnd)
select new
{
    Count = window.Count()
};

[Input] Result:

SimpleEventCount,Point,12:00:00.000,54
SimpleEventCount,Point,12:00:02.000,54
SimpleEventCount,Point,12:00:04.000,54
SimpleEventCount,Point,12:00:06.000,54
SimpleEventCount,Point,12:00:08.000,54
SimpleEventCount,Point,12:00:10.000,54
SimpleEventCount,Point,12:00:12.000,54
SimpleEventCount,Point,12:00:14.000,54
SimpleEventCount,Point,12:00:16.000,54

How can I calculate the average of each group of events in the last 10 seconds? 

For this example, we’ll calculate the average of each sensor every 10 seconds.  We do this by first grouping the events by the sensor ID, then performing the same aggregation operation from the average without grouping (with the addition of the group key – in this case, the sensor ID).

var query = from e in inputStream 
group e by e.SensorId into sensorGroups
from window in sensorGroups.TumblingWindow(
    TimeSpan.FromSeconds(10),
    HoppingWindowOutputPolicy.ClipToWindowEnd)
select new
{
    SensorId = sensorGroups.Key,
    Average = window.Avg(e => e.Value)
};

[Input] Result:

GroupedTumblingWindow,Point,12:00:00.000,42.1,1001
GroupedTumblingWindow,Point,12:00:00.000,34.6,1002
GroupedTumblingWindow,Point,12:00:00.000,39.7,1003
GroupedTumblingWindow,Point,12:00:00.000,50.9,1004
GroupedTumblingWindow,Point,12:00:00.000,30.7,1005
GroupedTumblingWindow,Point,12:00:00.000,38.3,1006
GroupedTumblingWindow,Point,12:00:00.000,36.8,1007
GroupedTumblingWindow,Point,12:00:00.000,41.7,1008
GroupedTumblingWindow,Point,12:00:00.000,34.4,1009
GroupedTumblingWindow,Point,12:00:10.000,46.375,1001
GroupedTumblingWindow,Point,12:00:10.000,40.2,1002
GroupedTumblingWindow,Point,12:00:10.000,36.1,1003
GroupedTumblingWindow,Point,12:00:10.000,37.3,1004
GroupedTumblingWindow,Point,12:00:10.000,31.8,1005
GroupedTumblingWindow,Point,12:00:10.000,41.4,1006
GroupedTumblingWindow,Point,12:00:10.000,40.3,1007
GroupedTumblingWindow,Point,12:00:10.000,43.5,1008
GroupedTumblingWindow,Point,12:00:10.000,43.3,1009

How can I calculate the average of each group of events in the last 20 seconds, every 10 seconds? 

 var query = from e in inputStream
group e by e.SensorId into sensorGroups
from window in sensorGroups.HoppingWindow(
    TimeSpan.FromSeconds(20), TimeSpan.FromSeconds(10),
    HoppingWindowOutputPolicy.ClipToWindowEnd)
select new
{
    SensorId = sensorGroups.Key,
    Average = window.Avg(e => e.Value)
};

[Input] Result:

GroupedHoppingWindow,Point,11:59:50.000,42.1,1001
GroupedHoppingWindow,Point,11:59:50.000,34.6,1002
GroupedHoppingWindow,Point,11:59:50.000,39.7,1003
GroupedHoppingWindow,Point,11:59:50.000,50.9,1004
GroupedHoppingWindow,Point,11:59:50.000,30.7,1005
GroupedHoppingWindow,Point,11:59:50.000,38.3,1006
GroupedHoppingWindow,Point,11:59:50.000,36.8,1007
GroupedHoppingWindow,Point,11:59:50.000,41.7,1008
GroupedHoppingWindow,Point,11:59:50.000,34.4,1009
GroupedHoppingWindow,Point,12:00:00.000,44,1001
GroupedHoppingWindow,Point,12:00:00.000,37.0,1002
GroupedHoppingWindow,Point,12:00:00.000,38.1,1003
GroupedHoppingWindow,Point,12:00:00.000,44.8,1004
GroupedHoppingWindow,Point,12:00:00.000,31.2,1005
GroupedHoppingWindow,Point,12:00:00.000,39.7,1006
GroupedHoppingWindow,Point,12:00:00.000,38.4,1007
GroupedHoppingWindow,Point,12:00:00.000,42.5,1008
GroupedHoppingWindow,Point,12:00:00.000,38.4,1009
GroupedHoppingWindow,Point,12:00:10.000,46.3,1001
GroupedHoppingWindow,Point,12:00:10.000,40.2,1002
GroupedHoppingWindow,Point,12:00:10.000,36.1,1003
GroupedHoppingWindow,Point,12:00:10.000,37.3,1004
GroupedHoppingWindow,Point,12:00:10.000,31.8,1005
GroupedHoppingWindow,Point,12:00:10.000,41.4,1006
GroupedHoppingWindow,Point,12:00:10.000,40.3,1007
GroupedHoppingWindow,Point,12:00:10.000,43.5,1008
GroupedHoppingWindow,Point,12:00:10.000,43.3,1009

How can I calculate the average of each group of events in the past 10 seconds, every time a new event in that group arrives?

var query = from e in inputStream
    .AlterEventDuration(e => TimeSpan.FromSeconds(10))
group e by e.SensorId into sensorGroups
from window in sensorGroups
    .SnapshotWindow(SnapshotWindowOutputPolicy.Clip)
select new
{
    SensorId = sensorGroups.Key,
    Average = window.Avg(e => e.Value)
};

[Input] Result:

<snip>

GroupedSlidingWindow,Point,12:00:17.017,39.3,1008
GroupedSlidingWindow,Point,12:00:17.018,39.4,1009
GroupedSlidingWindow,Point,12:00:17.019,43,1001
GroupedSlidingWindow,Point,12:00:17.020,43.1,1002
GroupedSlidingWindow,Point,12:00:17.021,37.8,1003
GroupedSlidingWindow,Point,12:00:17.022,41.8,1004
GroupedSlidingWindow,Point,12:00:17.023,33.7,1005
GroupedSlidingWindow,Point,12:00:17.024,39.2,1006
GroupedSlidingWindow,Point,12:00:17.025,40.3,1007
GroupedSlidingWindow,Point,12:00:17.026,38.9,1008
GroupedSlidingWindow,Point,12:00:17.027,40.8,1009
<snip>

How can I find the events with the largest value every 10 seconds?

var query = (from window in inputStream.TumblingWindow(
    TimeSpan.FromSeconds(10),
    HoppingWindowOutputPolicy.ClipToWindowEnd)
from e in window
orderby e.Value descending
select e).Take(1);

[Input] Result:

TopK_TumblingDescending,Point,12:00:00.000,1006,80
TopK_TumblingDescending,Point,12:00:00.000,1007,80
TopK_TumblingDescending,Point,12:00:00.000,1009,80
TopK_TumblingDescending,Point,12:00:10.000,1007,81

Note that for the first group of events, the are three events with the same maximum value (80) in the same window of time.  Thusly, all of the events with that maximum value are returned.

How can I find the events with the two smallest values in the last 10 seconds, every 5 seconds?

var query = (from window in inputStream.HoppingWindow(
   TimeSpan.FromSeconds(10), TimeSpan.FromSeconds(5),
   HoppingWindowOutputPolicy.ClipToWindowEnd)
from e in window
orderby e.Value ascending
select e).Take(2);

[Input] Result:

TopK_HoppingAscending,Point,11:59:55.000,1004,2
TopK_HoppingAscending,Point,11:59:55.000,1007,1
TopK_HoppingAscending,Point,11:59:55.000,1008,2
TopK_HoppingAscending,Point,12:00:00.000,1001,1
TopK_HoppingAscending,Point,12:00:00.000,1001,1
TopK_HoppingAscending,Point,12:00:00.000,1003,1
TopK_HoppingAscending,Point,12:00:00.000,1007,1
TopK_HoppingAscending,Point,12:00:05.000,1001,1
TopK_HoppingAscending,Point,12:00:05.000,1001,1
TopK_HoppingAscending,Point,12:00:05.000,1003,1
TopK_HoppingAscending,Point,12:00:10.000,1004,2
TopK_HoppingAscending,Point,12:00:10.000,1004,2
TopK_HoppingAscending,Point,12:00:10.000,1006,2
TopK_HoppingAscending,Point,12:00:10.000,1007,2
TopK_HoppingAscending,Point,12:00:15.000,1004,2
TopK_HoppingAscending,Point,12:00:15.000,1007,2

Note the same effect herein – multiple events having identical minimum values in the window.

How can I find the group of events with the highest average value in the last 10 seconds?

This query is performed in two stages.  In the first query we determine the average value of each sensor over the past 10 seconds (using a group by, a tumbling window and an Avg()).  Then we take the output of that aggregate and perform a snapshot on top of that (against each set of events from the first query), sort it and take the top value.

var avg = from e in inputStream
    group e by e.SensorId into sensorGroups
    from window in sensorGroups.TumblingWindow(
        TimeSpan.FromSeconds(10),
        HoppingWindowOutputPolicy.ClipToWindowEnd)
    select new
    {
        SensorId = sensorGroups.Key,
        Average = window.Avg(e => e.Value)
    };

var query = (from win in avg
        .SnapshotWindow(SnapshotWindowOutputPolicy.Clip)
    from e in win
    orderby e.Average descending
    select e).Take(1);

[Input] Results:

TopK_GroupTumblingDescending,Point,12:00:00.000,50.9,1004
TopK_GroupTumblingDescending,Point,12:00:10.000,46.3,1001

How can I find the group of events with the highest maximum value in the last 10 seconds, every 5 seconds?

var avg = from e in inputStream
    group e by e.SensorId into sensorGroups
    from window in sensorGroups.HoppingWindow(
        TimeSpan.FromSeconds(10), TimeSpan.FromSeconds(5),
        HoppingWindowOutputPolicy.ClipToWindowEnd)
    select new
    {
        SensorId = sensorGroups.Key,
        Maximum = window.Max(e => e.Value),
        Minimum = window.Min(e => e.Value)
    };

var query = (from win in avg
        .SnapshotWindow(SnapshotWindowOutputPolicy.Clip)
    from e in win
    orderby e.Maximum descending
    select e).Take(1);

[Input] Result:

TopK_GroupHoppingDescending,Point,11:59:55.000,80,6,1006
TopK_GroupHoppingDescending,Point,11:59:55.000,80,1,1007
TopK_GroupHoppingDescending,Point,11:59:55.000,80,4,1009
TopK_GroupHoppingDescending,Point,12:00:00.000,80,6,1006
TopK_GroupHoppingDescending,Point,12:00:00.000,80,6,1006
TopK_GroupHoppingDescending,Point,12:00:00.000,80,1,1007
TopK_GroupHoppingDescending,Point,12:00:00.000,80,1,1007
TopK_GroupHoppingDescending,Point,12:00:00.000,80,2,1009
TopK_GroupHoppingDescending,Point,12:00:00.000,80,4,1009
TopK_GroupHoppingDescending,Point,12:00:05.000,80,2,1006
TopK_GroupHoppingDescending,Point,12:00:05.000,80,6,1006
TopK_GroupHoppingDescending,Point,12:00:05.000,80,1,1007
TopK_GroupHoppingDescending,Point,12:00:05.000,80,2,1009
TopK_GroupHoppingDescending,Point,12:00:10.000,81,2,1007
TopK_GroupHoppingDescending,Point,12:00:15.000,81,2,1007
TopK_GroupHoppingDescending,Point,12:00:15.000,81,2,1007
TopK_GroupHoppingDescending,Point,12:00:20.000,81,2,1007

Again, note the StreamInsight engine delivering the values corresponding to the top maximum value (80 or 81).

How can I detect if two events with the same ID value are received within a specific time frame, such as a 5 minute window?

To implement this query, we use a sliding window to look at arriving events in a 5 minute window, grouped by their ID value.  We then count the number of events in the window with that ID, and filter out groups that have less than 2 events.  Or, to say it another say, we only want groups (sensors) that have more than 1 event, thus removing groups with only 1 event (there are no groups with zero :).

var eventCount = from e in inputStream.AlterEventDuration(e => TimeSpan.FromMinutes(5))
     group e by e.SensorId into sensorGroups
     from win in sensorGroups.SnapshotWindow(SnapshotWindowOutputPolicy.Clip)
     select new
     {
         SensorId = sensorGroups.Key,
         Count = win.Count()
     };

var query = from e in eventCount
where e.Count >= 2
select e;

[Input] Result:

DetectMultipleInWindow,Point,12:05:16.206,4,1008
DetectMultipleInWindow,Point,12:05:16.207,4,1009
DetectMultipleInWindow,Point,12:05:16.208,3,1001
DetectMultipleInWindow,Point,12:05:16.209,3,1002
<snip>