See you there!

• Rx .NET
• RxJS (JavaScript)
• Rx CPP
• RxJava
• Reactive Cocoa
• RxPython
• Reactive-Dart
• Rx.rb (Ruby)

This is pretty cool, because effectively we have a cross-platform lingua franca for event processing. The Rx .NET implementation is serving as the reference implementation, but it won't be long before we see some ports innovating at their own pace.

What do people think? Should there be an implementation-agnostic Rx specification? How far away are we from an Rx Summit and a committee for standardization?

 UPDATE (3/18/2013 3:34 PM PDT): Added RxPython
 UPDATE (3/18/2013 3:44 PM PDT): Added Dart, corrected link to Reactive Cocoa
 UPDATE (3/18/2013 9:50 PM PDT): Added Rx.rb 

Good news everyone! We can! The trick is reactive coincidence. Head over to Channel 9. Stay here for examples and code.

Let's abstract out the duration pattern. Assume we have events of the following type:

public class Activity
{
    public long Id {get; private set;} 
    public int Payload {get; private set;}      

    public Activity(long id) 
    { 
        Id = id; 
        Payload = (new Random()).Next(0,100); 
    }      

    public override string ToString() 
    { 
        return String.Format("ActivityId = {0}, Payload = {1}", Id, Payload); 
    }
}

And let us generate a couple of observable sequences, one for starts and one for ends:

    var starts = Observable 
                .Timer(TimeSpan.Zero, TimeSpan.FromSeconds(1)) 
                .Select(x => new Activity(x+5)) 
                .Timestamp()
                .Publish();      

    var ends = Observable 
                .Timer(TimeSpan.Zero, TimeSpan.FromSeconds(1)) 
                .Select(x => new Activity(x)) 
                .Timestamp()
                .Publish();  

Notice I'm offsetting the Ids in starts by 5 to simulate a duration of 5 seconds. Since I'm interested in computing duration, I need a handle on when events occurred. The Timestamp() operator in Rx returns an observable sequence of type Timestamped<T>, in which you have a Timestamp property and the original T is inside a Value property. Neat!

An intuitive way to compute duration is by using a cross product and a where (or, as we data folks commonly call it, a join!):

    var crossed = from s in starts 
                  from e in ends 
                  where s.Value.Id.Equals(e.Value.Id) 
                  select new
                  { 
                      Id = s.Value.Id,
                      Duration = e.Timestamp.Subtract(s.Timestamp) 
                  };

A join, you say? Then it should be possible to also use the Join operator in Rx:

    var joined = starts.Join(ends 
                             , _ => Observable.Never<Unit>() 
                             , _ => Observable.Never<Unit>() 
                             , (s, e) => new {Start = s, End = e}) 
                       .Where(t => t.Start.Value.Id == t.End.Value.Id)
                       .Select(t => new
                            { 
                                Id = t.Start.Value.Id 
                                , Duration = t.End.Timestamp
                                             .Subtract(t.Start.Timestamp) 
                            });

What's going on here? The second and third parameter are returning a non-terminating observable sequence. This means we are assuming the duration of events in both sequences is unknown when we evaluate the join, that is, we expect to be able to correlate them for as long as the sequences exist. This is exactly the implication in our crossed expression. However, there is a variation of the duration pattern that also occurs: How much time elapsed between starts and ends (assuming we should see an end within n seconds)? In some cases, we don't want to wait forever for an answer, and would like to know if an end didn't occur within a time bound. Turns out we can use the join operator to express the same condition:

    var boundJoined = starts.Join(ends 
                                 , _ => Observable.Timer(
                                         TimeSpan.FromSeconds(6)) 
                                 , _ => Observable.Empty<Unit>() 
                                 , (s, e) => new {Start = s, End = e}) 
                            .Where(t => t.Start.Value.Id == t.End.Value.Id) 
                            .Select(t => new
                            { 
                                Id = t.Start.Value.Id 
                                , Duration = t.End.Timestamp
                                             .Subtract(t.Start.Timestamp) 
                            });

The second parameter expresses our assumption that start events are "valid" for 6 seconds, and that end events are finite. So now you have a template for computing duration given a stream of start events and a stream of end events.

Try them out! (don't forget to Connect() the starts and ends sequences :) ) [LINQPad share]

Mark Simms and I have decided to start a new habit: If we find ourselves answering a similar question thrice, we'll blog about it. Without further ado, here are a few "I'd like to compute" statements expressed as Rx queries. They are a subset from a (now old) post. If you find yourself in the presence of data being pushed to you (instead of you pulling for data), you'll find all the tools you need to process it in .NET. Here's a brief overview of Observable Sequences, and simple patterns that illustrate how we can use Rx to express computations over observable sequences.

Observable Sequences

A sequence of data can be pulled or pushed. Pull-based sequences are very familiar to us because of the years of patterns we've been using: (1) Go read a fixed amount of data from a database, (2) Create a list, (3) Iterate over a collection, etc. In .NET, the IEnumerable<T> interface represents a pull-based sequence. With a push-based sequence, one does not usually control access to the data. Instead of iterating over it, we observe it. In .NET, the IObservable<T> interface represents this notion. Observable sequences are the dual of Enumerable sequences (check out a great explanation by the Rx team). I'd like to share with you a few common patterns to create observable sequences.

Creating observable sequences

Sometimes even when a source can be cast as an enumerable, we may want to iterate over it and observe it as we read it. Notice the use of ReadLineAsync so as to not block processing in what is commonly a slow I/O operation. Also notice how clean it is in C# 5.0 to make the lambda expression in Observable.Create() asynchronous, by just writing async (observer, c) and  await  in the asynchronous call (More on Rx and .NET 4.5 here):

    public static IObservable<string> FromFile(string fileName) 
    {
        return Observable.Create<string>(async (observer, c) =>
        {
            using (var sr = new StreamReader(path: fileName))
            {
                var line = default(string); 

                try
                {
                    while ((line = await sr.ReadLineAsync()) != null
                            && !c.IsCancellationRequested )
                    {
                        observer.OnNext(line);
                    }
                    observer.OnCompleted();
                }
                catch (Exception e)
                {
                    observer.OnError(e);
                } 
            }
        });
    }

 
Another common need: periodically polling a data source or periodically producing new data. We can use Observable.Timer() and compose on top of the resulting sequence to create a new sequence:

    public static IObservable<string> PeriodicProduction(TimeSpan startTime,
                                                         TimeSpan period) 
    {
        return Observable.Timer(dueTime: startTime, period: period)
               .Select(x => String.Format("A new ({0}) string!", x));
    }    

You can imagine the timer-based pattern doing other interesting things, such as polling from a queue.

Merging observable sequences

The Merge operator performs a union of observable sequences and produces a new one. The sequences must be union-compatible, i.e., they must be of the same type. A running example: suppose we want to simulate an observable sequence (based on timer) of one or more sensors reporting a particular metric periodically. Let's use the following schema:

    public class SensorData
    {
        public DateTime StartTime { get; set; }
        public int SensorId { get; set; }
        public long Value { get; set; }

        public override string ToString()
        {
            return String.Format("StartTime: {0}, SensorId: {1}, Value: {2}"
                                 , StartTime, SensorId, Value);
        }
    }

One way to produce the desired sequence is to create a method that returns an observable sequence for a single sensor and merge an array of individual sensor sequences. You can imagine this maps to sensor networks: each sensor produces a sequence, but we really want to query over a sequence of observations from all sensors:

    public static IObservable<SensorData> CreateSensor(TimeSpan reportingFrequency
                                                      , int sensorId) 
    {
        return Observable.Timer(TimeSpan.Zero, reportingFrequency)
               .Select(x => new SensorData()
               { 
                   StartTime = DateTime.Now
                             , SensorId = sensorId
                             , Value = (new Random(DateTime.Now.Millisecond
                                                   + sensorId)).Next(0, 100)
               });
    }

    var individualSensors = new[]
    {
        Helpers.CreateSensor(TimeSpan.FromSeconds(1), 1000),
        Helpers.CreateSensor(TimeSpan.FromSeconds(1), 1001),
        Helpers.CreateSensor(TimeSpan.FromSeconds(1), 1002)
    };

    var sensors = individualSensors.Merge();  

Expressing multiple queries over an observable sequence

We are now almost ready to get dangerous. I would like to express many queries over the same sequence (without evaluating the generating function for each query). The .Publish() method on observables produces a ConnectableObservable, which is a type of Subject (see Subjects). A ConnectableObservable is cold, which means it is not producing data until we ask it to via the .Connect() method (then it's hot). This is a very useful construct, since essentially we have a one-to-many pipeline with the generating expression being evaluated only once. Put another way, if you have an observable and run two queries over it, you have two copies of the sequence being generated. If you write two queries over a ConnectableObservable, once you invoke .Connect() on the latter, there is only one copy of the underlying sequence. Sharing is good J

Let's put it all together and create an observable sequence of sensor readings and a connectable observable:

    var sensors = individualSensors.Merge().Publish();

We are now ready to write some queries. We'll do Filters and Windowed aggregates next, and when we're ready to start the data flow, we'll write

    sensors.Connect();

Filters

How do I filter a stream, keeping only specific events?

     var q1 = from e in sensors
              where e.Value > 20
              select e;

How do I filter on multiple conditions?

     var q2 = from e in sensors
              where (e.SensorId == 1001) && (e.Value > 70)
              select e;

Windows and aggregations

How can I calculate the average of all events in the last 5 seconds?

      var q3 = from w in sensors.Window(timeSpan: TimeSpan.FromSeconds(5))
              from a in w.Average(_e => _e.Value)
              select new { Time = DateTime.Now, Average = a };

How can I calculate the average of all events in the past 5 seconds, every 2 seconds?

      var q4 = from w in sensors.Window(timeSpan: TimeSpan.FromSeconds(5)
                                       , timeShift: TimeSpan.FromSeconds(2))
              from a in w.Average(_e => _e.Value)
              select new { Time = DateTime.Now, Average = a };

How can I calculate the average of the past 5 events?

      var q5 = from w in sensors.Window(count: 5)
              from a in w.Average(_e => _e.Value)
              select new { Time = DateTime.Now, Average = a };

How do I measure the rate of arriving events every two seconds?

      var q6 = from w in sensors.Window(timeSpan: TimeSpan.FromSeconds(2))
              from a in w.Count()
              select new { Time = DateTime.Now, Count = a };

How can I calculate the average of each group of events in the last 10 seconds?

      var q7 = from e in sensors
              group e by e.SensorId into g
              from w in g.Buffer(timeSpan: TimeSpan.FromSeconds(10))
              select new { Time = DateTime.Now
                         , SensorId = g.Key
                         , Average = w.Average(_e => _e.Value)};

How can I calculate the average of each group of events in the last 20 seconds, every 10 seconds?

      var q8 = from e in sensors
              group e by e.SensorId into g
              from w in g.Buffer(timeSpan: TimeSpan.FromSeconds(20)
                                , timeShift: TimeSpan.FromSeconds(10))
              select new { Time = DateTime.Now
                         , SensorId = g.Key
                         , Average = w.Average(_e => _e.Value) };

How can I find the events with the largest value every 10 seconds?

      var q9 = from w in sensors.Window(timeSpan: TimeSpan.FromSeconds(10)) 
              from a in w.Max(e => e.Value)
              select new { Time = DateTime.Now, Max = a };

How can I find the events with the two smallest values in the last 10 seconds, every 5 seconds?

      var q10 = from w in sensors.Buffer(timeSpan: TimeSpan.FromSeconds(10)
                                        , timeShift: TimeSpan.FromSeconds(5))
               from a in w.OrderBy(x => x.Value).Take(2)
               select a;

How can I detect if two events with the same ID value are received within a specific time frame, such as a 5 minute window?

      var q11 = from e in sensors
               group e by e.SensorId into g
               from w in g.Buffer(timeSpan: TimeSpan.FromSeconds(5))
               where w.Count > 2
               select new { Time = DateTime.Now, SensorId = g.Key };

Final remarks

At this point, assuming we have subscribed an observer to each query, all we need to is sensors.Connect() and watch our Rx processing come to life. If you're hacking or prototyping and want a quick way to visualize the output, try the .DumpLive() method in LINQPad. Hope these patterns are useful – and let me know if there are other examples you'd like to see. And remember, StreamInsight 2.1. has a tight integration with Rx. Sometimes you want to process observable sequences, other times you want to work over temporal streams (CTIs).

Bonus: The queries and patterns in a self-contained LINQPad file.

Earlier work on stream engines exposed me to a very practical definition of a window, in which the function that partitions the underlying stream is parameterized by two attributes: range and slide. Range defines what's in a window, and slide defines when new windows start*. In my experience, people often talk about three types of windows:

1. Tumbling window (range = slide)
2. Sliding window (range > slide)
3. Hopping window (range < slide)

Additionally, people also distinguish whether windows are defined in terms of counts or in terms of time (Count vs. Temporal window). I'll try to illustrate these types of windows with a picture in Figure 1.

Figure 1. Windowing examples.

Let the x axis represent time. In Figure 1 (a), you'll see tumbling count-windows with range = slide = 2. Notice a window opens and closes based on event occurrence. For the stream illustrated here, we get three windows, each with two elements in it. Now assume we define windowing in terms of seconds, with slide = range = 2 seconds. In Figure 1 (b), you see tumbling temporal-windows. Notice windows open and close based on time instead of occurrence. Figure 1 (c) illustrates sliding count-windows with range =2 and slide =1, and Figure 1(d) shows hopping count-windows with range = 1 and slide = 2. (Note: The Rx team likes to visualize operations using marble diagrams, which I don't use here, but may use later.)

Enough said! Let's look at some Rx code and Count windows. I realize I'm repeating content that exists in many other sources, but if you think of windows the way I do, this may help you get started expressing them using Rx. Consider the observable sequence xs in Code Fragment 1. The Window operator in Rx is one way we can implement both count-based and temporal-based windows of the three types I described. One of its overloads takes two parameters: count (range) and skip (slide), which we apply in the expression q in Code Fragment 1.

            var xs = new [] {0,0,1,2,3,2,2,3,4,5,6}.ToObservable();

            var q = from w in xs.Window(count:2, skip:2)
                    select w;

Code Fragment 1. Windowing an observable sequence.

The expression q results in a stream of type IObservable<IObservable<int>>, which is a stream of streams -- therefore every window itself is an observable stream to which I can subscribe. OK, I can't resist the temptation. Figure 2 shows the windowing in a marble diagram.

Figure 2. Marble diagram for the expression q.

Since each window is an observable, I can subscribe to them. In Code Fragment 2, I show how we can inspect each window and its contents and confirm this is a tumbling window by subscribing to each window. Output 1 shows the console output.

            int counter = 1;

            q.Subscribe(w =>
            {
                int i = counter++;
                w.Subscribe(
                    x => Console.WriteLine("Window: {0}; Value: {1}", i,x)
                    );
            });  

Code Fragment 2. Subscribing to each window.

            Window: 1; Value: 0
            Window: 1; Value: 0
            Window: 2; Value: 1
            Window: 2; Value: 2
            Window: 3; Value: 3
            Window: 3; Value: 2
            Window: 4; Value: 2
            Window: 4; Value: 3
            Window: 5; Value: 4
            Window: 5; Value: 5
            Window: 6; Value: 6

Output 1. Console output for the tumbling count-window.

Usually we want to compute something over the contents of each window, such as an average. You can apply an aggregate function to each window and obtain a stream of results back, as shown in Code Fragment 3. The expression q2 computes the average of every window and results in a stream of type IObservable<double>. As before, we can inspect the stream by subscribing to it and printing out to console (see Output 2).

             var q2 = from w in xs.Window(2,2)
                      from x in w.Average()
                      select x;

                         q2.Subscribe(w => 
                 Console.WriteLine("Average: {0}", w)
                 );

Code Fragment 3. Windowed Average.

                        Average: 0
            Average: 1.5
            Average: 2.5
            Average: 2.5
            Average: 4.5
            Average: 6

Output 2. Windowed average over xs.

Temporal windows you ask? No problem. Window has an overload that takes two TimeSpan objects, timeSpan (range) and timeShift (slide). Moreover, if you find the idea of a stream of streams to be a little daunting and you just want a stream of collections (with each collection, List<T>, representing a window), you can look at the Buffer operator.

I keep encountering these simple patterns over and over again. Hopefully this taxonomy (Tumbling, Sliding, or Hopping on one dimension; Count, and Temporal on another) is useful to you. And as your stream processing needs grow, bear in mind Window and Buffer do many more wonderful things, such as using one stream to signal closing a window on another **.

---------------------------

 * Two nice papers that discuss various Windowing semantics:
- Li et al. Semantics and evaluation techniques for window aggregates in data streams. SIGMOD 05.
- Botan et al. SECRET: a model for analysis of the execution semantics of stream processing systems. VLDB 2010.

** Additional great reads on Window, Buffer, and Rx:
- Window and Buffer on MSDN.
- Campbell, Lee. Introduction to Rx. (Online book)

Thanks to Mark Simms, Christian Martinez, and Nina Sarawgi for their suggestions while preparing this post.

When it comes to Big Data definitions, I really enjoy Gartner's. It says Big Data "is the term adopted by the market to describe extreme information management and processing issues which exceed the capability of traditional information technology along one or multiple dimensions to support the use of the information assets." This statement is ambiguous enough to allow anybody with a problem they can't solve with what they have can feel they have a Big Data problem. And in my experience, this is mostly what ends up happening. Pain points about availability, resiliency, distribution, latency, and incremental aggregation are somehow bucketed behind Big Data. And then you see people trying to understand how they'll leverage a batch-oriented technique (which scales and distributes very easily) to deal with computations that are better served by a continuous event-processor, a well-tuned database, a non-relational indexed store, or a combination of all of them.

You see, I'm skeptical of the Big Data silver bullet. In fact, when a conversation starts gravitating towards a single-system solution, I feel we're failing to understand the problem(s) a customer may actually be trying to solve.

The way I see it, the term is not helping us have a discussion about the actual business and technology problems enterprises of all sizes are now worrying about. The size, cost, and volume of today's problems are actually making the bulk of enterprise customers face a new design challenge, in which they truly must develop distributed, heterogeneous solutions. To me, this is what truly lurks behind these conversations: businesses of not-so-massive size must now build a new class of systems for which they may be ill-equipped. How many companies can comfortably claim a decade or more of distributed systems know-how?

Try this: Next time you're in a Big Data conversation, substitute "Distributed System" for "Big Data". Chances are by doing so you'll realize these problems are fortunately not new. You'll suddenly recognize that our field has decades of research into this space, ready to be invoked when appropriate. All of the sudden, you'll start weaving through the actual "why is this hard" questions, instead of the "what this could mean" discussions.

With this in mind, also remember economics, of course, play an immensely important role here. It is unlikely that most companies will build their own datacenters. It is unlikely they will develop their own frameworks and implementations of the actor model to encapsulate the necessary primitives to build these new class of systems. It is also unlikely they will find a turnkey solution for their specific problem. It is, however, very likely they will find platforms and ecosystems that provide most of the pieces required to run this new class of businesses. And the last part is where I think the discussions should be focused: what are those pieces? What are the tradeoffs? How do they integrate with existing investments? How do they scale out gracefully? And more importantly, how can a business use those pieces and start on a happy path from day one?

I know distributed systems programming is tough. I know it doesn't sound sexy. But I think that's the hard problem: the current set of business requirements is best served by this class of applications, which most of us are currently ill-equipped to write. To me, that's the real problem we should be tackling, instead of getting lost in the buzz.

Consider a data source emitting a strictly monotonic sequence of natural numbers. Assume the data source can also filter out odd or even numbers. We would like to stimulate the data source to flip the filter based on a condition detected by a data consumer. We need a mechanism to enable communication contrary to the stream flow. For this purpose we will use a BehaviorSubject.

A BehaviorSubject is a type of subject that remembers the last event it has received. One can also declare an initial state, such as

 1: BehaviorSubject<bool> behaviorSubject = new BehaviorSubject<bool>(true);

We will use the last state of the subject as the gating logic that decides whether to produce odd or even numbers. Here is a sample observable source implementing this behavior:

 1: var data = from x in Observable.Interval(TimeSpan.FromSeconds(0.5)).Take(100)

 2:            where (x % 2 == 0) == behaviorSubject.First()

 3:            select x;

Last, we would like to express the logic that sends an event to the subject as a result of a condition in the consumer. Consider, for example, flipping from odd numbers to even numbers every time the consumer receives 10 events. Here is an example implementation of this logic:

 1: var printer = data

 2:     .Do(Console.WriteLine)

 3:     .Select((_, count) => count)

 4:     .Where(count => (count + 1) % 10 == 0)

 5:     .Do(_ => behaviorSubject.OnNext(!behaviorSubject.First()));

Notice the use of .Do() to side-effect (or “T”) console output, and the call to OnNext()on the BehaviorSubject.

Two practices to notice: (1) We exclusively use the subject to relay the state change from the consumer to the producer, and (2) we use side-effects to conduct this communication.

https://blogs.msdn.com/b/meek/archive/2012/03/21/sequences-and-streaminsight-initiating-a-computation.aspx