In my application I have a data source which is a continuous stream of events.  I have multiple consumers of this event stream, and each consumer wants to see whole or parts of the event stream depending on their needs.
In addition I wanted to stream different types of objects and have it be extensible.  As I was exploring the design space it became really interesting to consider using DS for this.

The advantages of DS are that:

• It allows for server side complex filtering
• It allows for projections (the client can select the properties they want)
• It publishes the schema without having to distribute libraries
• It doesn’t require a backing database (it can work against Plain Ol’ CLR objects )
• It integrates with Visual Studio and other tools such as LinqPad

That said, DS is really designed around the idea of finite result sets.  The big question was: “Could I get DS to deal with infinite results?”  I am glad to report that with some subtle changes I was  able to make this work, and that it works well.

This blog post shows how I figured out how tweak DS so that it would work with infinite result sets.  This has resulted in a very easy to implement and robust rich query event monitoring system.

Implementing the Data model
Just like any DS you need to define the data model by providing a class which has properties which are of IQueryable<>.  For purposes of this article I will create an arbitrary example class.

Sample Schema

IEnumerable<> with Yield == infinite result set
To make this model visible I need a class which exposes my result set.  What I’m doing to exposing a data service around an IEnumerable<> implementation which I ask for an IQueryable<> from.  The IEnumerable/Yield combination will generate objects on demand.

In this example I am assuming there is background mechanism which is keeping the queue filled with items from the quote source.  As you can see this enumerable is an infinite result set that will continue to return results as long as the client is connected and asking for them.

To create a top level entity set for it I simply get the Enumerator and call the LINQ extension .AsQueryable().  

Complex types
Since I want to return complex types as part of the results they need to be surfaced as real entities.  To do this requires 2 things:

1. That it has a DataServiceKeyAttribute() attribute on the class
2. That there is a root level IQueryable<> entity set.

In this example I am returning these objects on the fly without a backing database, so to support the company object then I will simply add an IQueryable<Company> which returns an empty result set.

Supporting Take() by breaking OrderBy

Here is one of the big subtleties I ran into.  If I have an infinite result set like this a logical thing to want to do is ask for a subset of it by using Take(X).  It turns out that DS assumes that a Take() operation only makes sense if it is against an ordered set, and so if there is no order by clause in the expression, it will apply one automatically when $top is passed. 

Since my IQueryable is backed by POCO, it attempts to do an in memory sort on the IEnumerator, which means it tries to get all of the objects from my infinite result set.  Of course, since the end never occurs, it will never return!

To fix this I created a wrapper around the IQueryable which ignores attempts to apply order to the result set.  When the expression is being built up at some point a OrderBy expression is added to the expression tree. This is given to the source to evaluate, so my StreamingQueryable<> template simply ignores it. 

Here is the class:

To use it I now create an instance around the IQueryable I return in my model.  This makes changes my Quotes property from above to now look like this:

And allows it have Take() clauses not wait forever.

Making the Web Service streaming

OK, so now that the data model is defined we need to do some extra magic to make the data service stream an infinite result set.  The problem is that DS out of the box is built on WCF which assumes that the payload will be a closed set, thus giving it a message that it will push on the transport.

What I wanted on the other hand was a continuous HTTP stream that never ended.  If I tried to run the above service in a normal WCF service it would never return (again). 

There were 2 solutions I could see…

1. figure out how to set all of the right magic knobs to get WCF to stop doing something that is pretty innate to WCF
2. simply bypass it entirely by writing an IHttpRequest handler. 

Since the later was both simpler (no WCF stack involved) and only about 20 lines of code I opted for the latter approach.

HttpHost
It turns out that DS uses an interface IDataServiceHost to be hosted within WCF.  If I simply implement that contract I can take over the details of how to communicate with the client that is requesting data.  I created a class called HttpHost which maps between an HttpContext and the DS, enabling compression for good measure.

StreamingDataService
Now that I have HttpHost in hand, I can go about implementing my IHttpRequest handler. 

I ended up encapsulating the IHttpRequest into a single class I called StreamingDataService<>.  This class derives from DataService<T> and implements IHttpRequest aournd a HttpHost to do the binding.

This class is pretty straight forward.  It simply handles an Http request by implementing ProcessRequest() calls.  In ProcessRequest() it creates an HttpHost, binds the 2 together and calls DS.ProcessRequest(). 

Define our data service and data service host

Now that we have all of these nice helper classes, I can simply define the data service and data service host. 

To do that

1. I create a new Generic Handler (.ASHX) file
2. Replace the IHttphandler with my StreamingDataService<>

Voila! I now have a streaming data service over an infinite result set! Hooray for me!

(BTW: An excellent tool for exploring a new DS such as this is the beta version of LinqPAD (http://linqpad.com) which supports querying DS services.  Just make sure you include a .Take() clause!)

Consuming the DS from the Client Code

Add Service Reference

Now I can simply go to VS and in Add a Service Reference to my service.  VS will automagically generate proxy client code around the metadata published by my DS.  Life is good.

Querying

To query I can now use LINQ and (as long as I provide a Take() clause ) I can access the data from the data service using the strongly typed objects generated by adding the service reference.  Look at how cool this is:

This code will take the first 5 quotes for IBM it finds in the continuous result stream. 

As long as all I want to do is take a snapshot of the live stream like this, I’m done.  Of course, the whole point of this blog post was that I wanted continuous streaming results, so I can’t stop here, I must press on. 
Processing Continuous Results
I wanted to be able to continuously monitor the live stream that I’m getting.  The out of box DataServiceQuery<> object from DS doesn’t work because like so many other components it wants to get a finite result set.

After much poking around I discovered that I could leverage DS to build the Url I want, but bypass the client logic for connecting to the data service and processing results.

The key is that I access the Url by simply calling ToString() on query. Cool! 

With the Url now in hand I can simply issue my own HTTP GET of the Url and get back a stream in the HttpRequest.ResponseStream of my data in either JSON or ATOM format. I can then simply create an XmlTextReader() around that stream and parse my data on the fly.

Ultimately, I ended up writing a class I called StreamingDataServiceQuery<T> which implemented IEnumerable<T> and used my own JSON deserializer I wrote to materialize the objects from a JSON stream. 

I did this for a number of reasons:

1. JSON is about 1/3 the size of XML serialized
2. Complex objects are stored inline in JSON which is easier to parse than ATOM, where the inner collections come as a later chunk of data that needs to be stitched up

Here is my hacked up JSONMaterializer

And my StreamingDataServiceQuery<> template:

This ultimately allowed me to write client code like this:

The foreach loop never ends (it’s infinite after all) but this allows me to created a server side filter on a live stream of data with very clear concise efficient code.

If you want to download a complete working solution with client code including silverlight download this solution

http://cid-8dc85127f8ce2f12.skydrive.live.com/self.aspx/Dropbox/StreamingDataServices.zip

I want to give a gianormous shout out to Pablo Castro, Architect on the ADO.NET/WCF Data Services team who provided a ton of insight and help as I went off down this rabbit hole.  He is a fantastic person to put up with my pestering questions and I couldn’t have figured all of this out without his help.

Unfortunately, there is a fly in the ointment when it comes to consuming data across multiple domains.  The problem is that the the implementation of ADO.NET Data Services (Astoria) doesn’t work cross domain from SilverLight, even if there is a clientaccesspolicy.xml file.  Apparently the network stack that it uses is not the normal Silverlight one.(See Pablo Castro’s blog for details)

This is a bummer, as now instead of having a great rich client object model around your database you are back in the business of having to create a proxy service implementation. 

This blog post shows a very easy way to work around this problem by create a proxy service which is fully functional object model with very little code.

Let’s start with an hypothetical project:

• http://foo.com/FooStore.svc  is Astoria service called FooStore.svc around a LINQ database. 
• http://myapp.com has a web service with a silverlight application in it where you want to consume FooStore.svc from.

Step 1- Add Service Reference to the Database Web Service

Add a service reference for http://foo.com/FooStore.svc to your web service http://myapp.com.  Visual Studio will create a DataServiceContext class around that service and the entities it provides.

This gives your application a full blown LINQ based web service with the ability to consume and interact with the database on foo.com. Cool!

Step 2 – Create new proxy service on the Application Web Service

Create a new ADO.NET Data Service on http://myapp.com called FooStoreProxy.svc. This will be the service that your silverlight application will consume, which you want to look exactly like FooStore.svc but be on your domain.

Step 3 – Wrap database service and override CreateDataSource()

Now you need to simply use the Astoria model which was generated in step 1 as the entity model that the new proxy service is wrapped around.  It will take anything that returns IQueryable as objects in the new service, and we have a brand new shiny one which we created in step 1.

You will probably need to override the CreateDataSource() method so you can pass your connection string to the ADO.NET Data Service you are consuming like this:

public class FooStoreProxy : DataService< FooStore>
{
    // This method is called only once to initialize service-wide policies.
    public static void InitializeService(IDataServiceConfiguration config)
    {
        // TODO: set rules to indicate which entity sets and service operations are 
        // visible, updatable, etc.
        // Examples:
    } 

    protected override FooStore CreateDataSource()
    {
        ConnectionStringsSection connectionStringsSection = 
               WebConfigurationManager.GetSection("connectionStrings") as ConnectionStringsSection;
        string connectionName = "default"; 
        return new FooStore(connectionStringsSection.ConnectionStrings[connectionName].ConnectionString);
    } 
} 

Step 4 – Add Service Reference to FooStoreProxy.svc in your Silverlight Application

Configure your silverlight app by adding a service reference to http://myapp.com/FooStoreProxy.svc This will auto-generate a client wrapper around the proxy service.

You now should be able to use the object model generated in your silverlight application to query the http://myapp.com/FooStoreProxy.svc service, which will then query the real http://foo.com/FooStore.svc service. 

If all you want to do is to have read only access you are done.  But if you want to be able to do CRUD operations then you have a couple more things will need to do.

Step 5 – implement IUpdateable on the application web service

The reason you can’t update your service is because the class doesn’t implement IUpdateable.  Luckily the generated class is a partial class, so you can just create a partial class file which adds the needed interface.

Here’s code which shows a complete implementation of IUpdateable which should be generic and work in most cases:

 // Add IUpdateable to our ADO.NET Data Services generated DataServiceContext wrapper
    public partial class FooStoreProxy : IUpdatable
    {
        #region IUpdatable Members
        public void AddReferenceToCollection(object targetResource, 
                                             string propertyName, 
                                             object resourceToBeAdded)
        {
            base.AddLink(targetResource, propertyName, resourceToBeAdded);
        }

        public void ClearChanges()
        {
            // clear out links
            foreach (var link in base.Links)
                base.DetachLink(link.Source, link.SourceProperty, link.Target);
            
            // clear out entities
            foreach (var entity in base.Entities)
                base.Detach(entity.Entity);
        }

        public object CreateResource(string containerName, string fullTypeName)
        {
            object obj = Activator.CreateInstance(this.GetType().Assembly.GetType(fullTypeName));
            AddObject(containerName, obj);
            return obj;
        }

        public void DeleteResource(object targetResource)
        {
            DeleteObject(targetResource);
        }

        public object GetResource(IQueryable query, string fullTypeName)
        {
            object resource = query.Cast<object>().SingleOrDefault();

            // fullTypeName can be null for deletes
            if (fullTypeName != null && resource.GetType().FullName != fullTypeName)
                throw new Exception("Unexpected type for resource");
            return resource;
        }

        public object GetValue(object targetResource, string propertyName)
        {
            Type type = targetResource.GetType();
            PropertyInfo propInfo = type.GetProperty(propertyName);
            if (propInfo == null)
                throw new Exception("Can't find property");
            object val = propInfo.GetValue(targetResource, null);
            return val;
        }

        public void RemoveReferenceFromCollection(object targetResource, 
                                                  string propertyName, 
                                                  object resourceToBeRemoved)
        {
            base.DeleteLink(targetResource, propertyName, resourceToBeRemoved);
        }

        public object ResetResource(object resource)
        {
            Detach(resource);
            return resource;
        }

        public object ResolveResource(object resource)
        {
            // just reflect the object back, since that's what we 
            // gave them in CreateResource
            return resource;
        }

        public void SaveChanges()
        {
            base.SaveChanges();
        }

        public void SetReference(object targetResource, string propertyName, object propertyValue)
        {
            base.SetLink(targetResource, propertyName, propertyValue);
        }

        public void SetValue(object targetResource, string propertyName, object propertyValue)
        {
            PropertyInfo propInfo = targetResource.GetType().GetProperty(propertyName);
            if (propInfo == null)
                throw new Exception("Can't find property");
            propInfo.SetValue(targetResource, propertyValue, null);
        }

        #endregion
    }
}

You should now be able to do CRUD operations against your store! 

Step 5 – Add IExpandProvider implementation to the application web service

There is one thing left which does not work quite yet.  ADO.NET Data Services has a special LINQ command called Expand().  This lets you control the shape of the results which come back to reduce roundtrips.  To make the proxy handle Expand correctly you need to implement IExpandProvider interface on FooStoreProxy.  Luckily, while the documentation is thin the implementation is quite straight forward.

Here is an implementation which you will need to replace only the switch block with each entity that is exposed by your service:

        /// 
        /// IExpandProvider 
        /// implements the $expand syntax by simply passing through to the proxied ADO.NET Data Service
        /// 
        #region IExpandProvider Members
        public IEnumerable ApplyExpansions(IQueryable queryable, ICollection expandPaths)
        {
            string expand = "";
            foreach (var expandSegmentCollection in expandPaths)
            {
                string segment = "";
                foreach (var expandSegment in expandSegmentCollection)
                {
                    segment += String.Format("{0}{1}", (segment.Length > 0) ? "/" : "", expandSegment.Name);
                }
                // whatever entity type this query is using needs to to use a 
                // strongly typed DataServiceQuery object to get access to the AddQueryOption...bummer
                switch (queryable.ElementType.Name)
                {
                    case "Item":
                        return ((DataServiceQuery<Item>)queryable).Expand(segment);
                    case "Person":
                        return ((DataServiceQuery<Person>)queryable).Expand(segment);
                      // add/remove entries here for each item type that is queried as a result set.                
                      // NOTE: The switch statement and DataServiceQuery<ElementType.Name> must match
                }
            }            
            // return massaged query
            return queryable;
        }
        #endregion

And there you go.  This is relatively easy to use and seems to work just great.  I would love to hear from others if this works for them too!

-Tom