Microsoft Power Query is an Excel add-in that enhances the self-service Business Intelligence experience in Excel by simplifying data discovery and access. Power Query enables users to easily discover, combine, and refine data for better analysis in Excel. Power Query includes a public search feature that is currently intended for use in the United States only.

This project was previously known as "Data Explorer". The new name reflects the direction that we are taking, plus, it happens to match my initials - so from now own I get to be PQ(PQ).

Regards,

PQ(PQ)

This obviously got our attention, so we ran the CSS selector profiler (again, Chrome comes with it preloaded in the developer tools). It turns out that there are some regular expression based selectors that are taking 95% of the calculation time. For example:

[data-sprite$="0"]:before { ... }
 
[data-sprite$="13"]:before { ... }

[class*="o-content"].role-link { ... }

These selectors walk every single element in the DOM tree and do regular expression matching on its attributes. These searches cannot be indexed and therefore can be extremely slow. Because browsers need to restyle on certain events, it made our UI felt as if the javascript handlers are taking forever to run, while in fact the handlers weren’t even called yet – the browser is busy restyling and not firing the events!

In our case the fix was simple: these selectors were part of an obsolete feature and were returning no matches. Simply deleting these CSS selectors boosts our UI responsiveness nearly 10 times in certain cases. Essentially we were paying a huge cost for a bunch of dead CSS selectors that nobody was using at all.

At the end of the day, a lesson learnt for all of us. So next time when you profile your Web App, make sure you look at the entire picture before jumping into javascript profiling, and also don’t forget: dead CSS is truly deadly!

Data Service State

When I talk about the “State of the client”, what I mean is one DataServiceContext instance containing a graph of entities, and a set of root level collections (may or may not be DataServiceCollections) that serve as “entry points” into the graph. Consider your typical master-detail page bound to a query of Customers expanding each customer’s Orders, in this scenario there is only one root level collection containing the customers. When you want to capture the state this client is in, you must consider both the context and the collection, although they will both eventually lead to the same set of entities. Hence the API we added in the CTP release requires the caller to pass in both. In this iteration we kept the same API (with a bit of renaming/refactoring around the return type), and I think it works best given the scenario we are going after. More details on that below. The new API will capture the sematic of “serialization”, and will return a string representing the serialized state. You can save this string to isolated storage or the application state, depending on how large your context is (larger states should be saved to isolated storage). These APIs are:

String DataServiceState.Serialized(DataServiceContext context, Dictionary<String, Object> rootCollections);
DataServiceState DataServiceState.Deserialize(String serializedState);

Graph, Navigation, Links, Relationships

Serialization of entity graphs, or whatever you might call it, is the most important scenario develops are facing today that the CTP did not address properly. The essence of the problem is universal, not just limited to the data service world. It’s how we represent references to the same object instance from multiple edges (including itself). If you’ve tried our CTP library and used the DataServiceState class then you’ve probably ran into this problem – and probably without knowing that you have. The symptom is non-trivial, you will most likely get an exception during deserialization complaining about “An entity with the same identity has already been tracked”, or worse, you could start making changes to your entities through the binding collections but not able to save these changes (SaveChanges does nothing). These are all caused by the fact that reference to the same entity instance has been deserialized into multiple copies.

Fortunately, the DataContractSerializer (used by NetCF to “tombstone” the application state) already has an API to deal with references. The way to do it on the NetCF is to mark ALL entities in the graph with DataContractAttribute, and set “IsReference” to true in the attribute declaration. In fact you can do this today with the CTP library and it will handle graphs nicely, although it will be a pain to do this by hand – note that if you mark a class with DataContract, then you must also mark its members with DataMember, since data contract uses an Opt-In approach. Because of this, we are also updating our code-gen assembly to automatically emit these attributes for you.

This does mean that you now absolutely have to bundle up everything related to your graph as a package and serialize that in one call. If you break away from this restriction then the graph will be broken, and the multi-instance problem is back. For example, if one of your class implements IXmlSerializable, and happens to have links back to the graph (a navigation property, for instance), then when DataContractSerializer encounters this object, it calls the WriteXml method, at which point the graph has been broken and you cannot continue to preserve reference for entities that are already written out. This is one restriction that one must live with – if you want to use IXmlSerializable on one of the entities, then you must use it on ALL entities, and do reference resolution on your own.

Projections and Server Driven Paging

Projection comes into play when you want to save a DataServiceQueryContinuation instance, the issue here is the additional “client-side” information that are captured by the query continuation are not easily serializable. These information are required to deserialize payloads returned from executing the continuation. For example, if you have a projection that look like:

from c in context.Customers
select new CustomerDisplay
{
    Name = c.FirstName + “ “ + c.LastName,
    Phone = c.Telephone == null ? “-“ : c.Telephone 
}

This is an example of what we called a non-entity projection, which means an entity type is been projected into a complex type. Here information sent to the server is captured by the URI:

service.svc/Customers?$select=”FirstName, LastName, Telephone”

As you can see the uri itself is missing the necessary information to construct an instance of CustomerDisplay class. This missing information is referred to as “Projection Plan”, or “Materialization Plan”, which takes the form of an expression tree. It was built from the original query expression tree, and transformed into a set of Materialization calls. Expressions trees are in general non-serializable, so we must discard this during serialization. Because of this, serialization for DataServiceQueryContinuation of queries containing projections that introduce additional client-side information is not supported.

There is the other category of projections that we refer to them as “entity projections”, which means they only contain narrowing down a type, rather than transformation. For example:

from c in context.Customers
select new NarrowCustomer
{
    ID = c.ID,
    Name = c.Name,
    Orders = new DataServiceCollection<NarrowOrder>(
        from o in c.Orders
        select new NarrowOrder
        {
            ID = o.ID
            Customer = o.Customer
        })
}

The above query can be materialized by just executing the resulting URI directly, since the narrowing semantics is captured by the $select query options. This is a supported scenario. However, this is one more complication. If you examine the output URI for this query, you’ll discover that the back reference to o.Customer is missing from the URI (it didn’t output $select=”Orders/Customer”). This is an unfortunate side-effect for this round. We’ll definitely be looking out how to make the experience better.

As a side note. I advice that you adopt a principal where everything you do on the context should be able to carry across the wire and deliver to the server side, either in the URI or the payload. A transforming projection cannot be expressed as an URI and carried out on the server side as of today, therefore in this case I advice that you bring whatever necessary information down to the client through entity projections first, then transform the result using Linq to Objects instead.

Multithreading

One should note that whenever you call Serialize, we take a snapshot of the context at that instance of time. This means if you are in the middle of an Asynchronous operation, the context will likely be in a transient state, and you might later have problems recovering from this situation. The general advice here is that you should save at certain “check points” where the state is known to you. For example, the WP7 development guideline advices that you save state whenever you navigate away from a page. This could be one solution out of many that are possible. Another possible check point is after all async operation completes. This also means that you definitely do not want to wait until the deactivating event and save there, since the event is triggered by the user and could happen at anytime.

Guidance on Tombstoning the DataServiceContext

The following are key take-away points to help you better support tombstoning in your data service app:

1. Mark all entity types as [DataContract(IsReference=true)], mark the member properties as [DataMember]

2. Do not use transforming projections, instead, do the transformation on entity projection results in memory.

3. Call DataServiceState.Serialize (DataServiceState.Save in the CTP) when you know the context is in a steady state – i.e., not in a middle of asynchronous operation.

4. In deactivation event, save the serialized string to IsolatedStorage if your context is large (100+ entities). Save to application state if your context is small (faster).

Cache Variation and Static Service Caching

ASP.net applications has the ability to cache the generated output on the server side. When a next matching request comes in, the output will then be delivered straight from the cache, rather than invoking the handler (calling data service). Note that this behavior applies to GET requests only. Note the word “matching” requests. How we match an request to a cached item is essential to delivering correct data to the customers. For a static (non-updating, we’ll take a look at caching for writable services too) WCF data service endpoint, the output will change depending on the URI path, all of the query parameters, and headers as well (accept, charset, etc.). ASP.net output caching has this notion of “VaryBy…”, which essentially means how we match incoming requests to items in the cache table (of course, non-matching items are added to the table). MSDN has an article that discusses asp.net caching in detail, so I won’t repeat what the parameters are here.

Let’s setup our example using a standard Northwind service over Entity Framework, everything is very standard at this point:

namespace DataServiceCache 
{ 
    [ServiceBehavior(IncludeExceptionDetailInFaults=true)] 
    public class NorthwindService : DataService<NorthwindEntities> 
    { 
        public static void InitializeService(DataServiceConfiguration config) 
        { 
            config.DataServiceBehavior.MaxProtocolVersion = DataServiceProtocolVersion.V3; 
            config.SetEntitySetAccessRule("*", EntitySetRights.All); 
            config.SetServiceOperationAccessRule("*", ServiceOperationRights.All); 
            config.UseVerboseErrors = true; 
        } 
    } 
}

Next, we override the OnStartProcessingRequest method to set the cache policy:

protected override void OnStartProcessingRequest(ProcessRequestArgs args) 
{ 
    base.OnStartProcessingRequest(args);

    HttpContext context = HttpContext.Current;    // set cache policy to this page 
    HttpCachePolicy cachePolicy = HttpContext.Current.Response.Cache;

    // server&private: server and client side cache only 
    cachePolicy.SetCacheability(HttpCacheability.ServerAndPrivate);

    // default cache expire: never 
    cachePolicy.SetExpires(DateTime.MaxValue);

    // cached output depends on: accept, charset, encoding, and all parameters (like $filter, etc) 
    cachePolicy.VaryByHeaders["Accept"] = true; 
    cachePolicy.VaryByHeaders["Accept-Charset"] = true; 
    cachePolicy.VaryByHeaders["Accept-Encoding"] = true; 
    cachePolicy.VaryByParams["*"] = true;

    cachePolicy.SetValidUntilExpires(true); 
}

We assume that this service is static here, (never changes shape), so we set the expire date to never expire (although ASP.net should auto adjust this back to expire in 1 year). Fire up the service now to test the cache – one indicator of whether the feed is cached is by examining the “Updated” timestamp. For debugging and testing purposes, let’s add a call count to OnStartProcessingRequest to make sure data service is never called for subsequent requests:

private const string processedRequestCount = "ProcessedRequestCount";

protected override void OnStartProcessingRequest(ProcessRequestArgs args) 
{ 
    if (HttpContext.Current.Application.Get(processedRequestCount) == null) 
    { 
        HttpContext.Current.Application.Set(processedRequestCount, 1); 
    } 
    else 
    { 
        int count = (int)HttpContext.Current.Application.Get(processedRequestCount); 
        HttpContext.Current.Application.Set(processedRequestCount, count + 1); 
    }

    base.OnStartProcessingRequest(args); 
    … 
}

We can retrieve this count via service operation. Note how we can control the service operation variation individually to “no cache”, this will cause the requests to /GetProcessedCount to always go through data service.

[WebGet] 
public int GetProcessedCount() 
{ 
    HttpContext.Current.Response.Cache.SetCacheability(HttpCacheability.NoCache);

    int count = Convert.ToInt32(HttpContext.Current.Application.Get(processedRequestCount));

    HttpContext.Current.Application.Set(processedRequestCount, 0); 
    return count; 
}

Next, let’s quickly write a client –side app to test this:

var context = new NorthwindEntities(new Uri("https://localhost:85/NorthwindService.svc"));

int callCount = context.Execute(new Uri("https://localhost:85/NorthwindService.svc/GetProcessedCount")).Single();

Console.WriteLine("Initial call count {0}", callCount);

IQueryable[] queries = new IQueryable[] { 
    context.CreateQuery("Customers").Skip(10).Take(10), 
    context.CreateQuery("Customers") , 
    context.CreateQuery("Orders").Skip(10).Take(10) , 
    context.CreateQuery("Orders") , 
    context.CreateQuery("Customers").Expand("Orders"), 
};

foreach (var q in queries) ((DataServiceQuery)q).Execute(); 
foreach (var q in queries) ((DataServiceQuery)q).Execute(); 
foreach (var q in queries) ((DataServiceQuery)q).Execute();

callCount = context.Execute(new Uri("https://localhost:85/NorthwindService.svc/GetProcessedCount")).Single(); 
Console.WriteLine("Call count after batch querying {0}", callCount);

The output is:

Initial call count 1
Call count after batch querying 6

As you can see, invoking the query twice will not hit the data service twice. The second request is served directly from the cache. Also, ASP.net is smart enough to cache a different page for each of the entity set, and for each variation of the parameter.

Cache Dependency and Writable Service Caching

So far we’ve looked up caching for static services. Not many services in the world are truly static, which makes caching kind of pointless if you cannot expire an cached item dynamically. The ASP.net output cache uses the idea of “Dependencies”, namely, an output cache item can take dependency on a data cache item – something you have full control over. When the latter one expires, the output cache will automatically expire too. This gives us a way to signal when we should clear the cache, and enables service writers to fine tune their expiration policy for better performances.

The basic idea is this, we can separate responses into cache-able groups, insert a dummy item into the data cache for each of the group, and then setup dependencies accordingly. Whenever we see a verb that’s not a GET, that means change has happened to the data source. We can then decide which group to refresh by expiring the corresponding item in the data cache. Let’s take the simplest approach for our example here – we expire ALL items related to the service whenever an update happens.

// group key 
private const string cacheDependencyItemKey = "DataServiceCacheItem";

protected override void OnStartProcessingRequest(ProcessRequestArgs args) 
{ 
    if (HttpContext.Current.Application.Get(processedRequestCount) == null) 
    { 
        HttpContext.Current.Application.Set(processedRequestCount, 1); 
    } 
    else 
    { 
        int count = (int)HttpContext.Current.Application.Get(processedRequestCount); 
        HttpContext.Current.Application.Set(processedRequestCount, count + 1); 
    }

    base.OnStartProcessingRequest(args);

    HttpContext context = HttpContext.Current;

    if (context.Request.HttpMethod.Equals("POST", StringComparison.OrdinalIgnoreCase) || 
        context.Request.HttpMethod.Equals("MERGE", StringComparison.OrdinalIgnoreCase) || 
        context.Request.HttpMethod.Equals("PUT", StringComparison.OrdinalIgnoreCase) || 
        context.Request.HttpMethod.Equals("DELETE", StringComparison.OrdinalIgnoreCase)) 
    { 
        // if we are making changes to this service, expire all caches 
        context.Cache.Remove(cacheDependencyItemKey); 
    } 
    else 
    { 
        Debug.Assert(context.Request.HttpMethod.Equals("GET", StringComparison.OrdinalIgnoreCase)); 
                
        // set cache policy to this page 
        HttpCachePolicy cachePolicy = HttpContext.Current.Response.Cache;

        // server&private: server and client side cache only 
        cachePolicy.SetCacheability(HttpCacheability.ServerAndPrivate);

        // default cache expire: never 
        cachePolicy.SetExpires(DateTime.MaxValue);

        // cached output depends on: accept, charset, encoding, and all parameters (like $filter, etc) 
        cachePolicy.VaryByHeaders["Accept"] = true; 
        cachePolicy.VaryByHeaders["Accept-Charset"] = true; 
        cachePolicy.VaryByHeaders["Accept-Encoding"] = true; 
        cachePolicy.VaryByParams["*"] = true;

        cachePolicy.SetValidUntilExpires(true);

        // output cache has dependency on the data service cache item 
        context.Response.AddCacheItemDependency(cacheDependencyItemKey); 
    }

    if (context.Cache.Get(cacheDependencyItemKey) == null) 
    { 
        // what the cache item value is doesn't really matter 
        context.Cache.Insert(cacheDependencyItemKey, "Item"); 
    } 
}

We can update the client-side application to test this:

var context = new NorthwindEntities(new Uri("https://localhost:85/NorthwindService.svc"));

var cust = context.CreateQuery("Customers").Take(1).FirstOrDefault(); 
context.UpdateObject(cust); 
context.SaveChanges();

int callCount = context.Execute(new Uri("https://localhost:85/NorthwindService.svc/GetProcessedCount")).Single();

Console.WriteLine("Initial call count {0}", callCount);

IQueryable[] queries = new IQueryable[] { 
    context.CreateQuery("Customers").Skip(10).Take(10), 
    context.CreateQuery("Customers") , 
    context.CreateQuery("Orders").Skip(10).Take(10) , 
    context.CreateQuery("Orders") , 
    context.CreateQuery("Customers").Expand("Orders"), 
};

foreach (var q in queries) ((DataServiceQuery)q).Execute();

callCount = context.Execute(new Uri("https://localhost:85/NorthwindService.svc/GetProcessedCount")).Single();

Console.WriteLine("Call count after query {0}", callCount);

foreach (var q in queries) ((DataServiceQuery)q).Execute(); 
foreach (var q in queries) ((DataServiceQuery)q).Execute(); 
foreach (var q in queries) ((DataServiceQuery)q).Execute();

callCount = context.Execute(new Uri("https://localhost:85/NorthwindService.svc/GetProcessedCount")).Single(); 
Console.WriteLine("Call count after batch querying {0}", callCount);

context.UpdateObject(cust); 
context.SaveChanges();

foreach (var q in queries) ((DataServiceQuery)q).Execute(); 
foreach (var q in queries) ((DataServiceQuery)q).Execute(); 
foreach (var q in queries) ((DataServiceQuery)q).Execute();

callCount = context.Execute(new Uri("https://localhost:85/NorthwindService.svc/GetProcessedCount")).Single(); 
Console.WriteLine("Call count after update and querying {0}", callCount);

And the output is:

Initial call count 3 <—1 update, 1 get, 1 service op
Call count after query 6
Call count after batch querying 1
Call count after update and querying 7

As you can see, requests will always go through data services after an update has occurred. In this example we grouped all items into one big group, that’s probably not the optimal solution for most people. An entity-set based grouping should be sufficient, and you can implement this with Query and Change interceptors.

Total Cache Control: The OutputCacheProvider

Still not satisfied with the level of control you have? You can choose to implement the abstract class OutputCacheProvider, and hook it up via a simple config file change. Note that if you go this route, then you cannot add output page dependencies to data cache items. You’ll have to implement the dependency logic inside your provider. I won’t go into details here but the MSDN documentation and this article should help if you decided to go this route.

%,&,*,:,<,>,+,#, /, ?,\

If any of the above characters are used inside a string key for an entity, then querying for the entity will resulting in an error, whether you escape the Uri or not.

The reason for these failures is actually many-fold. The main concern here is, of course, security. Allowing these characters can potentially lead to URI injection attacks and other security holes that leave your service vulnerable. Hence, before going forward and start allowing restricted characters, you should evaluate the security risks involved - these characters are restricted for good reasons.

The very first thing you can do is turning of ASP.net request filtering. You can do that by adding the following section in web.config:

<httpRuntime requestPathInvalidCharacters="" requestValidationMode="2.0"/> 
<pages validateRequest="false"/>

This will allow the first six characters (%,&,*,:,<,>) to be used in the path of request URI. If you just want to allow one of the six, you can add the rest to “requestPathInvalidCharacters”.

The next character is ‘+’, which when used inside a key will cause the server to throw 404 bad request. The reason for this is the IIS security filter rejecting double-escaped URL. You can allow this by adding the following into the web.config file:

<system.webServer> 
  <security> 
    <requestFiltering allowDoubleEscaping="true" /> 
  </security> 
</system.webServer>

Now the next character is a bit tricky. ‘#’ inside a string literal will cause WCF to truncate the Uri from that character and on (actually this may be a problem inside System.Uri), leaving a broken link to the server. You may think escaping the sequence will get you out of the problem – unfortunately the underlying host (either IIS or Asp.net) unescapes the Uri. Without a custom host, data service is normally two layers above ASP.net, with WCF host in between. Thus, WCF will see the unescaped URI passed out from ASP.net, and when it passes this URI out again to the service layer, it’s already been truncated. Luckily, you can still retrieve the original request uri through ASP.net’s HttpContext.Current.Request.Url. So to get around this problem, we are going to have to manufacture our own properly escaped URIs. To pass the homemade Uri to our server, we can use the IncomingMessageProperties discussed here. Basically, we are bypassing WCF host to retrieve the request URI.

The following code is a simple proof of concept parser, and it’s a very hacky one too. It will not work in all scenarios and it’s not meant to be used in production services. More importantly, it may totally screw up your server and cause severe damage to your data. So if you must go ahead with this workaround, then you should spend some time and write a more comprehensive parser. I should also note that this is a problem with the WCF host sitting below the data service layer. If you are using a custom host then you probably won’t see this problem, skip this workaround and go directly to the Uri configurations.

public WcfDataService1() 
{ 
    string uri = HttpContext.Current.Request.Url.OriginalString; 
    StringBuilder replaceUri = new StringBuilder();

    bool inquote = false; 
    for (int i = 0; i < uri.Length; ++i) 
    { 
        switch (uri[i]) 
        { 
            case '\'': 
                replaceUri.Append(uri[i]); 
                inquote = !inquote; 
                break; 
            case '#': 
            case '\\': 
            case '/': 
            case '?': 
                if (inquote) 
                { 
                    replaceUri.AppendFormat("%{0:X}", (int)uri[i]); 
                } 
                else 
                { 
                    replaceUri.Append(uri[i]); 
                } 

                break; 
            default: 
                replaceUri.Append(uri[i]); 
                break; 
        } 
    }

    OperationContext.Current.IncomingMessageProperties["MicrosoftDataServicesRequestUri"] = new Uri(replaceUri.ToString()); 
}

You’d noticed that this will also take care of the rest of the characters (\,/ and ?). Although they are caused by a different but similar problem – the underlying host unescapes URI and screws up the parser. However, to get the final three characters working, this is not enough. You also have to tell the Uri parser to allow slashes and question marks. This can be done by adding the following to the very top of the web.config file, just under the <configuration> session:

<configSections> 
    <section name="uri" type="System.Configuration.UriSection, System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089"/> 
</configSections> 
<uri> 
    <schemeSettings> 
        <add name="http" genericUriParserOptions="DontUnescapePathDotsAndSlashes"/> 
        <add name="https" genericUriParserOptions="DontUnescapePathDotsAndSlashes"/> 
    </schemeSettings> 
</uri>

And there you have it! You can now have any characters inside those string IDs and the server won’t choke on it. You’d also be happy to know that your service is now extremely vulnerable to URI attacks. So, after evaluating the security risks and if you decide to allow only a subset of these characters, you only have to enable the settings targeted specifically for them.

OperationContext.Current.IncomingMessageProperties["MicrosoftDataServicesRootUri"] = serviceUri;

You need to do this per instance of the data service, so it makes a lot of sense to do it in the service class’ constructor. Normally, WCF data service will try to resolve the absolute service URI by asking the underlying host. However, when there is a “MicrosoftDataServicesRootUri” key present, then the property will take precedence and become the service root URI. By default, the service root uri is used to generate identities and links in the content (such as edit links).

Doing just this, however, is not quite enough. Some of the links are generated by looking at the request uri, rather than the service root uri. For example, server driven paging’s next page link is constructed this way. Therefore, you also need to override request uri (and you need to do it per request, obviously). You do this in the “MicrosoftDataServicesRequestUri” property:

OperationContext.Current.IncomingMessageProperties["MicrosoftDataServicesRequestUri"] = requestUri;

As a real world example, let’s take a look at the OData demo service located at https://services.odata.org/(S(readwrite))/OData/OData.svc/. This service utilizes ASP.net cookieless session to allow multiple users reading/writing to the same structural data. If you follow this link, you’ll notice that the “readwrite” token inside the session ID will get replaced by an actual session ID, and from that point on all of the service content will be referred with this session key inside. For example:

<id>https://services.odata.org/(S(v43ws2r0y00yx42zc220o2ci))/OData/OData.svc/Products(0)</id>

<title type="text">Bread</title>

<summary type="text">Whole grain bread</summary>

By default, ASP.net will strip the session part of the URI out, leaving the service to think the host root is actually unsessioned. Therefore we need to utilize the above technique and “reconstruct” the sessioned uri (based on session IDs). We can construct the service root once per session, but the request URI must be reconstructed per request. Therefore, we ended up with the following code:

if (WebOperationContext.Current.IncomingRequest.UriTemplateMatch != null)
{
    Uri serviceUri = HttpContext.Current.Session["ServiceUri"] as Uri;
    Uri requestUri = null;

    UriTemplateMatch match = WebOperationContext.Current.IncomingRequest.UriTemplateMatch;

    if (serviceUri == null)
    {
        // create a service Uri that represent the current session
        UriBuilder serviceUriBuilder = new UriBuilder(match.BaseUri);
        serviceUriBuilder.Path = "(S(" + HttpContext.Current.Session.SessionID + "))" + serviceUriBuilder.Path;

        serviceUri = serviceUriBuilder.Uri;
        HttpContext.Current.Session["ServiceUri"] = serviceUri;
    }

    if (serviceUri != null)
    {
        OperationContext.Current.IncomingMessageProperties["MicrosoftDataServicesRootUri"] = serviceUri;

        UriBuilder requestUriBuilder = new UriBuilder(match.RequestUri);
        requestUriBuilder.Path = "(S(" + HttpContext.Current.Session.SessionID + "))" + requestUriBuilder.Path;
        // because we have overwritten the Root URI, we need to make sure the request URI shares the same host
        // (sometimes we have request URI resolving to a different host, if there are firewall re-directs
        requestUriBuilder.Host = serviceUri.Host;

        requestUri = requestUriBuilder.Uri;
        OperationContext.Current.IncomingMessageProperties["MicrosoftDataServicesRequestUri"] = requestUri;
    }
}

NOTE: the above source code is from the OData service sample code library, located at https://www.odata.org/developers/odata-sdk and under the MPL license.

Don’t let this exciting opportunity just slip away!

{ "d" : [ {entity1}, {entity2} ] }

Where “d” is the “data” wrapper, used as a security measure, and it is defined as the array of elements in the entity set. For example, if you hit a V1 service today, you will see the following response:

{
  "d" : [ {
    "__metadata": {
      "uri": "$identity",
      "etag": "$etag",
      "type": "$type"
    },
    "NavigationProperty": {
      "__deferred": {
        "uri": "$link"
      }
    },
    "Property" : Value
  },
  ...
  ]
}

This format has to be changed in V2 because we now have some additional attributes that are entity-set specific. Specifically, row count and server driven paging skip tokens. We originally thought that we can stuff then inside the data array as the first element and the last element of the array, as this would prevent a change of format. If you have followed our development closely, you would have seen this behavior in the V1.5 CTP1 release. We soon realized that having a mixed type array that sometimes gives the user additional information offers very poor usability. So we finally decided to “upgrade” our JSON format to 2.0, which will let us put the entity set level information where they logically belong to – well, on the entity set.

The new format changes the “d” wrapper from the actual entity array to a new object with multiple properties. The data now lives under “results”, the entity set level information will each become a property in that same object:

{
  "d" : {
    "results": [ {entity1}, {entity2}, ... ],
    "__count": "$count"
    "__next" : "$next-link"
  }
}

The serialization format for the individual entities stays the same.

To maintain backwards compatibility, the server will only output in the new format if a V2 specified feature had been requested. That is, when the server has to use a DSV=2.0 response (see my post on versioning). For the “normal” V1 requests, the old format will be used.

This does cause some pain when you try to support multiple versions of requests. You now must read the data service version header from the response, and decide which JSON parser you use to retrieve the data.

With the release of CTP2, we have introduced a couple of new features on the server. In the table below I'll list these features and their behavior when it comes to cross-version calls. In the table, minimum requesting DataServiceVersion means the client must send a DSV header that's greater than or equal to this value, and the response DSV means the server will response with this version (which also means the MDSV header sent by the client must be less than or equal to this value).

The rule of thumb here is we require the requesting DSV to be 2.0 for all new URI constructs ($inlinecount, $count, $skipToken, $select), and the response DSV will be 2.0 if the content is incompatible with V1 response. For example, for Server Driven Paging, when the server truncates the result set, we will bump the response DSV to 2.0, because V1.0 client will ignore the continuation token and thus unable to receive the full results.

The Max Protocol Version configuration

For folks out there wanting to use V2 service and want their service to be V1 protocol compliant, you can "turn off" the features that may require a DSV of 2.0 by setting the MaxProtocolVersion configuration*. This setting is available only during service initialization:

public static void InitializeService(DataServiceConfiguration config)
{
    config.SetEntitySetAccessRule("*", EntitySetRights.All);
    config.DataServiceBehavior.MaxProtocolVersion = DataServiceProtocolVersion.V1;
}  

You can think of this as the service-side enforced "MaxDataServiceVersion" header. When this is set, any request that requires a DSV greater than 1.0 will fail even if the proper headers are present. The error message you get in this case is "The response requires that version 2 of the protocol be used, but the MaxProtocolVersion of the data service is set to DataServiceProtocolVersion.V1", and the status code of the reply is BadRequest(400).

*NOTE: We have decided to ship V2 with MaxProtocolVersion default to V1, for safety and compatibility reasons. You MUST explicitly set this to V2 in order to use any of the V2 features.

ReadOnlyCollection<EntityDescriptor> Entities

where EntityDescriptor is a class that gives you access to the entity instance, the ETag and the state of the entity (changed, deleted, etc). The easiest way to query local copies of entities is to use LINQ to Objects. For example, if you are looking for a Customer who’s CustomerID is 0, and you want to see if it exists locally first, then you can write:

var q = from ed in Context.Entities
        where ed.Entity is Customer 
              && ((Customer)ed.Entity).ID == 0
        select (Customer)ed.Entity;

Customer cust = q.FirstOrDefault();

In the above example, when cust is not null, you can work with the copy and still have the context track the changes you make to the entity. If cust is null, you can then query the remote service and the next time you run this again, a local copy will exist.

The key to this is, of course, is to have a singleton DataServiceContext for all of your data access needs. In a sense, you are keeping a local "data store” and when possible, use it’s data instead of the remote one. Ultimately, you would want to have it retrieve remote data on demand.

A simple data store may look like this:

public class DataStore
{
    static DataServiceContext ctx = new DataServiceContext(new Uri(“ServiceUri”, UriKind.Absolute));

    public static DataServiceContext Context
    {
        get { return ctx; }
    }

    public static IQueryable<T> QueryLocal<T>()
    {
        return (from ed in ctx.Entities.AsQueryable()
                where ed.Entity is T
                select (T)ed.Entity);
    }

    public static IQueryable<T> QueryLocal<T>(Expression<Func<T,bool>> predicate)
    {
        return (from ed in ctx.Entities.AsQueryable()
                where ed.Entity is T
                select (T)ed.Entity).Where(predicate);
    }
}

With this setup, you can query the remote store with DataStore.Context.CreateQuery<T>(), and you can query the local store with DataStore.QueryLocal<T>(). The above Customer example simply become:

Customer cust = DataStore.QueryLocal<Customer>(c=>c.ID==0).FirstOrDefault();

Taking it one step further, you can imagine having a unified “QueryStore<T>()” function that queries local data first, and if it didn’t find anything, it would go ahead and ask the service for data. This approach can significantly reduce the amount of data you need to move across the wire for your service.

Of course, just like everything else in life, this approach also has trade-offs. The biggest one comes from the fact that the local copies are not “partitioned” or “indexed” in anyway. This means even if all you are looking for is one customer, you may have to loop through thousands of other irrelevant entities. In situation like this, you must make a decision to either minimize the bandwidth or maximize the performance – it could be faster if you just query the remote service instead. In short, whether having a local data store suites your needs depends on the structure of your data. Applied currently, it can significantly optimize your data service experience.