<link rel="alternate" type="application/rss+xml" title="Pablo Castro&#39;s blog (RSS 2.0)" href="http://blogs.msdn.com/b/pablo/rss.aspx" />

It would be great if all OData clients could automatically discover the location of the data feed that has the data represented by the current web page, or more in general, by the document fetched through some arbitrary URL. We had this discussion with Scott some time ago and he rolled the results in the NerdDinner.com site, but I failed to post this so it stays documented somewhere and others can follow, so here it goes.

Servers can advertise the OData endpoints that correspond to a resource using two mechanisms. Ideally servers would implement both, but sometimes limitations in the hosting environment and such may make it hard to support the header-based approach.

Using <link> elements

The first mechanism consists of adding one or more <link> headers to any HTML web page that shows data where that data can also be accessed as an OData feed using some other URL. There are two kinds of links for OData, those that direct clients to the service root (usually where the service document lives)and those that point at the specific dataset that corresponds to the data displayed in the page (e.g. point to a specific collection and includes whatever filters and sort order is used for display). Taking the example from the home page of NerdDinner.com:

<link rel="odata.service" title="NerdDinner.com OData Service" href="/Services/OData.svc" />
<link rel="odata.feed" title="NerdDinner.com OData Service - Dinners" href="/Services/OData.svc/Dinners" />

In this example the relation "odata.service" indicates that the href has a URL pointing to the root of a service and the relation "odata.feed" specifies the URL to the specific dataset for the page (dinners in this case). Note that it's perfectly fine to have multiple of each, although I would expect that the common thing to do would be go have one "odata.service" and zero or more "odata.feed", depending on the specific page you're looking at.

If your web page has the typical syndication icon to indicate that it has feeds in it, and you'd like to indicate visually that it also has links to OData feeds you can use the OData icon like NerdDinner.com does (next to the syndication icon on the right in the screenshot):

Using links in response headers

There is a proposal currently in-flight to standardize how servers can send links to related resources of a given resource using response headers instead of including them as part of the content of the response body. This is great because it means you don't have to know how to parse the content-type of the body (e.g. HTML) in order to obtain a link. In our context, your OData client wouldn't need to do any HTML parsing to obtain the OData feed related to a web page or some other random resource obtained through a user-provided URL.

The "Web Linking" proposal is described in the current draft in the IETF web site:

http://datatracker.ietf.org/doc/draft-nottingham-http-link-header/

As for the specifics for OData we would define two relations, "http://odata.org/link/service" and "http://odata.org/link/feed", corresponding to the "odata.service" and "odata.feed" relations used in the <link> element above. So for NerdDinner.com these would look like this in the response header section:

Link: </Services/OData.svc/>; rel="http://odata.org/link/service"; title="NerdDinner.com OData Service"
Link: </Services/OData.svc/Dinners/>; rel="http://odata.org/link/feed"; title="NerdDinner.com OData Service - Dinners"

For all folks hosting OData services, please consider adding one or if possible both of these to advertise your services. For anybody writing a client, this is the best way to discover services given an arbitrary URL that may point to a web page or something like that, and you can still make it nicely redirect to the right location and things will "just work" for your users.

-pablo

Back when we were working on the initial version of OData (Astoria back then) there were some discussion about introducing a new HTTP method called PATCH, which would be similar to PUT but instead of having to send a whole new version of the resource you wanted to update, it would let you send a patch document with incremental changes only. We needed exactly that, but since the proposal was still in flux we decided to use a different word (to avoid any future name clash) until PATCH was ready. So OData services use a custom method called "MERGE" for now.

Now HTTP PATCH is official (see http://tools.ietf.org/html/rfc5789). We'll queue it up and get it integrated into OData and our .NET implementation WCF Data Services when we have an opportunity. Of course those things take a while to rev, and we have to find a spot in the queue, etc. In the meanwhile, here is an extremely small WCF behavior adds PATCH support to any service created with WCF Data Service. This example is built using .NET 4.0, but it should work in 3.5 SP1 as well.

http://code.msdn.microsoft.com/DataServicesPatch

To use it, just include the sample code in PatchBehavior.cs in your project and add [PatchSupportBehavior] as an attribute to your service class. For example:

    [PatchSupportBehavior]
    public class Northwind : DataService<NorthwindEntities>
    {
        // your data service code
    }

To see it working you can use any environment with an HTTP stack. Below is an example using curl to update a single attribute (the CategoryName value) using PATCH. The first try shows what happens when you try PATCH against a service that doesn't have PATCH enabled:

curl -v -X PATCH -H content-type:application/json -d "{CategoryName:'Beverages'}" http://localhost:29050/Northwind.svc/Categories(1)
* About to connect() to localhost port 29050 (#0)
*   Trying 127.0.0.1... connected
* Connected to localhost (127.0.0.1) port 29050 (#0)
> PATCH /Northwind.svc/Categories(1) HTTP/1.1
> User-Agent: curl/7.19.3 (i386-pc-win32) libcurl/7.19.3 OpenSSL/0.9.8j zlib/1.2.3
> Host: localhost:29050
> Accept: */*
> content-type:application/json
> Content-Length: 26
>
< HTTP/1.1 501 Not Implemented
< Date: Mon, 14 Jun 2010 22:36:49 GMT
< DataServiceVersion: 1.0;
< Content-Length: 217
< Cache-Control: private
< Content-Type: application/xml
< Connection: Close
<
<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<error xmlns="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata">
  <code></code>
  <message xml:lang="en-US">Not Implemented</message>
</error>
* Closing connection #0

Once the PATCH support attribute is applied, things look better:

curl -v -X PATCH -H content-type:application/json -d "{CategoryName:'Beverages'}" http://localhost:29050/Northwind.svc/Categories(1)
* About to connect() to localhost port 29050 (#0)
*   Trying 127.0.0.1... connected
* Connected to localhost (127.0.0.1) port 29050 (#0)
> PATCH /Northwind.svc/Categories(1) HTTP/1.1
> User-Agent: curl/7.19.3 (i386-pc-win32) libcurl/7.19.3 OpenSSL/0.9.8j zlib/1.2.3
> Host: localhost:29050
> Accept: */*
> content-type:application/json
> Content-Length: 26
>
< HTTP/1.1 204 No Content
< Date: Mon, 14 Jun 2010 22:34:42 GMT
< DataServiceVersion: 1.0;
< Cache-Control: no-cache
< Content-Length: 0
< Connection: Close
<
* Closing connection #0

As for how it works, the behavior catches all requests before they go to the WCF Data Services runtime. For each request it inspects the HTTP method used and if it is PATCH then it changes it to MERGE to make the system believe it was MERGE all along.

Usual disclaimers:
- I tested this by running it exactly once. You may want to be more detailed on your quality process :)
- Once we do the real thing the system may or may not behave as this one, so you need to be prepared to deal with some potential differences between the two.

-pablo

The interview is now live.

This was a good warm-up exercise for next week. We'll have a lot to say about OData at Mix 2010 in Las Vegas. If you are planning on attending Mix you won't want to miss the keynotes and OData sessions that are happening on Tuesday. We'll also be hanging out in the common areas during the event so if you're into this topic and would like to chat definitely come and find us.

-pablo

OData is designed to be modular and grow as you need more features. We don't want to dictate exactly everything a service needs to do. Instead we want to make sure that if you choose to do something, you do it in a well-known way so everybody else can rely on that.

Here is one possible way of slicing the capabilities of the protocol into various click-stops. I'm sure there are others as well.

Rows

You can use OData just as a way of representing a bunch of rows, typically available on an HTTP end point and retrievable using HTTP GET (although a file on disk will do just as well if you let go of content type negotiation and such).  

A set of rows in OData is represented using an Atom feed, where each entry has the usual metadata plus structured data encoded as name/value pairs inside content. For example, check out the content of this feed (if you're using a browser that automatically renders Atom feeds in a nice view, you may want to turn that off):

http://ogdi.cloudapp.net/v1/dc/BankLocations

A couple of incremental things you can add:

- Type annotations to individual properties to have full type fidelity (visible in the example above)

- You can use Atom-style paging (e.g. a link with a rel of "next") for server-driven paging of large rowsets

What do you get? A server supporting this has enough, for example, for PowerPivot (http://powerpivot.com) to load data from it. The Data Services clients will also work if you hand-craft a type that matches the shape of the data (or the subset of the shape you care about).

Query

You can choose to support simple query options such as filter ($filter), sorting ($orderby), paging ($top, $skip), projections ($select) and inline expansion ($expand). For example:

http://ogdi.cloudapp.net/v1/dc/BankLocations?$filter=zipcode eq 20007

(this service endpoint only supports filter and paging, no sorting and no projections for now)

The actual definitions of all these options is documented in http://odata.org if you want to look at it in more detail.

What do you get? Developers using the Data Services client for .NET would be able to use LINQ against your service, at least for the operators that map to the query options you implemented.

Service Documents

You can choose to add a service document to collect links to multiple collections that make up a data set, for example:

http://ogdi.cloudapp.net/v1/dc/

What do you get? With this clients can discover what collections of data your service exposes. Some clients such as PowerPivot will use this to let the user browse data sets.

Metadata

You can choose to add rich metadata describing the collections, shapes of resources in each collections, and their associations through the Entity Data Model. For example (not the best example as it does not contain associations):

http://ogdi.cloudapp.net/v1/dc/$metadata

What do you get? If you have metadata then Visual Studio can automatically generate .NET classes for your service types, and a service entry point with properties for each of your top-level sets. The OData visualizer also can allow developers to view and understand the structure of your data service.

Update

You can choose to support updates through HTTP POST/PUT/DELETE verbs. Most services that allow updates don't allow anonymous access so I can't easily share an example, but it would be POSTing or PUTing the same payload you GET. For DELETE only the URL is needed. For all update operations you can use ETags for optimistic concurrency control if that's required for your scenarios.

What you get? Clients that support updates such as the .NET, AJAX, Silverlight, PHP and Java ones would be able to submit changes to the server.

Formats

While we encourage folks to support for Atom and JSON formats, you can get pretty far with Atom alone.

Batching

More sophisticated features in OData include batching and changesets for round-trip optimization and unit-of-work support.

You can check out the official description of the position here:
https://careers.microsoft.com/JobDetails.aspx?ss=&pg=0&so=&rw=1&jid=13620&jlang=EN

-pablo

I've always been attracted to things that bring data and the Web together. So a while back when I first saw browsers and databases in the same spec I had to get involved. There is a bunch of us at Microsoft interested in HTML5 from different angles, and we have now good momentum to explore this space. So now I'm spending a good chunk of time on the database aspects of HTML5, the API, the developer story, etc.

(btw - no, I haven't given up on Astoria or OData, I'm still plenty busy with that...but hey, how bad could be it to add a whole subject area to the work schedule :) )

Current state of things

There are currently two proposals for databases in the browser: Web SQL Database and Indexed Database API.

Web SQL Database was the first to appear. It consists of a relatively small Javascript API that allows developers to execute SQL statements. You do most things through SQL, including schema management, querying, updates, etc. While the spec doesn't (or didn't for a while) specifically say anything about the actual dialect of SQL to be used, in practice early implementations used SQLite, and thus directly exposed the SQLite SQL dialect and other details specific to it.

The second and more recent proposal is now called Indexed Database API (was WebSimpleDB) and it exposes an API in terms of ordered tables of Javascript objects. You get or put Javascript objects and use a key to identify and order them. For database people, this is basically an ISAM API with Javascript objects as the record format. You can create indexes to speed up lookups or scans in particular orders. Other than that there is no schema (any clonable Javascript object will do) and no query language.

The "right" level of abstraction

Of course there are "sides" for this debate about the right database API. I'll just be upfront and take mine: I like the indexed database API better. I have two main motivations for this:

Interoperability: this thing is going to be part of HTML and has to be implementable by multiple browser vendors and still be fully interoperable. Implementing multiple SQL databases and making them fully interoperable is extremely challenging. You would have to line up not only the SQL syntax, but also catalog names, type semantics, execution behavior and perhaps even isolation model and optimization strategies if you wanted really, really similar behavior across browsers. Of course most of this is already described in specifications such as SQL-92 (or SQL:1999, or any of the newer ones); however, databases often don't follow all of it for various (good or not) reasons. The ISAM-style API imposes substantially simpler requirements on the underlying implementation, making it a good candidate for independent, interoperable implementations.

Diversity: if you look at what folks out there have done with Javascript toolkits, it's pretty amazing. From jQuery to Dojo, they've created different abstractions over the base browser APIs, providing diversity of choice for developers writing Web applications. A simple and low level API enables this. Folks can write libraries that provide whatever abstraction they see fit. This includes anything from simple shims to full query support, in whatever language you think it's right to do it in the browser. With a low level API that's consistent across all browsers, and all the fancy abstractions built on top in Javascript, the goal of portability becomes clearly reachable.

The guiding principle can be summarized as: build into the standard only the things that you cannot build on top (there are a few exceptions for very common idioms, of course; those are, well...exceptions).

Folks from Mozilla share this perspective, which is encouraging. Nikunj, the author of the indexed database API, is from Oracle, and supports this sort of by definition, given that he's the editor of the spec. You can see more discussion about this in the webapps working group list archive, including this post where I outlined Microsoft's take.

We're exploring

We are working on understanding the API, providing feedback to the W3C WebApps working group, and creating experimental implementations to explore the space, try it out in applications and discover good and bad things about it.

-pablo

* NOTE: I refer here to HTML5 kind of loosely to refer to the "next round of HTML technology". There are actually several specs involved in addition to the core HTML5 spec itself

SharePoint is a repository of resources (list items and documents in document libraries) that are collected and manipulated collaboratively. Resources have a bunch of security and business logic attached to them, such as who can see each item, who gets to change it, or whether a particular column in a list needs to conform to a particular validation formula.

When SharePoint folks said they wanted a RESTful interface this was great news...the system is just a perfect fit. Not only it's a perfect fit for RESTful services in general, but also for Astoria in particular. In the end SharePoint is very data-centric in nature, it already supports queries and business logic as part of the uniform interface.

So we're really excited to announce that as of SharePoint 2010, every SharePoint server is an Astoria server out of the box. No configuration required or anything, just make sure the proper version of ADO.NET Data Services is in the box. For SharePoint 2010 beta, the "right" version is ADO.NET Data Services v1.5 CTP2. We'll put details out there for future iterations as they come.

Official write up in the Astoria team blog.

The SharePoint Data Service head is not just a side integration deal, it's a full-on REST-over-HTTP head for SharePoint. It supports browsing as well as modifying data using regular HTTP verbs (GET, PUT, DELETE, etc.), it does ETags for concurrency control, enforces business logic as part of side-effecting methods, and it handles the full range of Data Services conventions for URLs, Atom and JSON payload formats, etc. It also exposes full metadata like any other Astoria service, allowing Visual Studio and any other metadata-driven client to give you a great experience on the client side. Now if you need to get or manipulate data in SharePoint from any platform in any language, all you need is an HTTP stack.

The other cool aspect is that the SharePoint folks where able to build this entirely on top of public bits, using our new fancy provider model for advanced data sources I discussed here some time ago.

SharePoint joins a growing family of Microsoft products that enable users to share data through the use of a simple RESTful interface that follows the Astoria conventions when needed (e.g. URLs, structured data in Atom). Another example on the server side include SQL Server Reporting Services, which in SQL Server 2008 R2 can now render any report as Atom feed that follows the Astoria conventions (examples here). On the client, in addition to developer-oriented options such as .NET, Silverlight, AJAX, PHP, Java and more, we now also have PowerPivot (f.k.a. Gemini) which can pull data from any data service and do fancy analysis and publishing over it, making it trivial to bring data from Data Services into Microsoft Excel 2010 among other things.

I find this to be extremely important. A simple, uniform way of sharing data at the protocol level, and in a way that truly enables the lowest possible bar of entry, is key to enable broad integration across products and break data silos that form around applications.

I'll be talking about this at the SharePoint conference 2009 in Las Vegas this week, and at the Professional Developer Conference (PDC 2009) in late November. If you're attending any of these and this sounds interesting, these sessions will drill into plenty of details.

-pablo

You can check out the announcement in the Interoperability's Team blog.

-pablo

Official announcement is here, and includes details on where to download from, what to watch for (it's still a CTP, comes with fine-print :) ), and a list of the new and updated features.

Also, many folks in the Astoria Team have been writing a bunch of content to cover the new features, so watch the Astoria Team blog as well as related blogs for all sorts of samples and details to come up.

-pablo

The best part is that it's all written in PHP (both code-gen tool and runtime), so it runs in any environment where PHP and the toolkit dependencies (XML, XSLT, CURL) are available.

For more details check out the post in the interoperability team's blog, the kick off Channel 9 video, and the PHP toolkit project in codeplex.

-pablo