Streaming Provider (WCF Data Services)

Article
09/02/2011

A data service can expose large object binary data. This binary data might represent video and audio streams, images, document files, or other types of binary media. When an entity in the data model includes one or more binary properties, the data service returns this binary data encoded as base-64 inside the entry in the response feed. Because loading and serializing large binary data in this manner can affect performance, the Open Data Protocol (OData) defines a mechanism for retrieving binary data independent of the entity to which it belongs. This is accomplished by separating the binary data from the entity into one or more data streams.

Media resource - binary data that belongs to an entity, such as a video, audio, image or other type of media resource stream.
Media link entry - an entity that has a reference to a related media resource stream.

With WCF Data Services, you define a binary resource stream by implementing a streaming data provider. The streaming provider implementation supplies the data service with the media resource stream associated with a specific entity as an Stream object. This implementation enables the data service to accept and return media resources over HTTP as binary data streams of a specified MIME type.

Configuring a data service to support the streaming of binary data requires the following steps:

Attribute one or more entities in the data model as a media link entry. These entities should not include the binary data to be streamed. Any binary properties of an entity are always returned in the entry as base-64 encoded binary.
Implement the T:System.Data.Services.Providers.IDataServiceStreamProvider interface.
Define a data service that implements the IServiceProvider interface. The data service uses the GetService implementation to access the streaming data provider implementation. This method returns the appropriate streaming provider implementation.
Enable large message streams in the Web application configuration.
Enable access to binary resources on the server or in a data source.

The examples in this topic are based on a sample streaming photo service, which is discussed in depth in the post Data Services Streaming Provider Series: Implementing a Streaming Provider (Part 1). The source code for this sample service is available on the Streaming Photo Data Service Sample page on MSDN Code Gallery.

Defining a Media Link Entry in the Data Model

The data source provider determines the way that an entity is defined as a media link entry in the data model.

Entity Framework Provider
To indicate that an entity is a media link entry, add the HasStream attribute to the entity type definition in the conceptual model, as in the following example:

<EntityType xmlns:m="https://schemas.microsoft.com/ado/2007/08/dataservices/metadata"
            Name="PhotoInfo" m:HasStream="true">
  <Key>
    <PropertyRef Name="PhotoId" />
  </Key>
  <Property Name="PhotoId" Type="Int32" Nullable="false" 
            annotation:StoreGeneratedPattern="Identity" />
  <Property Name="FileName" Type="String" Nullable="false" />
  <Property Name="FileSize" Type="Int32" Nullable="true" />
  <Property Name="DateTaken" Type="DateTime" Nullable="true" />
  <Property Name="TakenBy" Type="String" Nullable="true" />
  <Property Name="DateAdded" Type="DateTime" Nullable="false" />
  <Property Name="Exposure" Type="PhotoData.Exposure" Nullable="false" />
  <Property Name="Dimensions" Type="PhotoData.Dimensions" Nullable="false" />
  <Property Name="DateModified" Type="DateTime" Nullable="false" />
  <Property Name="Comments" Type="String" MaxLength="Max" 
            FixedLength="false" Unicode="true" />
  <Property Name="ContentType" Type="String" MaxLength="50" FixedLength="false" Unicode="true" />
</EntityType>

You must also add the namespace xmlns:m=https://schemas.microsoft.com/ado/2007/08/dataservices/metadata either to the entity or to the root of the .edmx or .csdl file that defines the data model.

For an example of a data service that uses the Entity Framework provider and exposes a media resource, see the post Data Services Streaming Provider Series: Implementing a Streaming Provider (Part 1).

Reflection Provider
To indicate that an entity is a media link entry, add the HasStreamAttribute to the class that defines the entity type in the reflection provider.

Custom Data Service Provider
When using custom service providers, you implement the IDataServiceMetadataProvider interface to define the metadata for your data service. For more information, see Custom Data Service Providers (WCF Data Services). You indicate that a binary resource stream belongs to a ResourceType by setting the IsMediaLinkEntry property to true on the ResourceType that represents the entity type, which is a media link entry.

Implementing the IDataServiceStreamProvider Interface

To create a data service that supports binary data streams, you must implement the IDataServiceStreamProvider interface. This implementation enables the data service to return binary data as a stream to the client and consume binary data as a stream sent from the client. The data service creates an instance of this interface whenever it needs to access binary data as a stream. The IDataServiceStreamProvider interface specifies the following members:

Member name	Description
DeleteStream	This method is invoked by the data service to delete the corresponding media resource when its media link entry is deleted. When you implement IDataServiceStreamProvider, this method contains the code that deletes the media resource associated with the supplied media link entry.
GetReadStream	This method is invoked by the data service to return a media resource as a stream. When you implement IDataServiceStreamProvider, this method contains the code that provides a stream that is used by the data service to the return media resource that is associated with the provided media link entry.
GetReadStreamUri	This method is invoked by the data service to return the URI that is used to request the media resource for the media link entry. This value is used to create the src attribute in the content element of the media link entry and that is used to request the data stream. When this method returns null, the data service automatically determines the URI. Use this method when you need to provide clients with direct access to binary data without using the steam provider.
GetStreamContentType	This method is invoked by the data service to return the Content-Type value of the media resource that is associated with the specified media link entry.
GetStreamETag	This method is invoked by the data service to return the eTag of the data stream that is associated with the specified entity. This method is used when you manage concurrency for the binary data. When this method returns null, the data service does not track concurrency.
GetWriteStream	This method is invoked by the data service to obtain the stream that is used when receiving the stream sent from the client. When you implement IDataServiceStreamProvider, you must return a writable stream to which the data service writes received stream data.
ResolveType	Returns a namespace-qualified type name that represents the type that the data service runtime must create for the media link entry that is associated with the data stream for the media resource that is being inserted.

Creating the Streaming Data Service

To provide the WCF Data Services runtime with access to the IDataServiceStreamProvider implementation, the data service that you create must also implement the IServiceProvider interface. The following example shows how to implement the GetService method to return an instance of the PhotoServiceStreamProvider class that implements IDataServiceStreamProvider.

Partial Public Class PhotoData
    Inherits DataService(Of PhotoDataContainer)
    Implements IServiceProvider

    ' This method is called only once to initialize service-wide policies.
    Public Shared Sub InitializeService(ByVal config As DataServiceConfiguration)
        config.SetEntitySetAccessRule("PhotoInfo", _
            EntitySetRights.ReadMultiple Or _
            EntitySetRights.ReadSingle Or _
            EntitySetRights.AllWrite)

        config.DataServiceBehavior.MaxProtocolVersion = DataServiceProtocolVersion.V2
    End Sub
#Region "IServiceProvider Members"
    Public Function GetService(ByVal serviceType As Type) As Object _
    Implements IServiceProvider.GetService
        If serviceType Is GetType(IDataServiceStreamProvider) _
            Or serviceType Is GetType(IDataServiceStreamProvider) Then
            Return New PhotoServiceStreamProvider(Me.CurrentDataSource)
        End If
        Return Nothing
    End Function
#End Region
End Class

public partial class PhotoData : DataService<PhotoDataContainer>, IServiceProvider
{
    // This method is called only once to initialize service-wide policies.
    public static void InitializeService(DataServiceConfiguration config)
    {
        config.SetEntitySetAccessRule("PhotoInfo",
            EntitySetRights.ReadMultiple |
            EntitySetRights.ReadSingle |
            EntitySetRights.AllWrite);

        config.DataServiceBehavior.MaxProtocolVersion = DataServiceProtocolVersion.V2;
    }
    public object GetService(Type serviceType)
    {
        if (serviceType == typeof(IDataServiceStreamProvider))
        {
            // Return the stream provider to the data service.
            return new PhotoServiceStreamProvider(this.CurrentDataSource);
        }
        
        return null;
    }
}

For general information about how to create a data service, see Configuring the Data Service (WCF Data Services).

Enabling Large Binary Streams in the Hosting Environment

When you create a data service in an ASP.NET Web application, Windows Communication Foundation (WCF) is used to provide the HTTP protocol implementation. By default, WCF limits the size of HTTP messages to only 65K bytes. To be able to stream large binary data to and from the data service, you must also configure the Web application to enable large binary files and to use streams for transfer. To do this, add the following in the <configuration /> element of the application's Web.config file:

<system.serviceModel>
  <serviceHostingEnvironment aspNetCompatibilityEnabled="true"/>
  <services>
    <!-- The name of the service -->
    <service name="PhotoService.PhotoData">
      <!--you can leave the address blank or specify your end point URI-->
      <endpoint binding="webHttpBinding" bindingConfiguration="higherMessageSize" contract="System.Data.Services.IRequestHandler"/>
    </service>
  </services>
  <bindings>
    <webHttpBinding>
      <!-- configure the maxReceivedMessageSize value to suit the max size of 
 the request (in bytes) you want the service to receive-->
      <binding name="higherMessageSize" transferMode="Streamed" maxReceivedMessageSize="2147483647"/>
    </webHttpBinding>
  </bindings>
</system.serviceModel>

Note

You must use a System.ServiceModel.TransferMode.Streamed transfer mode to ensure that the binary data in both the request and response messages are streamed and not buffered by WCF.

For more information, see Streaming Message Transfer and Transport Quotas.

By default, Internet Information Services (IIS) also limits the size of requests to 4MB. To enable your data service to receive streams larger than 4MB when running on IIS, you must also set the maxRequestLength attribute of the httpRuntime Element in the <system.web /> configuration section, as shown in the following example:

<system.web>
  <!-- maxRequestLength (in KB): default=4000 (4MB); max size=2048MB. -->
  <httpRuntime maxRequestLength="2000000"/>
  <compilation debug="true"/>
</system.web>

Using Data Streams in a Client Application

The WCF Data Services client library enables you to both retrieve and update these exposed resources as binary streams on the client. For more information, see Working with Binary Data (WCF Data Services).

Considerations for Working with a Streaming Provider

The following are things to consider when you implement a streaming provider and when you access media resources from a data service.

MERGE requests are not supported for media resources. Use a PUT request to change the media resource of an existing entity.
A POST request cannot be used to create a new media link entry. Instead, you must issue a POST request to create a new media resource, and the data service creates a new media link entry with default values. This new entity can be updated by a subsequent MERGE or PUT request. You may also consider caching the entity and make updates in the disposer, such as setting the property value to the value of the Slug header in the POST request.
When a POST request is received, the data service calls GetWriteStream to create the media resource before it calls SaveChanges to create the media link entry.
An implementation of GetWriteStream should not return a MemoryStream object. When you use this kind of stream, memory resource issues will occur when the service receives very large data streams.
The following are things to consider when storing media resources in a database:
- A binary property that is a media resource should not be included in the data model. All properties exposed in a data model are returned in the entry in a response feed.
- To improve performance with a large binary stream, we recommend that you create a custom stream class to store binary data in the database. This class is returned by your GetWriteStream implementation and sends the binary data to the database in chunks. For a SQL Server database, we recommend that you use a FILESTREAM to stream data into the database when the binary data is larger than 1MB.
- Ensure that your database is designed to store the binary large streams that are to be received by your data service.
- When a client sends a POST request to insert a media link entry with a media resource in a single request, GetWriteStream is called to obtain the stream before the data service inserts the new entity into the database. A streaming provider implementation must be able to handle this data service behavior. Consider using a separate data table to store the binary data or store the data stream in a file until after the entity has been inserted into the database.
When you implement the DeleteStream, GetReadStream, or GetWriteStream methods, you must use the eTag and Content-Type values that are supplied as method parameters. Do not set eTag or Content-Type headers in your IDataServiceStreamProvider provider implementation.
By default, the client sends large binary streams by using a chunked HTTP Transfer-Encoding. Because the ASP.NET Development Server does not support this kind of encoding, you cannot use this Web server to host a streaming data service that must accept large binary streams. For more information on ASP.NET Development Server, see Web Servers in Visual Web Developer.

Versioning Requirements

The streaming provider has the following OData protocol versioning requirements:

The streaming provider requires that the data service support version 2.0 of the OData protocol and later versions.

For more information, see Data Service Versioning (WCF Data Services).