WCF Extensibility – Message Encoders

Article
11/08/2011

This post is part of a series about WCF extensibility points. For a list of all previous posts and planned future ones, go to the index page .

Now we’re out of the metadata realm, we’re back to the components which are executed while the service is running. This post will focus on the message encoder, which is in theory part of the channel model of WCF, although it doesn’t sit nicely in the channel stack as other “normal” channels (see the diagram in the first post about channels). The encoder is the component responsible for converting between the Message object (used throughout the WCF stack) and the actual bytes which need to be transmitted over the wire. In the majority of the scenarios, using one of the out-of-the-box encoders is enough (since they can handle XML and JSON, the most common formats), but there are cases where we need to define a new format, or even to tweak the output of one of the existing encoders, so a custom encoder is a good alternative.

Implementing a message encoder is not something as simple as implementing an interface or defining one subclass of a WCF class. Because encoders are (somewhat) part of the channel stack, in order to add a custom encoder to the WCF pipeline we need to implement three classes: a binding element, an encoder factory, and finally the encoder itself. The binding element class must derive from MessageEncodingBindingElement. Similarly to the channels, the binding element’s BuildChannelFactory<TChannel> on the client side (and BuildChannelListener<TChannel> on the server side) is invoked when the channel stack is being created. Unlike the “normal” channels, on the message encoding binding element the class doesn’t create a factory / listener at that point. Instead, the encoder class is expected to add itself to the BindingParameters property of the BindingContext parameter of the method – and later the transport will query the binding parameters (if the transport decides to do so), and ask the encoding binding element to be created at a later time.

The differences continue – while in the “normal” channel at the server side a channel listener is created by the binding element, and at the client side a channel factory is created, with an encoding binding element only knows how to create a MessageEncoderFactory. The factory then is responsible for finally creating the MessageEncoder object (in two different ways – see section about the message encoder factory class for a discussion on session-full encoders below). Finally, the MessageEncoder class implements the conversion methods between Message and bytes (again in two different ways – see the section about the message encoder class for more information on streaming below) – the transport channel is the component which will invoke the encoder methods (although, as I already pointed in the post about channels, the transport doesn’t really need to use the encoder from the binding context, it can completely ignore it if it so wishes – but all out-of-the-box WCF transports do use the encoders if they’re available).

But why would one implement a custom encoder anyway? There are actually quite a few scenarios I’ve seen in forums where a custom message encoder was the solution for the issue: enforcing a maximum sent message size (WCF has a quota for received messages, not for sent, and the user wanted to limit their bandwidth usage); supporting a larger set of XML than the supported by the default WCF XML reader / writers (which don’t support things such as DTDs, processing instructions and entity references); updating the shape of the outgoing XML to comply with third-party stacks which don’t like the way WCF produces output (for example, setting specific prefixes for some XML elements); making composite encoders (the MTOM encoder accepts both text and MTOM input, but always writes MTOM; we had an internal customer which wanted an encoder to understand both MTOM and text, but always write text); supporting a new format which suits some need (such as the GZip encoder sample). And the list goes on. Encoders are quite powerful components, and hopefully this post will give you some better understanding of them in case the need arises in the future.

Public implementations in WCF

There are a few public implementations of the abstract MessageEncodingBindingElement class in WCF, but none of the MessageEncoderFactory or the MessageEncoder (the public binding element classes return internal implementations of them).

TextMessageEncodingBindingElement: the encoding which uses “traditional” XML (i.e., angle-braces) to encode a message. The preferred encoding for interoperability (most web services stack can talk XML), but it’s come at the expense of performance - it’s the one where the messages are encoded with the larger number of bytes.
BinaryMessageEncodingBindingElement: an encoding which encodes the message in a binary version of XML. The binary format protocol is publicly defined, so in theory it’s interoperable, but in practice I haven’t heard of any stack which implements this protocol, so it’s limited to WCF-WCF communication only. On the other hand, this is the encoding which gives the best performance regarding both message size (besides the reduced XML payload, it’s also has a dictionary feature which reduces the size of well-known string values)
MtomMessageEncodingBindingElement: an encoding in the mid-point between the text and binary encodings for data with large binary (xs:base64Binary) data. It encodes the messages using MTOM (a really interoperable specification, defined by W3C, and implemented by many service stacks), and it optimizes binary data, which would normally be encoded in base64 (which increases the payload size in roughly 33%) as MIME parts which don’t have the base64 size penalty.
WebMessageEncodingBindingElement (new in 3.5): a composite encoder which is capable of converting between message objects and three types of encodings: plain old XML (POX), JSON and a raw mode (which maps a certain XML schema, described in the example in the post about message inspectors). It’s used mostly in “web” scenarios, such as serving AJAX calls and REST services.

Another encoder which is worth noting is the “Custom Text Encoder” sample, which overcomes some of the limitations of the TextMessageEncodingBindingElement – namely, support for only three character encodings (UTF-8, UTF-16LE and UTF-16BE) and limited support for the full XML vocabulary (as I mentioned before).

Class definitions

public abstract class MessageEncodingBindingElement : BindingElement
{
public abstract MessageEncoderFactory CreateMessageEncoderFactory();
public override T GetProperty<T>(BindingContext context) where T: class;
public abstract MessageVersion MessageVersion { get; set; }
// From BindingElement
public virtual IChannelFactory<TChannel> BuildChannelFactory<TChannel>(BindingContext context);
public virtual IChannelListener<TChannel> BuildChannelListener<TChannel>(BindingContext context) where TChannel: class, IChannel;
public virtual bool CanBuildChannelFactory<TChannel>(BindingContext context);
public virtual bool CanBuildChannelListener<TChannel>(BindingContext context) where TChannel: class, IChannel;
}

The encoding binding element class is large, but it’s usually a lot of boilerplate code. There is the abstract MessageVersion property, which identifies to the runtime which message versions it will accept (the WCF runtime will only give to the message encoders messages which match the version); there’s the abstract CreateMessageEncoderFactory, which returns the appropriate factory capable of creating the message encoder objects. And it can also override GetProperty<T> if it wants to return any dynamic data during the runtime, but this isn’t too common. Finally, BuildChannelFactory<TChannel> (on the client side) and BuildChannelListener<TChannel> (on the server side) need to be overridden to add the encoder to the binding context (see an example in the next section, how to add a message encoder).

public abstract class MessageEncoderFactory
{
public virtual MessageEncoder CreateSessionEncoder();
public abstract MessageEncoder Encoder { get; }
public abstract MessageVersion MessageVersion { get; }
}

The message encoder factory is usually fairly simple: once more return the message version (somehow the check needs to be done throughout the stack), and return the encoder via its Encoder property. The encoder returned by this property should be thread-safe, since there’s no guarantee that the transport won’t reuse the encoder implementation for multiple connections. Also, it’s possible that the property will be called multiple times, so the encoder implementation can be cached and a single instance returned by that property (that’s what the WCF encoders do).

There’s another type of encoders, however, which are guaranteed not to be shared by multiple connections in the transport. The session encoder, returned by the CreateSessionEncoder method, is used only in transports which can send / receive multiple messages in a single connection – in WCF, those are the TCP and named pipes transport (also called session-full transports), but not HTTP or UDP (new in 4.5), since those protocols are stateless (thus session-less) by definition.

On a session-full transport, the encoder can take advantage of this fact to optimize when multiple messages are sent over the same connection. The binary encoder does exactly that – it maintains a dynamic dictionary with strings which have been sent on the same connection, so that in subsequent messages those strings can be encoded very efficiently (usually in only 1-3 bytes regardless of the string size). I’ve posted before about the dynamic dictionary in this blog, so I won’t go into many details here. If the encoder doesn’t need to take advantage of the session-full feature of the transport, it can simply skip overriding the CreateSessionEncoder method, and the implementation of the base class (to simply return the value of the Encoder property) will be inherited.

public abstract class MessageEncoder
{
public virtual T GetProperty<T>() where T : class;
public virtual bool IsContentTypeSupported(string contentType);
public Message ReadMessage(ArraySegment<byte> buffer, BufferManager bufferManager);
public Message ReadMessage(Stream stream, int maxSizeOfHeaders);
public abstract Message ReadMessage(ArraySegment<byte> buffer, BufferManager bufferManager, string contentType);
public abstract Message ReadMessage(Stream stream, int maxSizeOfHeaders, string contentType);
public abstract void WriteMessage(Message message, Stream stream);
public ArraySegment<byte> WriteMessage(Message message, int maxMessageSize, BufferManager bufferManager);
public abstract ArraySegment<byte> WriteMessage(Message message, int maxMessageSize, BufferManager bufferManager, int messageOffset);
public abstract string ContentType { get; }
public abstract string MediaType { get; }
public abstract MessageVersion MessageVersion { get; }
// New in 4.5
public virtual IAsyncResult BeginWriteMessage(Message message, Stream stream, AsyncCallback callback, object state);
public virtual void EndWriteMessage(IAsyncResult result);
}

Finally, we need to implement the encoder itself. First, there are some properties which need to be overridden: MediaType (used to identify if the encoder supports incoming requests based on the Content-Type); ContentType (used, along with MediaType, to identify whether the encoder supports the incoming request, and also to determine the Content-Type of outgoing messages written by this encoder) and MessageVersion (again, message version validation happens in all layers). Still on the determination of whether the encoder supports incoming requests, overriding IsContentTypeSupported is often necessary, especially on composite encoders (encoders which delegate requests to multiple inner encoders).

Now comes the actual encoding and decoding of the messages. The many overloads of WriteMessage are used to convert between the Message object and the bytes to be sent over the wire, while the overloads of ReadMessage are used in the opposite direction. Which overload which is used depends on how the encoder is being used: whether it’s being used in buffered or streamed mode – and this is determined by the transport channel.

When a transport is operating in buffered mode, for incoming messages it first receives the whole message and stores it in memory, then it gives it to the encoder (using the overload of ReadMessage which takes an ArraySegment<byte> which contains the whole message). This is useful in a sense that the XML reader can operate over an array of bytes and can work with a better performance than one operating over a stream, but this limits the size of messages which can be processed (buffering very large messages would quickly exhaust the resources of the component). One thing which should be noted is that the encoder is responsible for releasing the buffer passed to it (by calling ReturnBuffer on the buffer manager object passed to ReadMessage).

For outgoing messages in buffered mode, the overload which takes a BufferManager parameter and returns an ArraySegment<byte> is invoked. It’s the encoder’s responsibility to return a buffer taken from that buffer manager (using the BufferManager.TakeBuffer method) which will later be released by the transport.

When the transport is operating in streamed mode, it doesn’t read the whole message before passing it to the application. Instead, the encoder is responsible for reading only the message headers, and returning a Message object which holds a cursor to the transport body, and that can later be consumed when the message reaches the application layer. There are no buffer managers in this scenario.

Finally, for outgoing messages in streamed mode, WriteMessage(Message, Stream) is called and the encoder will pump from the message and write it to the transport stream as the message is being consumed. Notice that this is a synchronous call which blocks the calling thread until the message is completed, so in the .NET Framework 4.5 WCF is adding an asynchronous version of this method (BeginWriteMessage and EndWriteMessage) to enable for a better performance in high-throughput scenarios.

How to add a message encoder

As I mentioned before, adding a message encoder to the WCF pipeline is done in a way similar to protocol channels, but instead of adding itself directly in the pipeline the encoder simply adds itself to the binding context, so that some other channel (in this case, the transport), will pick it up and use it when necessary. The binding element needs to derive from . To add an encoder to the server side only, this needs to be done at BuildChannelListener<TChannel>, and to to add an encoder at the client side only, this needs to be done at BuildChannelFactory<TChannel> (to add the encoder on both sides, both methods need to be overridden). The binding element must also override CreateMessageEncoderFactory (otherwise the code wouldn’t compile), and there return an instance of a class derived from MessageEncoderFactory. Finally, the factory must at least return in its Encoder property and instance of a class derived from MessageEncoder. The code below shows an encoder which can be used on both server and client.

public class MyMessageEncodingBindingElement : MessageEncodingBindingElement
{
public override MessageEncoderFactory CreateMessageEncoderFactory()
{
return new MyMessageEncoderFactory();
}
public override MessageVersion MessageVersion
{
get { return MessageVersion.Soap11; }
set
{
if (value != MessageVersion.Soap11)
{
throw new ArgumentException("Invalid message version");
}
}
}
public override BindingElement Clone()
{
return new MyMessageEncodingBindingElement();
}
public override IChannelFactory<TChannel> BuildChannelFactory<TChannel>(BindingContext context)
{
context.BindingParameters.Add(this);
return context.BuildInnerChannelFactory<TChannel>();
}
public override IChannelListener<TChannel> BuildChannelListener<TChannel>(BindingContext context)
{
context.BindingParameters.Add(this);
return context.BuildInnerChannelListener<TChannel>();
}
class MyMessageEncoderFactory : MessageEncoderFactory
{
MessageEncoder encoder;
public override MessageEncoder Encoder
{
get
{
if (this.encoder != null)
{
this.encoder = new MyMessageEncoder();
}
return this.encoder;
}
}
public override MessageVersion MessageVersion
{
get { return MessageVersion.Soap11; }
}
}
class MyMessageEncoder : MessageEncoder
{
// Implementation of MessageEncoder
}
}

Notice that this encoder shown above supports a single message version (SOAP 1.1, without addressing), but there are many encoders which support multiple message versions, such as the TextMessageEncodingBindingElement and the MtomMessageEncodingBindingElement (the binary encoder only supports SOAP 1.2 with WS-Addressing version 1.0, and the web encoder doesn’t support SOAP – i.e., it requires MessageVersion.None).

Real world scenario – a composite encoder

There were quite a few good examples which I’ve found in the past, but a composite encoder is a good example because it’s fairly simple enough because I’ve seen it in quite a few occasions. This scenario happened for an internal team in Microsoft, and they needed a server endpoint which could receive multiple formats, but always respond with “normal”, text-based XML. A custom encoder is a fairly simple way of implementing it (at least as simple as a channel-level extensibility can be in WCF).

To start off, a simple contract we’ll use in this example. Since MTOM is one of the accepted encodings, let’s use a data contract with a byte[] member, so it can be optimized by MTOM for large messages.

[DataContract]
public class MyDC
{
[DataMember]
public string Name;
[DataMember]
public byte[] Contents;
}
[ServiceContract]
public interface ITest
{
[OperationContract]
MyDC Echo(MyDC input);
}
public class Service : ITest
{
public MyDC Echo(MyDC input)
{
HttpRequestMessageProperty requestProperty;
requestProperty = (HttpRequestMessageProperty)OperationContext.Current.IncomingMessageProperties[HttpRequestMessageProperty.Name];
Console.WriteLine("In service, incoming content-type = {0}", requestProperty.Headers[HttpRequestHeader.ContentType]);
return input;
}
}

And on to the encoding binding element. As most composite encoders, it stores references to the nested binding elements. And since we have a message version validation throughout the whole stack, we need to validate that we don’t get two encoders with different versions.

class MyReadMtomWriteXmlEncodingBindingElement : MessageEncodingBindingElement
{
public MtomMessageEncodingBindingElement mtomBE;
public TextMessageEncodingBindingElement textBE;
public MyReadMtomWriteXmlEncodingBindingElement()
{
this.mtomBE = new MtomMessageEncodingBindingElement();
this.textBE = new TextMessageEncodingBindingElement();
}
public MyReadMtomWriteXmlEncodingBindingElement(MtomMessageEncodingBindingElement mtomBE, TextMessageEncodingBindingElement textBE)
{
if (textBE.MessageVersion != mtomBE.MessageVersion)
{
throw new ArgumentException("MessageVersion of the two inner encodings must be the same.");
}
this.mtomBE = mtomBE;
this.textBE = textBE;
}
}

And before going further, the usual disclaimer: this is a sample for illustrating the topic of this post, this is not production-ready code. I tested it for a few contracts and it worked, but I cannot guarantee that it will work for all scenarios (please let me know if you find a bug or something missing). The error checking is kept to a minimum to make the code focus on the topic of this post.

Now for the rest of the binding element class. Since we’re only using this encoder at the server, we don’t need to override at BuildChannelFactory<TChannel>, only BuildChannelListener<TChannel>. And the abstract class also requires us to provide a Clone method, so we have it as well. And, in an usual pattern for composite encoders, we pass to the factory constructor a factory for each of the encoder types which are being wrapped.

public override MessageEncoderFactory CreateMessageEncoderFactory()
{
return new MyReadMtomWriteXmlEncoderFactory(
this.mtomBE.CreateMessageEncoderFactory(),
this.textBE.CreateMessageEncoderFactory());
}
public override MessageVersion MessageVersion
{
get { return this.mtomBE.MessageVersion; }
set
{
this.textBE.MessageVersion = value;
this.mtomBE.MessageVersion = value;
}
}
public override BindingElement Clone()
{
return new MyReadMtomWriteXmlEncodingBindingElement(
(MtomMessageEncodingBindingElement)this.mtomBE.Clone(),
(TextMessageEncodingBindingElement)this.textBE.Clone());
}
public override bool CanBuildChannelListener<TChannel>(BindingContext context)
{
return this.mtomBE.CanBuildChannelListener<TChannel>(context);
}
public override IChannelListener<TChannel> BuildChannelListener<TChannel>(BindingContext context)
{
context.BindingParameters.Add(this);
return context.BuildInnerChannelListener<TChannel>();
}

The factory class is fairly simple. Since this encoder doesn’t deal with sessions, we don’t need to override the CreateSessionEncoder method. We’re also returning a new instance of the encoder class every time the Encoder property is called, but it could also be cached in a local variable for a simple performance improvement.

class MyReadMtomWriteXmlEncoderFactory : MessageEncoderFactory
{
MessageEncoderFactory mtom;
MessageEncoderFactory text;
public MyReadMtomWriteXmlEncoderFactory(MessageEncoderFactory mtom, MessageEncoderFactory text)
{
this.mtom = mtom;
this.text = text;
}
public override MessageEncoder Encoder
{
get { return new MyReadMtomWriteXmlEncoder(this.mtom.Encoder, this.text.Encoder); }
}
public override MessageVersion MessageVersion
{
get { return this.mtom.MessageVersion; }
}
}

Now to the encoder. Both the content and media type properties are coming from the text encoder (since we want it to encode outgoing messages using the text encoder). The message version property could come from either encoder, since we enforced at the binding element level that they should match. And for the IsContentTypeSupported method, a typical implementation would be to return true if either inner encoders supported the given content type. However, the MTOM message encoder can correctly decode messages encoded using the text XML, so it covers all the cases by itself.

class MyReadMtomWriteXmlEncoder : MessageEncoder
{
MessageEncoder mtom;
MessageEncoder text;
public MyReadMtomWriteXmlEncoder(MessageEncoder mtom, MessageEncoder text)
{
this.mtom = mtom;
this.text = text;
}
public override string ContentType
{
get { return this.text.ContentType; }
}
public override string MediaType
{
get { return this.text.MediaType; }
}
public override MessageVersion MessageVersion
{
get { return this.text.MessageVersion; }
}
public override bool IsContentTypeSupported(string contentType)
{
return this.mtom.IsContentTypeSupported(contentType);
}
}

The implementation of WriteMessage and ReadMessage are trivial in this scenario: since the MTOM encoder can correctly parse both text and MTOM requests, we simply delegate reading the message to it. And since we always want to write the message using the text encoder, we delegate writing the message to this encoder. There are other composite scenarios, where for example we want to respond using the same content-type as the request, and those require more work. Since the ContentType property doesn’t vary based on the outgoing message, we need to use another component (such as a message inspector) to, on incoming messages (on AfterReceiveRequest), retrieve the content type and return it as the correlation state. On the inspector, on BeforeSendReply, take the content-type passed as the correlation state and store it in the message properties. And finally, on the encoder we’d look at the property of the message being encoded, and then decide which nested encoder to use to encode that message.

public override Message ReadMessage(ArraySegment<byte> buffer, BufferManager bufferManager, string contentType)
{
return this.mtom.ReadMessage(buffer, bufferManager, contentType);
}
public override Message ReadMessage(Stream stream, int maxSizeOfHeaders, string contentType)
{
return this.mtom.ReadMessage(stream, maxSizeOfHeaders, contentType);
}
public override ArraySegment<byte> WriteMessage(Message message, int maxMessageSize, BufferManager bufferManager, int messageOffset)
{
return this.text.WriteMessage(message, maxMessageSize, bufferManager, messageOffset);
}
public override void WriteMessage(Message message, Stream stream)
{
this.text.WriteMessage(message, stream);
}

Now for testing the encoder. We’re setting up a simple service, with the composite encoder, and sending two requests to it, one using the MTOM encoding, and one using the text encoding. We’re printing the content-type of the response, and it shows that the server can accept both encodings, but is always responding with the one we want.

public class Program
{
static Binding GetServerBinding()
{
return new CustomBinding(
new MyReadMtomWriteXmlEncodingBindingElement(),
new HttpTransportBindingElement());
}
public static void Main()
{
string baseAddress = "https://" + Environment.MachineName + ":8000/Service";
ServiceHost host = new ServiceHost(typeof(Service), new Uri(baseAddress));
host.AddServiceEndpoint(typeof(ITest), GetServerBinding(), "");
host.Open();
Console.WriteLine("Host opened");
foreach (bool useMtom in new bool[] { false, true })
{
MessageEncodingBindingElement clientEncoding;
if (useMtom)
{
clientEncoding = new MtomMessageEncodingBindingElement();
}
else
{
clientEncoding = new TextMessageEncodingBindingElement();
}
CustomBinding binding = new CustomBinding(clientEncoding, new HttpTransportBindingElement());
ChannelFactory<ITest> factory = new ChannelFactory<ITest>(binding, new EndpointAddress(baseAddress));
ITest proxy = factory.CreateChannel();
byte[] fileContents = new byte[10000];
for (int i = 0; i < fileContents.Length; i++)
{
fileContents[i] = (byte)('a' + (i % 26));
}
using (new OperationContextScope((IContextChannel)proxy))
{
proxy.Echo(new MyDC { Name = "FileName.bin", Contents = fileContents });
HttpResponseMessageProperty responseProperty;
responseProperty = (HttpResponseMessageProperty)OperationContext.Current.IncomingMessageProperties[HttpResponseMessageProperty.Name];
Console.WriteLine("In client, response content-type: {0}", responseProperty.Headers[HttpResponseHeader.ContentType]);
}
((IClientChannel)proxy).Close();
factory.Close();
}
Console.Write("Press ENTER to close the host");
Console.ReadLine();
host.Close();
}
}

And that’s it about this sample.

[Code in this post]

[Back to the index]

Comments

Anonymous
November 27, 2011
WCF 4 comes with a new MessageEncoder named 'ByteStreamMessageEncodingBindingElement', but I've found little information about it. Appreciate if you could illustrate more on this binding. Thanks!
Anonymous
December 07, 2011
Hi Elvin, the ByteStreamMessageEncodingBindingElement is an encoder similar to the "raw" encoder which is part of the WebMessageEncodingBindingElement - it converts raw bytes into a message whose body consists of a element <Binary>, and the bytes as a text xs:base64Binary node inside that element. On the sample for the new post (blogs.msdn.com/.../wcf-extensibility-transport-channels-request-channels-part-1.aspx) I use that encoder, if you want to take a look.
Anonymous
August 13, 2014
Hello!The problem is that the service is responding with text/plain when it should be responding with text/xml which is what your WCF binding expects in order to properly handle it. Fundamentally this is an "issue" on the service side: they are returning XML but calling it plaintext, which technically is accurate but not as specific as an XML content type. On the WCF side, the built-in protocol handling is fairly rigid and expects the content type to be text/xml or application/xml in order to proceed with deserializing it.You could thwart their content type problem by writing a custom WCF encoder that supports text/plain but parses it as XML.

Partager via