Share via

Binary Encoding, Part 4

  • Article

Past parts in the series:

Now that you’ve gotten an introduction to the principles and capabilities of the binary encoding format, let’s jump into looking at some examples of messages to see how it works.
Here’s a very short but inefficiently encoded binary message. We’ll see later how to make it quite a lot shorter.

0x41, 0x01, 0x73, 0x08, 0x45, 0x6E, 0x76, 0x65, 0x6C, 0x6F, 0x70, 0x65, 0x09, 0x01, 0x73, 0x27, 0x68, 0x74, 0x74, 0x70, 0x3A, 0x2F, 0x2F, 0x77, 0x77, 0x77, 0x2E, 0x77, 0x33, 0x2E, 0x6F, 0x72, 0x67, 0x2F, 0x32, 0x30, 0x30, 0x33, 0x2F, 0x30, 0x35, 0x2F, 0x73, 0x6F, 0x61, 0x70, 0x2D, 0x65, 0x6E, 0x76, 0x65, 0x6C, 0x6F, 0x70, 0x65, 0x01

This is admittedly unenlightening so let’s start parsing out the pieces of the message. Here’s the procedure I described earlier in part 2:

Each record starts with a one byte record type value. The record type byte is then followed by binary content of variable format and size based on the type. Each record in the stream of records translates into a document fragment. By concatenating all of the fragments produced from the record stream together we can obtain a document based on the original XML infoset.

The first byte in the message, 0x41, is the record type for an XML element. The format of an element record is the length of the prefix string, the bytes for the prefix, the length of the element name, and the bytes for the element name.

The next byte in the message, 0x01, is the length of the prefix, which means that 0x73 are the bytes for the prefix. If you go to your standard UTF-8 or ASCII character table, you’ll see that 0x73 is the byte sequence for the letter "s".

Similarly, the next byte in the message, 0x08, is the length of the element name, which means that 0x45, 0x6E, 0x76, 0x65, 0x6C, 0x6F, 0x70, 0x65 are the bytes for the element name. Translating that byte sequence into characters we get the letters "Envelope".

That means we’ve just processed a record that gives us the XML fragment "<s:Envelope". We don't know yet whether we've seen the entire XML element yet. For example, the following records could be attributes or just as easily could be the content within the element. We'll find out the context when we get there.

The next byte in the message, 0x09, is the record type for an XML namespace declaration. Recall that we used the prefix "s" with the envelope element so we do owe the reader a definition of that namespace. The format of a namespace record is the length of the prefix string followed by the bytes for the prefix.

The next byte in the message, 0x01, is the length of the prefix, which means that 0x73 are the bytes for the prefix and we indeed are now defining that prefix we saw earlier. Then, the next byte in the message is 0x27 indicating that we have 39 bytes of character data coming. That character data forms the string "www.w3.org/2003/05/soap-envelope".

The XML fragment for this record is "xmlns:s="www.w3.org/2003/05/soap-envelope"".

Finally, the next and final byte in the message, 0x01, is the record type for closing an element. There's no ambiguity about the element to close because the binary encoding only supports well-formed XML and we always know the most recently started element. Therefore, the end element record doesn't have any data.

The XML fragment for this record is "</s:Envelope>".

That was the last record in the message so we can put the fragments together to see that the message spells out a very common portion of a SOAP message:

<s:Envelope xmlns:s="www.w3.org/2003/05/soap-envelope"></s:Envelope>

It would be nice to not have to carry all of that character data because the same strings are going to be in virtually every SOAP message that is sent. There was virtually no size advantage to using the binary format in this case as compared to a text encoding. Next time we'll look at how to squeeze down the size of the message.

Comments

  • Anonymous
    September 22, 2009
    Nicholas, Thanks greatly for this background.  Until recently there was nothing out there detailing how the NetTcp protocol worked.  I have a few questions: I believe that the WCF binary protocol requires that the serialized objects are projected through XML.  I believe this to be true since the DCS has overloads that take a Stream but they are never called.  Rather only the overloads for XmlDictionaryWriter are called.  This means that an object is serialized to an XmlDictionaryWriter first and then encoded to binary.  Please confirm my understanding here. If above is true, wouldn't it be more efficient to allow objects to serialize direct to binary and bypass going to the XmlDictionaryWriter? I like the NetTcp protocol efficiency compared to SOAP, but it is not interoperable with other client stacks.  Can you consider leveraging something like Protocol Buffers so that the wire format would be compatible with downstream clients? Thanks, -Scott  

  • Anonymous
    September 22, 2009
    Hi Scott, You'll notice on XmlDictionaryWriter that there is a factory for creating instances of binary writers.  There is an internal class that this factory produces that looks like an XmlWriter but directly outputs the data in the binary format.  We're already skipping the intermediate step here. Protocol buffers can't represent everything that WCF produces but the encoding of content is quite similar to the binary format.  I think you could write a message encoder that could handle it without too much trouble.  It would be a bit more work to integrate protocol buffer format files with data and message contracts for your service and proxies though.

  • Anonymous
    September 24, 2009
    Thanks. I am using the protobuf.net library from Marc Gravel(http://code.google.com/p/protobuf-net/) which replaces the DCS to serialize the objects directly to the PB format.  The problem is that the encoder will never call the Stream overloads even when I use a binary encoder(at least I can't get it to).  So, I am forced to take the protobuf serialized buffer and Base64 encode it and place it into the XmlDictionaryWriter calling WriteBase64.  This is not as efficient as simply sending the binary buffer over the wire.  If you have any thoughts, let me know. All of this works from .net clients to servers now as long as I swap the DCS out on both sides by passing the base64 encoded buffer. You are correct in that protobuf only supports a subset of what is in WCF.  But it has the benefits of being extremely efficient and portable which is vital since we have a wide variety of client stacks accessing your services. Finally, you elude to the fact that this technique plays with the MEX exposure so that foreign clients would not understand that the buffer is not SOAP.  I believe the eventual answer here is to expose either .proto files and/or an XSD/WSDL to .proto generator.  These definitions can then be plugged into client stacks. Your thoughts would be appreciated.  I can send you code samples if this would make more sense.  

  • Anonymous
    September 28, 2009
    Hi Scott, As you've seen, the method for writing binary content is called WriteBase64 but the argument it takes is the unencoded array of bytes. public abstract void WriteBase64(byte[] buffer, int index, int count) Similarly, when the message encoder is outputting the content, it is writing the unencoded array of bytes to the wire. If you've written a message encoder, then you're the one responsible for creating the XmlWriter and control the implementation of WriteBase64.  You're also the one responsible for outputting the content.  Who can prove whether you actually converted it to base 64 in the middle? (Answer: only if an extensibility point such as a layered channel or message inspector is being used to examine the message- but reading the message implies making a copy so the slow path of having to actually perform the base 64 encoding is only a portion of the cost.)