Sidebar - What is really going on with my stream?

This posting might not directly help you implement something, but it should give some insight as to what's going on under the hood with a stream sent to a client from a service.

Scenario

There's a WCF service with an operation which returns a Stream.

A WCF Client invokes this operation.

The Transfer Mode is Streamed.

So, here's what happens, diagram first:

To elaborate:

1. The client call is received by the service.  The service creates some custom stream in the operation and returns it.

2. Because transfer mode is Streamed, the Encoder gets a wire stream handed to it from the transport layer. 
3. Using the wire stream and the stream handed from the service (which I'll call CustomStream from here on), the encoder then begins to create a message.  It starts with all the Soap stuff, but when it comes to the body, it starts the streamed transfer.  It calls a method which takes the two streams.  This method basically gets a buffer from the wire stream and fills it up with bytes from the CustomStream.  Then it calls Write on the wire stream, passing the buffer.  This is what sends the bytes out on the wire.  If the Read call on the CustomStream returned exactly the same value as that buffer, then we know there are more bytes to write.  To make things more efficient, subsequent Read calls get a larger buffer allocated.

What if I used TransferMode.Buffered?

Instead of the Transport giving the WireStream to the encoder, the encoder will simply read the entire  CustomStream, package it up in a single message, and hand that message off to the transport which will then send it on the wire.  If your implementation is actually returning very small streams with many client calls, this might actually be optimal.  Otherwise, TransferMode.Streamed is the better option.

Last time, I introduced this topic and gave an example of a custom stream implementation.  This time, I'm going to make a few enhancements to this implementation and get into the topic of using it in a WCF Service.

Streaming with WCF

Part 2: Plugging a custom stream into a WCF service

So, I've made a couple modifications to the CustomStreamImplementation.  I added a couple knobs to control when it stops streaming.  Here's the complete code:

using System;

using System.IO;

using System.Threading;

namespace CustomStreamImplementation

{

    public class ContinuousStream : Stream

    {

        // Used to control when Read will return 0.

        public bool StopStreaming { get; set; }

        public TimeSpan ReadThrottle { get; set; }

        // Only set this if you don't want to manually control when

        // the stream stops.

        // Keep it low - less than 1 second.  The server can send bytes very quickly (without a throttle), so

        // sending a continuous stream will easily blow the MaxReceivedMessageSize buffer.

        public TimeSpan StreamDuration { get; set; }

        DateTime readStartedTime;

        long totalBytesRead = 0;

        public override bool CanRead

        {

            get { return !StopStreaming; }

        }

        public override bool CanSeek

        {

            get { return false; }

        }

        public override bool CanWrite

        {

            get { return false; }

        }

        public override long Length

        {

            get { throw new NotImplementedException(); }

        }

        public override long Position

        {

            get

            {

                return totalBytesRead;

            }

            set

            {

                totalBytesRead = value;

            }

        }

        public override void Flush()

        {

        }

        public override int Read(byte[] buffer, int offset, int count)

        {

            // Duration-based streaming logic: Control the "StopStreaming" flag based on a Duration

            if (StreamDuration != TimeSpan.Zero)

            {

                if (readStartedTime == DateTime.MinValue)

                {

                    readStartedTime = DateTime.Now;

                }

                if (DateTime.Now - readStartedTime >= StreamDuration)

                {

                    StopStreaming = true;

                }

            }

            if (StopStreaming)

            {

                buffer[offset] = 0;

                return 0;

            }

            // Allow Read to continue as long as StopStreaming is false.

            // Just fill buffer with as many random bytes as necessary.

            int seed = DateTime.Now.Millisecond;

            Random rand = new Random(seed);

            byte[] randomBuffer = new byte[count];

            rand.NextBytes(randomBuffer);

            randomBuffer.CopyTo(buffer, offset);

            totalBytesRead += count;

            if (ReadThrottle != TimeSpan.Zero)

            {

                Thread.CurrentThread.Join(ReadThrottle);

            }

            return count;

        }

        public override long Seek(long offset, SeekOrigin origin)

        {

            throw new NotImplementedException();

        }

        public override void SetLength(long value)

        {

            throw new NotImplementedException();

        }

        public override void Write(byte[] buffer, int offset, int count)

        {

            throw new NotImplementedException();

        }

    }

}

Downloading the stream from the service

First, I'll create the WCF service.  I'll make it selfhosted to simplify things a bit and give me full control of what code I add.

The service is a new Console Application in my solution.  I added a reference to the CustomStreamImplementation project.  The first time I ever created a WCF service that was supposed to return a custom stream, I thought I'd end up having a contract that returned something of type "CustomStream."  But then I was trying to figure out how to stream something from within the service operation, yet still return from the operation with the stream still going.  It just didn't make sense, until I asked around and figured out what really happens.  Your Stream object is basically passed down to the channel layer, which then invokes Read on the stream when the request comes in to download the stream.  Bytes from the stream are loaded into a buffer and sent on the wire.  As long as Read doesn't return 0, then it continues to create buffers and send them on the wire.  I'll go into high-level details about how this works in a separate post.

Service Implementation

I'll just paste the entire code; it's pretty simple.

Contract:

using System.IO;

using System.ServiceModel;

namespace CustomStreamService

{

    [ServiceContract]

    public interface ICustomStreamService

    {

        [OperationContract]

        Stream DownloadStream();

    }

}

Implementation:

using System;

using System.IO;

using System.ServiceModel;

using CustomStreamImplementation;

namespace CustomStreamService

{

    [ServiceBehavior]

    class CustomStreamService : ICustomStreamService

    {

        [OperationBehavior]

        public Stream DownloadStream()

        {

            ContinuousStream continuousStream = new ContinuousStream();

            continuousStream.StreamDuration = TimeSpan.FromSeconds(30);

            continuousStream.ReadThrottle = TimeSpan.FromSeconds(1);

            return continuousStream;

        }

    }

}

Driver - The service host and client code:

using System;

using System.IO;

using System.ServiceModel;

namespace CustomStreamService

{

    class Program

    {

        static void Main(string[] args)

        {

            // Service code

            string address = "https://localhost/CustomStreamService";

            BasicHttpBinding binding = new BasicHttpBinding();

            binding.TransferMode = TransferMode.Streamed;

            ServiceHost host = new ServiceHost(typeof(CustomStreamService), new Uri(address));

            host.AddServiceEndpoint(typeof(ICustomStreamService), binding, address);

            host.Open();

            Console.WriteLine("Service is {0}, Press ENTER to continue.", host.State);

            Console.ReadLine();

            // Client code

            // For ad-hoc testing, I'll just re-use the binding, address, and contract that's defined in this project.

            // If I wanted to test the client cross-machine, I would have to either generate it using wsdl and svcutil, or

            // simply copy-paste the contract (ICustomStreamService) into the client project.  Then make sure the binding and address are correct.

            ChannelFactory<ICustomStreamService> factory = new ChannelFactory<ICustomStreamService>(binding, address);

            ICustomStreamService client = factory.CreateChannel();

            Stream streamFromService = client.DownloadStream(); // Note that my Client knows nothing about the stream implementation.

            DateTime beginReadTime = DateTime.Now;

            Console.WriteLine("{0}: starting to read.", beginReadTime);

            // My oddball way of reading the bytes from the stream and writing them to the console.

            int bytesRead;

            int bufferSize = 256;

            byte[] buffer = new byte[bufferSize];

            do

            {

                bytesRead = streamFromService.Read(buffer, 0, buffer.Length);

                foreach (byte b in buffer)

                {

                    Console.Write(b);

                    Console.Write(' ');

                }

            } while (bytesRead != 0);

            Console.WriteLine("{0}: done reading.  Elapsed time is {1}", DateTime.Now, DateTime.Now - beginReadTime);

            Console.ReadLine();

        }

    }

}

Troubleshooting

If you were to add all this code into a console app and run it, you'd see the client downloading bytes for a few seconds, then the streamFromService.Read call would eventually throw an IOException ("An exception has been thrown when reading the stream.")  Examining the exception shows the following inner exception:

"The maximum message size quota for incoming messages (65536) has been exceeded. To increase the quota, use the MaxReceivedMessageSize property on the appropriate binding element."

But I thought I was Streaming?

Don't worry, you still are.  But the MaxReceivedMessageSize is a mechanism for "time-boxing" the reading of the stream.  Generally, you'd want this on a service that reads a stream from a client.  You don't want a client to be able to tie up the service forever with an infinite stream, so this property takes care of that.  Similarly, the property applies for clients.  If you want to read a larger stream, simply do what the exception says.  In this scenario, I'm just going to open the floodgates and let the client read  as long as I can:

            binding.MaxReceivedMessageSize = int.MaxValue;

With that, my client could read the entire 30 seconds of the stream from the service:

17/2011 10:12:51 AM: done reading.  Elapsed time is 00:00:34.6390000

If you're wondering about that extra 4.639 seconds of processing time, increasing the buffer size used by the read operation can increase the performance of my ad-hoc client.  But I suggest you play with it - there's certainly an optimal value for this buffer, based on the size of the buffer used by the wire stream on the service.  Set the bufferSize to something large, like 4k, and you'll see what I mean.  Of course, logging each byte has the largest impact on performance.

Still can't get streaming to work?

Leave me a comment.  I'll try to troubleshoot any issues with streaming and WCF that I can.  I'm considering writing a brief posting about the things in WCF that don't support streaming.  Off the top of my head, I can think of 2: MessageSecurity and Streaming on a Duplex Callback.  There's probably a handful of other streaming-not-supported scenarios.

After realizing I wasn't the only one stumbling around with this, I decided I should write a series of posts related to streaming in WCF.  So with that, let me start the posts.

Streaming with WCF

Part 1: Custom Stream Implementations

Before I get into details of WCF and streaming, I first want to give an example of a custom stream implementation.  See, anyone can stream to or from, say, a file with the following code:

        public static void TestFileStream()

        {

            // Writing

            FileStream fs = new FileStream("C:\\test.txt", FileMode.OpenOrCreate);

            byte[] bytesToWrite = Encoding.ASCII.GetBytes("Yeah, this is the API for writing a string to a file.");

            fs.Write(bytesToWrite, 0, bytesToWrite.Length);

            fs.Close();

            // Reading

            fs = new FileStream("C:\\test.txt", FileMode.Open);

            byte[] buffer = new byte[bytesToWrite.Length];

            fs.Read(buffer, 0, bytesToWrite.Length);

            Console.WriteLine(Encoding.ASCII.GetString(buffer));

        }

"fs" is the stream, obviously.  Which is fine for basic tests, but what if I wanted to send an infinite stream?  What if I wanted to open a stream to the server and start pushing bytes, one at a time, until I hit CTRL + C?  What if someone had a WCF service that would stream data back to the client on the fly?  How do you write a custom stream for that?

1. I start by extending the Stream class:

    public class ContinuousStream : Stream

    {

2. Next, I think about what I want to do.  I want it to be able to be used by a WCF service to write bytes to the wire continually.  How do you override the way a stream writes bytes?  Why, override the Read method, of course:

        public override int Read(byte[] buffer, int offset, int count)

        {

            // Allow Read to continue forever

            // Just fill buffer with as many random bytes as necessary.

            int seed = DateTime.Now.Millisecond;

            Random rand = new Random(seed);

            byte[] randomBuffer = new byte[count];

            rand.NextBytes(randomBuffer);

            randomBuffer.CopyTo(buffer, offset);

            totalBytesRead += count;

            return count;

        }

Now, it's time to give some background information about how WCF sends a stream on the wire.  If I were to use this in a service operation called, "DownloadStream" it would return an object of type Stream and take no parameters.  I would just have to return a new instance of this object.  When the service gets invoked by a client call, assuming it's using streamed transfer mode, WCF will basically hand this object down to the transport which will call the Read method, passing in a buffer that will be sent on the wire.  Assuming Read doesn't return 0, indicating there are no bytes to read, it will continually call Read and send the filled buffers on the wire. 

With that bit of knowledge, the first thing I want to add to the Continuous stream is a throttle to slow down the time between Read calls.  To do this, I add a property to the ContinuousStream class:

        public TimeSpan ReadThrottle { get; set; }

And I change the implementation of Read:

        public override int Read(byte[] buffer, int offset, int count)

        {

            // Allow Read to continue forever

            // Just fill buffer with as many random bytes as necessary.

            int seed = DateTime.Now.Millisecond;

            Random rand = new Random(seed);

            byte[] randomBuffer = new byte[count];

            randomBuffer.CopyTo(buffer, offset);

            totalBytesRead += count;

            if (ReadThrottle != TimeSpan.Zero)

            {

                Thread.CurrentThread.Join(ReadThrottle);

            }

            return count;

        }

How you override the Read method is totally up to you.  You could create a WCF client in here and call a service to get some data which you then put in the buffer somehow.

Next time, I'll show some more minor enhancements to the ContinuousStream and use it in a WCF service.

As a tester in .NET (more specifically, WCF), every now and then, I end up defining a test requirement that isn't possible to cover with simply using our WCF product code.  For example, in 4.0, I wanted to test hitting a service with an Http request that contained an invalid host header.  Now, the first thing I turned to was the familiar HttpWebRequest.  Poking around at the properties, I couldn't find any way to provida an intentionally incorrect Host header.  It was at that time I realized the only way to send this type of request to my server was to use a raw socket.  Today, I'd like to share that code with you.  The following snippet sends a GET request to Bing and prints the result to the console.  Use your imagination with this on how you can potentially modify the raw request sent on the wire.  Basically, with this snippet, you have absolute control over the bytes you send.

class Program

{

    static void Main(string[] args)

    {

        // Create the message to be sent on the wire

        // The first header contains the resource to which the request is directed.

        // It's basically just "https://{host}/{resource}".  Here, my resource is "/".

        string message = "GET https://www.bing.com/ HTTP/1.1";

        message += "\nAccept: text/html, application/xhtml+xml, */*";

        message += "\nAccept-Language: en-US";

        message += "\nHost: www.bing.com";

        message += "\n\n";

        // Don't forget those 2 newlines. After that, the message body could be added, but

        // for this example, I'm just sending a GET which isn't allowed to contain a body.

        // (Though I could still add one since I'm accessing the stream directly.  You can't do that with Fiddler.)

        // The TcpClient specifies the destination of the request (host) as well as the port.

        TcpClient client = new TcpClient("www.bing.com", 80);

        NetworkStream networkStream = client.GetStream();

        Byte[] data = Encoding.ASCII.GetBytes(message);

        networkStream.Write(data, 0, data.Length);

        networkStream.Flush();

        string responseData = string.Empty;

        Int32 bytes;

        do

        {

            data = new Byte[4]; // Yes, this is reading very small chunks at a time.

            bytes = networkStream.Read(data, 0, data.Length);

            responseData += Encoding.UTF8.GetString(data, 0, bytes);

        } while (networkStream.DataAvailable);

        // Now we have the response, in a string.

        Console.WriteLine(responseData);

        Console.ReadLine();

    }

}

Disclaimer: The information contained in this posting is not intended for production environments.  But it should effectively mimic a production environment.

A while back, I posted a lengthy how-to for setting up a selfhosted WCF service that uses Transport level security (https.)  I found that posting useful, when trying to remember how to set that up, but recently I was helping a colleague set up a webhosted service using https.  Now, for most cases, testing this with your own self-signed certificate works fine, but if you ever try browsing to the service cross-machine, you'll notice the certificate isn't trusted.  In this posting, I'll review how to create the chain-signed certificate and how to export the trusted root certificate to a "client" machine.

Setting up the service

For reference, I'll be using the service implementation in the "Selfhosting a WCF service over Https" posting.

The difference is that I've changed the project type to a class library and added a HelloWorldService.svc file with the following content:

<%@ ServiceHost  Debug="true" Service="HelloWorldWebhost.HelloWorldService" Language="C#" Factory="System.ServiceModel.Activation.ServiceHostFactory" %>

And I added a Web.Config with the following content:

<configuration>

  <system.serviceModel>

    <bindings>

      <basicHttpBinding>

        <binding name="MyBinding"/>

      </basicHttpBinding>

    </bindings>

    <behaviors>

      <serviceBehaviors>

        <behavior name="metadata">

          <serviceMetadata httpGetEnabled="true"/>

        </behavior>

      </serviceBehaviors>

    </behaviors>

    <services>

      <service name="HelloWorldWebhost.HelloWorldService" behaviorConfiguration="metadata">

        <endpoint address="" binding="basicHttpBinding" bindingConfiguration="MyBinding" contract="HelloWorldWebhost.IHelloWorld"/>

      </service>

    </services>

  </system.serviceModel>

</configuration>

Finally, I changed the output path from "bin\Debug\" to "bin\" in the project properties.  Next, I open IIS (IIS7, in my case), right click "Default Web Site" and select "Add Application."  The virtual path (Alias) is AdHocHelloWorld and the physical path is the project folder I was using in VS.  Next, I right-click the application in IIS and select "Edit Permissions" to give IIS_IUSRS access to the folder.  Finally, I switched to Content View in IIS and browsed to the .svc file to make sure I got the service activation page.

Modifying for https

To change the service to use Transport security, all I have to do is change the binding:

      <basicHttpBinding>

        <binding name="MyBinding">

          <security mode="Transport"/>

        </binding>

      </basicHttpBinding>

Now, if I just refresh the service in IE, I get the following error:

Could not find a base address that matches scheme https for the endpoint with binding BasicHttpBinding. Registered base address schemes are [http].

This is where you need to add a binding in IIS for https.  And to do that, you need a certificate.  Looking at my old blog post, I run the following two commands on my machine:

makecert.exe -sk RootCA -sky signature -pe -n CN=<machineName> -r -sr LocalMachine -ss Root MyCA.cer

makecert.exe -sk server -sky exchange -pe -n CN=<machineName> -ir LocalMachine -is Root -ic MyCA.cer -sr LocalMachine -ss My <certificate path>

Of course, I changed <machineName> to be my machine's fully-qualified machine name.  And the <certificate path> was MyAdHocWebhostHttpsCert.cer.

Next, I right-click Default Web Site, select Edit Bindings…  Under Type, I select https.  The SSL certificate dropdown now is populated with the cert I created in the above steps.  These are labeled by "CN" parameter used. 

Now, when I refresh my IE page to the service, it activates just fine.  In fact, if I look at the wsdl, I see the address to the service uses the https scheme now.

Setting up the remote client machine

If I now browse to the service (on https) using IE on a remote machine, I get the following (after remembering to allow Http and Https publishing through my firewall):

You can make your client machine trust the server's certificate by exporting the trusted root certificate from the server machine and importing it on the client machine.  Here's how to do this.  On the server machine, run mmc.exe.  File -> Add or Remove Snap-ins -> Certificates -> Add.  Select Computer account (Local).  Click Ok and unfold down to Trusted Root Certification Authorities.  In there, you should find the certificate that was installed by the first makecert.exe command.  The "Issued To" and "Issued By" are both the machine name.  Right click it and select Export.  For this test, I'm not exporting the private key.  I select the defaults on everything else and pick a file name at the end, placing it in my documents folder. 

Next, I copy the certificate to the client machine.  This is sort of mimicking how Verisign installs their trusted root certificate on all windows machines.  Of course, providers then need to ask Verisign to issue them a certificate to secure their web services with.  But more on that topic is probably widely available elsewhere. 

On the client machine, simply double-click the certificate that was copied.  When the wizard asks which certificate store to install it to, select Trusted Root Certification Authorities.  Everything else, just use the defaults.  Now, if you look at the certificates on the client machine (mmc.exe - Certificates) you should see the certificate under Trusted Root Certification Authorities.  Further, refreshing the IE window on the client machine shows there is no longer a problem with the certificate.

Happy testing!

Today, I just installed the release of WebMatrix, along with the Beta version of SP1 for Visual Studio 2010.  One of the many cool features added with this combination is the IISExpress integration in Visual Studio.  Prior to this, the only options you had was to either use the default ASP.NET Development server in Visual Studio to test your web applications or web sites, or to use the Local IIS.  But assume you don't have IIS installed.  Or you don't have admin rights on your development box.  With IISExpress, you get the advantage on running on the same code base as IIS 7.5 when you test your site.  Even if you are running on a machine without IIS installed.

I was considering publishing an end-to-end walkthrough of creating an ad-hoc HelloWorld WCF service entirely by hand and how to make Visual Studio host it in IISExpress, but this is so simple, once you know where to look, that I think I'll just point it out.

Scenario 1: Web Application

Assuming you have a Web Application in Visual Studio.  When you hit F5, it fires up the ASP.NET Development Server.  To change that to use IISExpress, open the project properties window.  With WebMatrix installed along with SP1 (I'm using Beta as of this post), you'll see the option to use IISExpress:

 Scenario 2: Web Site

Assuming you have a Web Site in Visual Studio, there's a couple ways you can configure it to be launched using IISExpress.

1. From Publish Web Site
   1. Right-click the project and select "Publish Web Site."
   2. Under Target Location, click the […] button.
   3. Select the "Local IIS" button on the left.  IIS Express Sites shows up there, and you can select which vdir to use, or use the "Create new site" button at the top right to make a new one.

2. From "Use IIS Express…"
   1. Right-click the project and select "Use IIS Express…"  Click Yes and OK on the dialogs that follow:

In Beta3, there was an issue where WCF trace logs would be truncated after closing WebMatrix.  See the posting here for historical details.

With the full release of WebMatrix, there is a way to get non-truncated logs.  If you are hosting the site from within WebMatrix UI and just close WebMatrix or select Exit from the menu, you still will likely get truncated logs.  To cleanly exit WebMatrix, first shut down your service by Exiting IISExpress in the system tray:

This will allow WebMatrix to flush the trace data to the log and release the handle on the log file.  You can then view the log and keep WebMatrix open.  Later, you can start the service up again from within WebMatrix.

Recently, I've come across an interesting exception message in one of my Ad-Hoc WCF projects using .Net 4.0.  The exception message is:

The HTTP request is unauthorized with client authentication scheme  'Basic', The authentication header received from the server was 'Negotiate,NTLM,Basic realm="[omitted]"'.

This exception message is a bit odd because it says my client was using Basic, and that the server only allows requests using Negotiate, NTLM, or Basic.  But I used Basic, so why would it throw?

The two possibilities are:

1. The username or password were wrong.  This is the most obvious.

2. The Auth schemes on the service don't match those supported by the host.  In this case, by host, I mean IIS:

So, the host supports Anonymous, Basic, and Windows (NTLM + Negotiate).  (If the host didn't support the auth mode set on the service binding, I would have gotten an activation exception.)  So, a quick peek at the service binding shows that the binding set the ClientCredentialType to "Windows."

Most of the time, this wouldn't pose a problem.  Especially if you are dynamically building the clients from WSDL, because the auth schemes for the binding are exposed there.