In honor of Mandelbrot, here are two very simple takes on the Mandelbrot fractal that I’ve written in the last couple years.

Mandelbrot in the F# Interactive

One of the first pieces of F# code I wrote is below.  I blogged about this here.

Mandelbrot in 175 characters of Javascript

Inspired by the js1k.com competition earlier this year, I adapted Ken Perlin’s very cool C program to Javascript as a 175 character program for rendering a Mandelbrot fractal in any browser.  Couldn’t quite get it to fit in a single tweet though (rule #6 of the js1k challenge).

y=-16;s="<pre>";while(s+='\n',y++<15)for(x=0;x++<84;s+=(" 
.:-;!/>)|&IH%*#"[k&15]))for(i=k=r=0;j=r*r-i*i-2+x/25,i=2*r
*i+y/10,j*j+i*i<11&&k++<111;r=j);document.body.innerHTML=s

Check it out here for the full page source.

[UPDATE:  And here's a slightly more fully featured implementation, using Javascript and HTML5 canvas. ] 

OAuth Implementation

open System
open System.IO
open System.Net
open System.Security.Cryptography
open System.Text

// Twitter OAuth Constants
let consumerKey : string = failwith "Must provide the consumerKey for an app registered at https://dev.twitter.com/apps/new"
let consumerSecret : string = failwith "Must provide the consumerSecret for an app registered at https://dev.twitter.com/apps/new"
let requestTokenURI = "https://api.twitter.com/oauth/request_token"
let accessTokenURI = "https://api.twitter.com/oauth/access_token"
let authorizeURI = "https://api.twitter.com/oauth/authorize"


// Utilities
let unreservedChars = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789-_.~";
let urlEncode str = 
    String.init (String.length str) (fun i -> 
        let symbol = str.[i]
        if unreservedChars.IndexOf(symbol) = -1 then
            "%" + String.Format("{0:X2}", int symbol)
        else
            string symbol)


// Core Algorithms
let hmacsha1 signingKey str = 
    let converter = new HMACSHA1(Encoding.ASCII.GetBytes(signingKey : string))
    let inBytes = Encoding.ASCII.GetBytes(str : string)
    let outBytes = converter.ComputeHash(inBytes)
    Convert.ToBase64String(outBytes)

let compositeSigningKey consumerSecret tokenSecret = 
    urlEncode(consumerSecret) + "&" + urlEncode(tokenSecret)

let baseString httpMethod baseUri queryParameters = 
    httpMethod + "&" + 
    urlEncode(baseUri) + "&" +
    (queryParameters 
     |> Seq.sortBy (fun (k,v) -> k)
     |> Seq.map (fun (k,v) -> urlEncode(k)+"%3D"+urlEncode(v))
     |> String.concat "%26") 

let createAuthorizeHeader queryParameters = 
    let headerValue = 
        "OAuth " + 
        (queryParameters
         |> Seq.map (fun (k,v) -> urlEncode(k)+"\x3D\""+urlEncode(v)+"\"")
         |> String.concat ",")
    headerValue

let currentUnixTime() = floor (DateTime.UtcNow - DateTime(1970, 1, 1, 0, 0, 0, 0)).TotalSeconds


/// Request a token from Twitter and return:
///  oauth_token, oauth_token_secret, oauth_callback_confirmed
let requestToken() = 
    let signingKey = compositeSigningKey consumerSecret ""

    let queryParameters = 
        ["oauth_callback", "oob";
         "oauth_consumer_key", consumerKey;
         "oauth_nonce", System.Guid.NewGuid().ToString().Substring(24);
         "oauth_signature_method", "HMAC-SHA1";
         "oauth_timestamp", currentUnixTime().ToString();
         "oauth_version", "1.0"]

    let signingString = baseString "POST" requestTokenURI queryParameters
    let oauth_signature = hmacsha1 signingKey signingString

    let realQueryParameters = ("oauth_signature", oauth_signature)::queryParameters

    let req = WebRequest.Create(requestTokenURI, Method="POST")
    let headerValue = createAuthorizeHeader realQueryParameters
    req.Headers.Add(HttpRequestHeader.Authorization, headerValue)
    
    let resp = req.GetResponse()
    let stream = resp.GetResponseStream()
    let txt = (new StreamReader(stream)).ReadToEnd()
    
    let parts = txt.Split('&')
    (parts.[0].Split('=').[1],
     parts.[1].Split('=').[1],
     parts.[2].Split('=').[1] = "true")

/// Get an access token from Twitter and returns:
///   oauth_token, oauth_token_secret
let accessToken token tokenSecret verifier =
    let signingKey = compositeSigningKey consumerSecret tokenSecret

    let queryParameters = 
        ["oauth_consumer_key", consumerKey;
         "oauth_nonce", System.Guid.NewGuid().ToString().Substring(24);
         "oauth_signature_method", "HMAC-SHA1";
         "oauth_token", token;
         "oauth_timestamp", currentUnixTime().ToString();
         "oauth_verifier", verifier;
         "oauth_version", "1.0"]

    let signingString = baseString "POST" accessTokenURI queryParameters
    let oauth_signature = hmacsha1 signingKey signingString
    
    let realQueryParameters = ("oauth_signature", oauth_signature)::queryParameters
    
    let req = WebRequest.Create(accessTokenURI, Method="POST")
    let headerValue = createAuthorizeHeader realQueryParameters
    req.Headers.Add(HttpRequestHeader.Authorization, headerValue)
    
    let resp = req.GetResponse()
    let stream = resp.GetResponseStream()
    let txt = (new StreamReader(stream)).ReadToEnd()
    
    let parts = txt.Split('&')
    (parts.[0].Split('=').[1],
     parts.[1].Split('=').[1])

/// Compute the 'Authorization' header for the given request data
let authHeaderAfterAuthenticated url httpMethod token tokenSecret queryParams = 
    let signingKey = compositeSigningKey consumerSecret tokenSecret

    let queryParameters = 
            ["oauth_consumer_key", consumerKey;
             "oauth_nonce", System.Guid.NewGuid().ToString().Substring(24);
             "oauth_signature_method", "HMAC-SHA1";
             "oauth_token", token;
             "oauth_timestamp", currentUnixTime().ToString();
             "oauth_version", "1.0"]

    let signingQueryParameters = 
        List.append queryParameters queryParams

    let signingString = baseString httpMethod url signingQueryParameters
    let oauth_signature = hmacsha1 signingKey signingString
    let realQueryParameters = ("oauth_signature", oauth_signature)::queryParameters
    let headerValue = createAuthorizeHeader realQueryParameters
    headerValue

/// Add an Authorization header to an existing WebRequest 
let addAuthHeaderForUser (webRequest : WebRequest) token tokenSecret queryParams = 
    let url = webRequest.RequestUri.ToString()
    let httpMethod = webRequest.Method
    let header = authHeaderAfterAuthenticated url httpMethod token tokenSecret queryParams
    webRequest.Headers.Add(HttpRequestHeader.Authorization, header)

type System.Net.WebRequest with
    /// Add an Authorization header to the WebRequest for the provided user authorization tokens and query parameters
    member this.AddOAuthHeader(userToken, userTokenSecret, queryParams) =
        addAuthHeaderForUser this userToken userTokenSecret queryParams




let testing() =   

    // Compute URL to send user to to allow our app to connect with their credentials,
    // then open the browser to have them accept
    let oauth_token'', oauth_token_secret'', oauth_callback_confirmed = requestToken()
    let url = authorizeURI + "?oauth_token=" + oauth_token''
    System.Diagnostics.Process.Start("iexplore.exe", url)
    
    // *******NOTE********:
    // Get the 7 digit number from the web page, pass it to the function below to get oauth_token
    // Sample result if things go okay:
    //    val oauth_token_secret' : string = "lbll17CpNUlSt0FbfMKyfrzBwyHUrRrY8Ge2rEhs"
    //    val oauth_token' : string = "17033946-8vqDO7foX0TUNpqNg9MyJpO3Qui3nunkZPixxLs"
    let oauth_token, oauth_token_secret = accessToken oauth_token'' oauth_token_secret'' ("1579754")


    // Test 1:
    let streamSampleUrl2 = "https://api.twitter.com/1/statuses/home_timeline.xml"
    let req = WebRequest.Create(streamSampleUrl2) 
    req.AddOAuthHeader(oauth_token, oauth_token_secret, [])
    let resp = req.GetResponse()
    let strm = resp.GetResponseStream()
    let text = (new StreamReader(strm)).ReadToEnd()
    text

    // Test 2:
    System.Net.ServicePointManager.Expect100Continue <- false
    let statusUrl = "https://twitter.com/statuses/update.xml"
    let request = WebRequest.Create (statusUrl, Method="POST")
    let tweet =  urlEncode("Hello!")
    request.AddOAuthHeader(oauth_token,oauth_token_secret,["status",tweet])
    let bodyStream = request.GetRequestStream()
    let bodyWriter = new StreamWriter(bodyStream)
    bodyWriter.Write("status=" + tweet)
    bodyWriter.Close()
    let resp = request.GetResponse()
    let strm = resp.GetResponseStream()
    let text = (new StreamReader(strm)).ReadToEnd()
    text

• F# enables exciting new developer scenarios in Visual Studio (explorative bottom-up programming in the F# Interactive)
• F# can leverage many/most of the powerful developer tools in Visual Studio (projects, debugging, editing, help integration, etc.) and .NET (produce and consume vanilla .NET APIs)
• F# has unique strengths in key target domains and extends the .NET platform to these audiences
• F# scales smoothly from explorative to .NET component development (start with ideas and iteratively and smoothly add structure as methods, types and projects)

The video of the talk is available now on TechEd Online and here is the demo code below for those interested in taking a look. 

TechEd2010Demos.zip

Those who read my blog may recognize a number of the examples as things I've posted about in the past.   The F# Silverlight apps used in the demos are also available in those previous posts.

There was one theme that seemed to resonate especially well - here's a few more thoughts on the idea of scaling from explorative to component building in F# from the perspective of using the "tab" refactoring in F#.

"tab" as Extract Method (or, more generally, Encapsulate)

Indentation is meaningful in F#, and is used to indicate nesting.  This, along with type inference and a strong uniformity of syntax at different levels of nesting, enables a simple but really fundamental shift to how you can think about encapsulation in F#. 

An example. I start by solving a problem by writing code directly at the top level.  I don't know what shape the solution has to take yet, but I leverage the F# Interactive to prototype and explore the solution.

let url = "https://www.yahoo.com"
let req    = WebRequest.Create(Uri(url))
let stream = req.GetResponse().GetResponseStream()
let reader = new StreamReader(stream)
let html   = reader.ReadToEnd()

html.Length

Notice that because we're in the F# Interactive, we tend to have some sample data we want to experiment with bound in our environment - like "url" above.

Once we establish that this code does what we want - we can "give it a name" - using a simple and local set of changes:

let url = "https://www.yahoo.com"

let http url = 
    let req    = WebRequest.Create(Uri(url))
    let stream = req.GetResponse().GetResponseStream()
    let reader = new StreamReader(stream)
    let html   = reader.ReadToEnd()
    html

let html = http url
html.Length

We just hit "tab", and then wrote out the signature for the new function.  Note that we were able to "parameterize over" any of the names used in this piece of code that we wanted to be variable in our abstraction.  So in this case, we parameterize over "url".  Any later dependencies on the results of the original code can be left intact by re-binding the name "html" to a call to the new function - thereby making this a truly local code change.

Notice that a key to this was the uniformity of "let" between the top level and the function body scope. 

Later in my coding I may end up with something like this:

let username = "abc"
let password = "def"
    
let http url = 
    let credentials = NetworkCredential(username, password)
    let req    = WebRequest.Create(Uri(url), Credentials=credentials)
    let stream = req.GetResponse().GetResponseStream()
    let reader = new StreamReader(stream)
    let html   = reader.ReadToEnd()
    html
    
let homeTimeline() = http "https://api.twitter.com/statuses/home_timeline.xml"
let publicTimeline() = http "https://api.twitter.com/statuses/public_timeline.xml"
let userTimeline() = http "https://api.twitter.com/statuses/user_timeline.xml"

At this point, I have some interesting functionality - and may want to introduce a true component to encapsulate the implementation and expose an intentionally designed API.  OO is an excellent and proven tool for component design, and is a key part of F# programming.  It is also very easy to smoothly transition from simple top level coding into F# component design with OO.  And again - it's mostly just about "tab".

let username = "abc"
let password = "def"

type Twitter(username, password) =     
    let http url = 
        let credentials = NetworkCredential(username, password)
        let req    = WebRequest.Create(Uri(url), Credentials=credentials)
        let stream = req.GetResponse().GetResponseStream()
        let reader = new StreamReader(stream)
        let html   = reader.ReadToEnd()
        html
    
    let homeTimeline() = http "https://api.twitter.com/statuses/home_timeline.xml"
    let userTimeline() = http "https://api.twitter.com/statuses/user_timeline.xml"
    let publicTimeline() = http "https://api.twitter.com/statuses/public_timeline.xml"
    
    member this.HomeTimeline() = homeTimeline()
    member this.UserTimeline() = userTimeline()
    member this.PublicTimeline() = publicTimeline()

let twitter = new Twitter(username, password)
let homeTimeline = twitter.HomeTimeline()
let userTimeline = twitter.UserTimeline()
let publicTimeline = twitter.PublicTimeline()

Notice how similar this transition was to the change we needed to give the "http" function a name.  In this case, we just tab in all the code except the things we want to parameterize over, give the signature of the type and it's constructor (again, the things we want to parameterize over), and then define the API we want to expose from the functionality as "member" definitions.

So we've now introduced a true .NET class with the API we want.  The last step is to transition from the explorative environment to a component.  We can just take the code above - strip off the sample data that was just there to experiment in the F# Interactive - and compile into a .NET assembly:

namespace TwitterAPI

open System
open System.Net
open System.IO

type Twitter(username : string, password : string) =     
    let http url = 
        let credentials = NetworkCredential(username, password)
        let req    = WebRequest.Create(Uri(url), Credentials=credentials)
        let stream = req.GetResponse().GetResponseStream()
        let reader = new StreamReader(stream)
        let html   = reader.ReadToEnd()
        html
    
    let homeTimeline() = http "https://api.twitter.com/statuses/home_timeline.xml"
    let userTimeline() = http "https://api.twitter.com/statuses/user_timeline.xml"
    let publicTimeline() = http "https://api.twitter.com/statuses/public_timeline.xml"
    
    member this.HomeTimeline() = homeTimeline()
    member this.UserTimeline() = userTimeline()
    member this.PublicTimeline() = publicTimeline()

And then we can use this from C# (or VB, or any other .NET language):

using System;
using TwitterAPI;

class Program
{
    static void Main(string[] args)
    {
        Twitter twitter = new Twitter("abc", "def");
        string homeTimeline = twitter.HomeTimeline();
        Console.WriteLine(homeTimeline);
    }
}

Summary

The TechEd talk covered a number of interesting aspects of using F# in Visual Studio - tools, core syntax, some examples of current commercial users, and the physical structure of F# projects used in client applications.  But the theme of scaling from explorative programming to .NET component is one that I find particularly powerful, and which is a really unique strength of the F# language.

TechEd2010Demos.zip

Last November at PDC 2009 in Los Angeles I gave a talk on F# for Parallel and Asynchronous Programming. 

• Video of the talk is available online at https://microsoftpdc.com/Sessions/FT20. 
• Slides and demo content are on Code Gallery at PDC2009 Demo Code- F# for Parallel And Asynchronous Programming.

The talk begins by covering basic F# concepts, and then focuses on four challenging issues related to concurrency and the tools F# brings for addressing these - immutability, async workflows, and agents.  In trying to cover this ground in a 60 minute talk, there were naturally a lot of relevant topics I had to skip, and I also got a number of questions about things I'd glossed over after the talk.  So I thought I'd use this post to discuss some of the other features and topics I would have highlighted in this presentation if there'd been more time.
 

Note:  The topics covered below mostly assume you've either watched the session video or taken a look through the source code attached above.  If you haven't yet - check out the talk.

Some F# Parallel and Async Topics

PSeq

The talk briefly shows the use of a PSeq module to write code like the following:

let rec sumOfSquares nums =     
    nums    
    |> PSeq.map sqr    
    |> PSeq.sum

As a few folks have noted, the F# core library does not actually provide this PSeq module.  In the demo code, I was just using my own little implementation of this, to show off the benefits of the more declarative programming style used here. 

The PSeq I used is just a very simple wrapper around the functionality provided in PLinq - for example, PSeq.map just calls ParallelEnumerable.Select.  We are expecting a future release of the F# PowerPack to include this wrapper library.  In the the meantime, you can check out Matt Podwysocki's blog post on using PLinq from F#, or Talbot Crowell's recent post with an updated Beta2 version.

Using Begin/End Methods with F# Async

In showing the ease of turning synchronous code which calls long-running I/O operations into asynchronous code, I showed adding the pieces highlighed in red below to the code below to make it async:

let downloadImageAsync(blob : CloudBlob) =
  async {
    let! pixels = blob.AsyncDownloadByteArray()
    let fileName = "thumbs-" + blob.Uri.Segments.[blob.Uri.Segments.Length-1]
    use outStream =  File.OpenWrite(fileName)
    do! outStream.AsyncWrite(pixels, 0, pixels.Length)
    return fileName  }

I got a very good question after the talk though: "Where did the AsyncDownloadByteArray function come from?" 

In general, for any asynchronous API provided on the platform, it can be simply wrapped and exposed as a method returning Async<T> to be used within F# async workflows as above.  In the core F# libraries, we wrap many of the core async APIs in .NET Framework, providing extension methods on the relevant types.  This means that functions like Stream.AsyncWrite used above are available by default in F#.  However, CloudBlob.DownloadByteArray is defined in the libraries provided with the Windows Azure SDK, and so we need to create a wrapper.

In fact, the Azure SDK doesn't even provide a BeginDownloadByteArray/EndDownloadByteArray pair - but it does provide a more fundamental async operation BeginDownloadToStream/EndDownloadToStream:

public IAsyncResult BeginDownloadToStream(Stream target, AsyncCallback callback, object state);

public void EndDownloadToStream(IAsyncResult asyncResult);

Notably, it's likely that the reason for not including the higher level async operations is just due to the substantial complexity of authoring and maintaining these methods in C# today. Had the Azure team written their API in F#, it's likely they would have found it sufficiently easy to add these.

So, what does the wrapper look like?

type CloudBlob with
     /// Asynchronously download blob contents as an array
    member blob.AsyncDownloadByteArray() = async {
        let ms = new MemoryStream(int blob.Properties.Length)
        do! Async.FromBeginEnd(
                    (fun (cb,s) -> blob.BeginDownloadToStream(ms,cb,s)),
                     blob.EndDownloadToStream)
        return ms.ToArray() }

Some notes on this implementation:

Async.FromBeginEnd

The key tool for wrapping many .NET async APIs is to use Async.FromBeginEnd.  This method takes two functions as arguments - the Begin/End pair of the method being wrapped - and returns a function of type Async<T> where T is the type returned from the End method. When directly wrapping a method, this is often trivial - member foo.AsyncBar() = Async.FromBeginEnd(foo.BeginBar, foo.EndBar).

Composing Async Operations

There are generally two ways to create Async<T> values - call a function which returns one, like Async.FromBeginEnd, Stream.AsyncWrite, or Async.Parallel - or write an async {.}  block to include any arbitrary F# code as part of the composition.  In the case above, we need to both call the underlying Begin/End pair, but also construct a stream to download into, and ultimately return the array of bytes left in the resulting stream.  This composition of asynchronous operations with some synchronous code is easy to do inside the async {.} block.

Other Async Interop Utilities

We saw details above of wrapping Begin/End pairs as Async<T> objects - there are a few other interesting interop scenarios which are nicely supported by the Async API in F#:

• Async.AsBeginEnd - coverts an async operation into a set of Begin/End/Cancel functions which can be used to expose an implementation of the .NET Asynchronous Programming Model to other .NET consumers.
  
• Async.AwaitEvent/AwaitWaitHandle/AwaitIAsyncResult/AwaitTask - there are many other async models used in .NET beyond the APM, and F# provides primitives for easily wrapping any of these.

Exceptions with Async

In the talk, I commented on how hard it is to do correct exception handling with typical asynchronous programming tools available today in C#/VB.  But I didn't actually show how much easier this is with F# async workflows.  Here's what it looks like to make our downloadImageAsync function robust to the many exceptions that could possibly be raised during it's execution:

let downloadImageAsync(blob : CloudBlob) =  async {
    try
        let! pixels = blob.AsyncDownloadByteArray()
        let fileName = "thumbs-" + blob.Uri.Segments.[blob.Uri.Segments.Length-1]
        use outStream =  File.OpenWrite(fileName)
        do! outStream.AsyncWrite(pixels, 0, pixels.Length)
        return fileName
    with
    | e ->
        Log.Write(e.ToString())
        return ""  }

The key here is that exception handling is no different for async code than for synchronous code.  Under the hood, F# will ensure that exceptions occurring anywhere in the workflow are bubbled up through the non-blocking calls and ultimately handled by the exception handler code.  This will happen as expected even when the code executes on multiple different threads during it's operation.

Cancellation with Async

When we have long-running operations in our programs, as well as making them non-blocking, we very often want to make them cancellable.  F# async workflows support cancellation automatically, inserting cancellation checks at each let! or do!, and also at each iteration of a loop.  Cancellation is tracked and triggered using the new unified cancellation model in the .NET Framework, "System.Threading.CancellationToken".  For example, if we want to make downloadImageAsync cancellable, we simply need to pass in a cancellation token when we start the workflow, and then hook up a button to cancel a running operation.  The deltas to the PhotoViewer application are indicated in red below:

F# code:

member this.DownloadAll(cancellationToken) =
    let work =
        container.ListBlobs()
        |> Seq.map (fun blob ->
            downloadImageAsync(container.GetBlobReference(blob.Uri.ToString())))
        |> Async.Parallel
    Async.StartAsTask(work, cancellationToken = cancellationToken) 

C# client code:

CancellationTokenSource cancellableWork;

private void Start_Click(object sender, RoutedEventArgs e){
    cancellableWork = new CancellationTokenSource();
    downloader.DownloadAll(cancellableWork.Token);
}

private void Cancel_Click(object sender, RoutedEventArgs e){
    if (cancellableWork != null)
        cancellableWork.Cancel();
}

Other Great F# Async Samples

Don has recently posted two great articles on some other end-to-end samples built on F# Async.  These posts cover many of the same core F# Async topics, plus some additional topics not touched on here.  The samples are also a lot of fun, using data pulled from Twitter and Bing Translation services.

• Async and Parallel Design Patterns in F# Part1: Parallelizing CPU and I/O Computations
• Async and Parallel Design Patterns in F# Part 2: Reporting Progress with Events (plus Twitter Sample)

Summary

F#'s immutability, async workflows and agents are useful features for enabling easier parallel and asynchronous programming in application code.  The PDC talk provides a good starting point for understanding how these features can be applied.  The topics above cover a few of the related topics that I've been asked about since then, but there is also much, much more that can be enabled on top of these core features.

Last year, I posted an F# implementation of the 2006 ICFP contest problem, which was an amazing and complex set of puzzles inside a custom virtual machine.   Here’s a Silverlight version of that virtual machine, implemented on top of the Console sample I posted yesterday.

Silverlight UM – ICFP 2006

See https://boundvariable.org/ and my previous post for details of what’s running in this console.

Launch the UM

 /// Start the UM running on a background thread.
 /// The input and output functions will be called on the 
 /// background thread, and block the execution of the UM
 static member Launch(binaryUri : Uri, input : Stream, output : Stream) =
     async {
         try
 let writer = new System.IO.StreamWriter(output, AutoFlush=true)
             let reader = new System.IO.StreamReader(input)
             do writer.WriteLine("Press <enter> to begin. Note: Will download a large UMIX image.")
             do reader.ReadLine() |> ignore
             let request = System.Net.WebRequest.Create(binaryUri)
             do writer.WriteLine(binaryUri)
             let! response = request.AsyncGetResponse()
             let stream = response.GetResponseStream()
             do writer.WriteLine("Downloading...")
             let progress(percent) = writer.WriteLine("Downloaded: "+percent.ToString()+"%")
             let! bytes = asyncReadAllBytesFromStreamWithProgress (stream, int response.ContentLength, progress) 
 do writer.WriteLine("Booting...")
             let ee = UM(bytes, Func<int>(input.ReadByte), Action<byte>(output.WriteByte))
             ee.Run()
         with
 | e -> System.Diagnostics.Debug.WriteLine("UM failed with {0}", e)
     } |> Async.Start     

F# Async in Silverlight

In Silverlight, the APIs for synchronous I/O have been removed, which forces developers to do the ‘right thing’ and make async calls for long-running I/O operations, thus (hopefully) keeping the UI responsive.  This makes F# a really nice implementation language for Silverlight applications, using F#’s async workflows.  In the code above, an asynchronous workflow describes the flow of control of launching the UM.  At the two points of long running network connections, async calls are made with “let!”, so that the UI thread is not blocked and the application remains responsive.  This ultimately hides a lot of complexity that would be added in chaining these asynchronous operations together in a standard .NET approach to this.

Note also that the code above calls a helper function “asyncReadAllBytesFromStreamWithProgress”, which shows how async code in F# can be nicely factored using the same sort of “extract method” factoring you are used to in synchronous programming.  The implementation of this helper also shows that async calls with ‘let!’ can be placed inside ‘for’ and ‘while’ loops.  Chances are you don’t even want to try doing that with standard Begin/End calls!

let internal asyncReadAllBytesFromStreamWithProgress(stream:Stream, length:int, progress:int->unit) =
    async {
        let offset = ref 0
        let count = ref length
        let buffer = Array.zeroCreate<byte> !count
        while !count > 0 do
 let! bytesRead = stream.AsyncRead(buffer, !offset, (min 524288 !count))
            if bytesRead = 0 then raise (new EndOfStreamException("Read beyond the EOF"))
            do offset := !offset + bytesRead
            do count := !count - bytesRead
            progress(100 * !offset / length)
        return buffer
    }

Consuming F# from C#+XAML

The majority of this app is written in F# – the console control is authored in F# and the application logic is in F#.  Here is the C# and XAML that ties it together:

<UserControl xmlns:my="clr-namespace:System.Windows.Controls;assembly=SilverlightConsole" x:Class="UM.Page"
 xmlns=https://schemas.microsoft.com/winfx/2006/xaml/presentation
 xmlns:x=https://schemas.microsoft.com/winfx/2006/xaml
 Width="600" Height="400">
 <Grid x:Name="LayoutRoot" Background="White">
 <my:SilverlightConsole x:Name="console" Width="600" Height="400"
 FontFamily="Courier New" FontSize="13" FontWeight="Bold"
 Foreground="GreenYellow" Background="Black" Text="" />
 </Grid>
</UserControl>

public partial class Page : UserControl {
    public Page() {
        InitializeComponent();
        Uri curPage = System.Windows.Application.Current.Host.Source;
        UriBuilder builder = new UriBuilder(curPage.Scheme, curPage.Host, curPage.Port, "umix.dll");
        ICFP.UM.Launch(builder.Uri, console.InputStream, console.OutputStream);
    }
}

Summary

Check out ICFP Programming Contest  2009 this weekend, and take F#+Async+Silverlight for a spin.

With the recent F# May CTP though, we now provide a Silverlight version of the F# runtime, FSharp.Core.dll, along with the F# release.  This enables building truly first-class Silverlight components using F#.

To make this easier, I’ve posted some Silverlight F# project templates and samples on Code Gallery. 

Download

F# Templates and Samples for Silverlight

Templates

Samples

L-Systems

Lindenmayer Systems are an interesting way of generating a variety of fractals using a simple set of rewrite rules.  Check out the fascinating book The Algorithmic Beauty of Plants for details.  The Silverlight application below uses an L-System rewriter and rendered written in F#.

• F, G, A, B are drawn as a move forward.
• + is a turn right.
• - is a turn right.
• X, Yand anything else is skipped during rendering.

Launch the L-System demo 

open System.Windows
open System.Windows.Shapes

let rec internal applyRulesInOrder rules c =
    match rules with
    | [] -> string c
    | rule::rules' -> match rule c with | None -> applyRulesInOrder rules' c
        | Some result -> result

let internal step rules current =
     current
     |> String.collect (applyRulesInOrder rules)

let internal rotate (x,y) theta =
    let x' = x * cos theta - y * sin theta
    let y' = x * sin theta + y * cos theta
    (x',y')

let rec internal render (x,y) (dx,dy) angle points system =
    match system with | [] -> (x,-y)::points
    | 'A'::system' | 'B'::system' | 'F'::system' | 'G'::system' -> let x',y' = x+dx,y+dy
        render (x',y') (dx,dy) angle ((x,-y)::points)  system'
    | '+'::system' -> let (dx',dy') = rotate (dx,dy) angle
        render (x,y) (dx',dy') angle points system'
    | '-'::system' -> let (dx',dy') = rotate (dx,dy) (-angle)
        render (x,y) (dx',dy') angle points system'
    | _::system' -> render (x,y) (dx,dy) angle points system' let rec internal applyN f n x =
    if n = 0 then x
    else f (applyN f (n-1) x)

let internal normalize points =
    let minX = points |> Seq.map (fun (x,_) -> x) |> Seq.min
    let minY = points |> Seq.map (fun (_,y) -> y) |> Seq.min
    points |> List.map (fun (x,y) -> new Point(x-minX, y-minY)) type LSystem(rulesString:string, start:string, angle:int, stepSize:int, n:int) =
    let expanded,isError =
        try let rules =
                rulesString.Split([|"\r";"\n"|], System.StringSplitOptions.RemoveEmptyEntries)
                |> Array.map (fun line -> line.Split([|"->"|], System.StringSplitOptions.RemoveEmptyEntries))
                |> Array.map (fun fromAndTo -> (fromAndTo.[0].[0], fromAndTo.[1]))
            let ruleFunctions = [ for (c, s) in rules -> fun x -> if x = c then Some s else None]
            applyN (step ruleFunctions) n start, false with | e -> "", true member this.Render(polyline : Polyline) =
        let points = render (0.0,0.0) (float stepSize,0.0) (float angle * System.Math.PI / 180.0) [] (List.ofSeq expanded)
        for pt in normalize points do polyline.Points.Add(pt)
        isError

        

Console Control

A resuable Silverlight control providing a console emulation. The control exposes input and ouput streams akin to those on the System.Console class. Could be used to provide console input and output as part of a Silverlight application, or as a way to convert Windows Console apps to Silverlight apps.

This samples hooks the Console up to a simple echo loop.

Launch the Console Control

namespace System.Windows.Controls

open System
open System.IO
open System.Windows
open System.Windows.Controls
open System.Windows.Input
open SilverlightContrib.Utilities.ClipboardHelper
open System.Text

// A shared base implementation of Stream for 
// use by the console input and output streams
[<AbstractClass>] type private ConsoleStream(isRead) = 
    inherit Stream()
    override this.CanRead = isRead
    override this.CanWrite = not isRead
    override this.CanSeek = false override this.Position
        with get() = raise (new NotSupportedException("Console stream does not have a position"))
        and set(v) = raise (new NotSupportedException("Console stream does not have a position"))
    override this.Length = raise (new NotSupportedException("Console stream does not have a length"))
    override this.Flush() = ()
    override this.Seek(offset, origin) = raise (new NotSupportedException("Console stream cannot seek"))
    override this.SetLength(v) = raise (new NotSupportedException("Console stream does not have a length")) 

/// A control representing a Console window 
/// Provides an InputStream and OutputStream 
/// for reading an writing character input. 
/// Also supports copy/paste on some browsers
type SilverlightConsole() as self = 
    inherit TextBox()

    // The queue of user input which has been collected by the 
 // console, but not yet read from the input stream 
 let readQueue = new System.Collections.Generic.Queue<int>()

    // A stream that reads characters from user input 
 let inputStream = 
        { new ConsoleStream(true) with override this.Write(buffer,offset,count) = 
                raise (new NotSupportedException("Cannot write from Console input stream"))
            override this.Read(buffer,offset,count) = 
                do System.Diagnostics.Debug.WriteLine("Starting to read {0} bytes", count)
                let rec waitForAtLeastOneByte() =
                    let shouldSleep = ref true let ret = ref [||]
                    lock readQueue  (fun () -> shouldSleep := readQueue.Count < 1
                        if not !shouldSleep then let lengthToRead = min readQueue.Count count
                            ret := Array.init lengthToRead (fun i -> byte (readQueue.Dequeue())))
                    if !shouldSleep
                    then System.Threading.Thread.Sleep(100); waitForAtLeastOneByte()
                    else !ret
                let bytes = waitForAtLeastOneByte()
                System.Array.Copy(bytes, 0, buffer, offset, bytes.Length)
                do System.Diagnostics.Debug.WriteLine("Finished reading {0} bytes", bytes.Length)
                bytes.Length
        }

    // A stream that sends character output onto the console screen 
 let outputStream =
        { new ConsoleStream(false) with override this.Read(buffer,offset,count) = 
                raise (new NotSupportedException("Cannot read from Console output stream"))
            override this.Write(buffer,offset,count) = 
                let isDelete = offset < 0
                let newText =
                     if isDelete
                     then "" else UnicodeEncoding.UTF8.GetString(buffer, offset, count)
                let _ = self.Dispatcher.BeginInvoke(fun () -> if isDelete then if self.Text.Length >= count then self.Text <- self.Text.Substring(0, self.Text.Length - count)
                    else do self.Text <- self.Text + newText
                    do self.SelectionStart <- self.Text.Length
                    do self.SelectionLength <- 0)
                ()
        }

    let shiftNumbers = [|')';'!';'@';'#';'$';'%';'^';'&';'*';'('|]
    let currentInputLine = new System.Collections.Generic.List<int>()

    // Handles key down events 
 // Processes the pressed key and turns it into console input 
 // Also echos the pressed key to the console 
 let keyDownHandler(keyArgs : KeyEventArgs) =
        try do keyArgs.Handled <- true let shiftDown = Keyboard.Modifiers &&& ModifierKeys.Shift <> (enum 0)
            let ctrlDown = Keyboard.Modifiers &&& ModifierKeys.Control <> (enum 0)
            let p = keyArgs.PlatformKeyCode
            if ctrlDown || keyArgs.Key = Key.Ctrl then if keyArgs.Key = Key.V then lock currentInputLine (fun () -> let clipboard = new ClipboardHelper()
                        let fromClipboard = clipboard.GetData()
                        for c in fromClipboard do do currentInputLine.Add(int c)
                            outputStream.WriteByte(byte c)
                            if c = '\n' then for i in currentInputLine do do System.Diagnostics.Debug.WriteLine("Enqueued {0}", char i)
                                    do readQueue.Enqueue(i)
                                do currentInputLine.Clear()
                    )
                elif keyArgs.Key = Key.C then let text = self.SelectedText
                    let clipboard = new ClipboardHelper()
                    clipboard.SetData(text)
            else System.Diagnostics.Debug.WriteLine("Got key {0} {1} {2}", p, char p, keyArgs.Key)
                let ascii =
                    match p with | n when n >= 65 && n <= 90 -> if shiftDown then p else p+32
                    | n when n >= 48 && n <= 57 -> if shiftDown then int shiftNumbers.[p-48] else p
                    | 8 -> 8 
 // backspace 
                    | 13 -> int '\n' 
                    | 32 -> int ' ' 
                    | 186 -> if shiftDown then int ':' else int ';' 
                    | 187 -> if shiftDown then int '+' else int '=' 
                    | 188 -> if shiftDown then int '<' else int ',' 
                    | 189 -> if shiftDown then int '_' else int '-' 
                    | 190 -> if shiftDown then int '>' else int '.' 
                    | 191 -> if shiftDown then int '?' else int '/' 
                    | 192 -> if shiftDown then int '~' else int '`' 
                    | 219 -> if shiftDown then int '{' else int '[' 
                    | 220 -> if shiftDown then int '|' else int '\\' 
                    | 221 -> if shiftDown then int '}' else int ']' 
                    | 222 -> if shiftDown then int '\"' else int '\'' 
                    | _ -> -1
                if ascii = 8 then lock currentInputLine (fun () -> if currentInputLine.Count > 0 then currentInputLine.RemoveAt(currentInputLine.Count - 1)
                        outputStream.Write([||], -1, 1)
                    )
                elif ascii > 0 then lock currentInputLine (fun () -> do currentInputLine.Add(ascii)
                        outputStream.WriteByte(byte ascii)
                    )
                if ascii = int '\n' then lock currentInputLine (fun () -> for i in currentInputLine do do System.Diagnostics.Debug.WriteLine("Enqueued {0}", char i)
                            if i = 10 then do readQueue.Enqueue(13)
                            do readQueue.Enqueue(i)
                        do currentInputLine.Clear())
                do self.SelectionStart <- self.Text.Length
                do self.SelectionLength <- 0
        with | e -> System.Diagnostics.Debug.WriteLine(e)

    // Lazily initialized StreamReader/StreamWriter 
 let outReader = lazy (new System.IO.StreamWriter(outputStream, Encoding.UTF8, 256, AutoFlush=true))
    let inReader = lazy (new System.IO.StreamReader(inputStream, Encoding.UTF8, false, 256))

    // Manually handle the Return key so we can accept newlines 
 do self.AcceptsReturn <- true 
 // Make sure a vertical scrollbar appears when needed 
 do self.VerticalScrollBarVisibility <- ScrollBarVisibility.Auto
    // Make the control read-only so that users cannot move the cusor or change the contents 
 // Unfortunatley, this also greys it out - ideally we could seperate theese two. 
 do self.IsReadOnly <- true 
 // Hookup the keyDownHandler 
 do self.KeyDown.Add(keyDownHandler)

    /// The raw input stream for the Console 
 member this.InputStream = inputStream :> Stream
    /// The raw ouput stream for the Console 
 member this.OutputStream = outputStream :> Stream

    /// A StreamWriter for writing to the Console 
 member this.Out = outReader.Value
    /// A StreamReader for reading from the Console 
 member this.In = inReader.Value

Summary

Try out F# with Silverlight using the F# May CTP and the F# for Silverlight templates.

Azure supports running .NET applications, which means you can build Azure worker roles using F#! The tools released with Azure don’t have F# support out of the box though, so I’ve posted a few simple templates and samples up on Code Gallery.

Download

F# Templates and Samples for Windows Azure

Templates

Cloud WebCrawl Sample

namespace SearchEngine_WorkerRole

open System
open System.Threading
open Microsoft.ServiceHosting.ServiceRuntime
open System.Net
open System.IO
open System.Text.RegularExpressions
open Microsoft.Samples.ServiceHosting.StorageClient;
open System.Web
open System.Runtime.Serialization.Formatters.Binary

type WorkerRole() =
    inherit RoleEntryPoint()

    // The page to start crawling from
    let startpage = @"https://blogs.msdn.com/lukeh"
    // The filter to apply to links while crawling
    let pageFilter = fun (url:string) -> url.StartsWith("https://blogs.msdn.com/")

    /// Get the contents of a given url
    let http(url: string) = 
        let req    = WebRequest.Create(url) 
        use resp   = req.GetResponse()
        use stream = resp.GetResponseStream() 
        use reader = new StreamReader(stream) 
        let html   = reader.ReadToEnd()
        html

    /// Get the links from a page of HTML
    let linkPat = "href=\s*\"[^\"h]*(https://[^&\"]*)\""
    let getLinks text =  [ for m in Regex.Matches(text,linkPat)  -> m.Groups.Item(1).Value ]
    
    /// Handle the message msg using the given queue and blob container
    let HandleMessage (msg : Message) (queue : MessageQueue, container: BlobContainer) =
        // There was a new item, get the contents
        let url = msg.ContentAsString();
        let urlBlobName = HttpUtility.UrlEncode(url)
        // Don't get the page if we've already seen it
        if not(container.DoesBlobExist(urlBlobName)) 
        then
            do RoleManager.WriteToLog("Information", String.Format("Handling new url: '{0}'", url));
            try
                // Get the contents of the page
                let content = http url
                // Store the page into the blob store
                let props = new BlobProperties(urlBlobName)
                let _ = container.CreateBlob(props, new BlobContents(System.Text.UTF8Encoding.Default.GetBytes(content)), true);
                
                // Get the links from the page
                let links = getLinks content
                
                // Filter down the links and then create a new work item for each
                links
                |> Seq.filter pageFilter
                |> Seq.distinct
                |> Seq.filter (fun link -> not(container.DoesBlobExist(HttpUtility.UrlEncode(link))))
                |> Seq.iter (fun link -> queue.PutMessage(new Message(link)) |> ignore)
                queue.DeleteMessage(msg) |> ignore
            with
            | _ ->()
    
    /// Main loop of worker process
    let rec Loop (queue : MessageQueue, container: BlobContainer) = 
        // Get the next page to crawl from the queue
        let msg = queue.GetMessage(240);
        if msg = null
        then Thread.Sleep(1000)
        else HandleMessage msg (queue, container)
        Loop(queue,container)
    
    override wp.Start() =
        // Initialize the Blob storage
        let blobStorage = BlobStorage.Create(StorageAccountInfo.GetDefaultBlobStorageAccountFromConfiguration());
        let container = blobStorage.GetBlobContainer("searchengine");
        let a = container.CreateContainer(null, ContainerAccessControl.Public);

        // Initialize the Queue storage
        let queueStorage = QueueStorage.Create(StorageAccountInfo.GetDefaultQueueStorageAccountFromConfiguration());
        let queue = queueStorage.GetQueue("searchworker");
        let b = queue.CreateQueue()
        
        // Put an initial message in the queue, using the start page
        let c = queue.PutMessage(new Message(startpage));
        
        // Begin the main loop, processing messages in the queue
        Loop(queue, container)
        
    override wp.GetHealthStatus() = RoleStatus.Healthy 

Worker Roles

The code above defines the implementation of a Worker Role – a process which runs in the background, waiting for work to do, and then processing these work requests.  The worker role is set to run 4 instance simultaneously, which means that there will be 4 instances of this worker processing work items as they come in.  This gives an implicit parallelism – in fact, the initial release of Azure will run one process per core, so you really are getting effective parallelism this way.  Notice also that this requires that the worker processes are inherently stateless.  Both aspects make typical functional design approaches that are common in F# natural for developing these worker roles.

Queues and Blobs

This sample uses two of the three data formats supported by Windows Azure.  The queue storage holds the work items. The blob storage holds the pages visited during the web crawl.  When an instance of the worker role Starts, it connects to the blob store and the queue store, then puts an initial work item in the queue and goes into a loop processing work items out of the queue.

Conclusion

The following isn’t exactly how the conversation went - but roughly captures what we talked about, and covers a lot of fun F# topics.  There’s some allusions here to ideas from statistics, time-series analysis, CEP and FRP.

“How would you compute standard deviation over a stream of data in F#?”

Standard deviation is a pretty straightforward computation with F#.  There’s a few ways to write it, but with “average” and “averageBy” built-in, they provide a particularly simple option.

/// Square a number
let sqr x = x * x

/// Compute the standard deviation of a list of numbers
let stddev nums = 
    let mean = nums |> List.average
    let variance = nums |> List.averageBy (fun x -> sqr(x - mean))
    sqrt(variance)

“Okay – but what if I want the inputs to be long, potentially infinite sources of data?”

The previous function worked over lists.  We can change it to work over sequences (IEnumerables) instead.  Pleasantly, it’s basically the same code.  But the result of working over sequences is that this function will work with any sort of data.  It could be a sequence of numbers that result from a LINQ to SQL query, it could be a sequence of numbers read lazily from a very long file, etc. 

/// Compute the standard deviation of a sequence of numbers
let stddevSeq numSeq = 
    let mean = 
        numSeq |> Seq.average
    let variance = 
        numSeq |> Seq.averageBy (fun x -> sqr(x - mean))
    sqrt(variance)

“That’s computing what I want, but can I get all the partial results as my stream of data is processed? ”

This makes things more interesting.  Instead of just computing the standard deviation, we really want to do an “online” calculation, where we compute enough information so that as each new datapoint comes in, we can easily produce the new standard deviation.  This is a different algorithmic approach to the same mathematical formula.  Luckily, wikipedia has what we need- an on-line algorithm for standard deviation from Knuth.

/// Compute the running mean and standard deviation
/// over a sequence of numbers
let meanAndStdDev nums = 
    // A function which updates the on-line computation of 
    // n, mean and M2 given a new datapoint x
    let newValues (n, mean, M2) x = 
        let n' = n+1.
        let delta = x - mean
        let mean' = mean + delta/n'
        let M2' = M2 + delta*(x - mean')
        (n', mean', M2')
    // The initial vaues of n, mean and M2
    let startValues = (0., 0., 0.)
    // The resulting event stream is a scan over the input events 
    nums 
    |> Seq.scan newValues startValues
    |> Seq.map (fun (n, mean, M2) -> (mean, sqrt(M2/(n))))

This time the code has a few interesting features:

1. A local function “newValues” updates n, mean and M2 based on old values and a new datapoint x.  Notice that (a) this is strikingly similar to the pseudo-code in the wikipedia article but (b) this is a functional implementation, so instead of changing the values, we just compute new ones.
2. The function “Seq.scan” takes care of the heavy lifting.  This function, and the related “fold” and “reduce” can be used to capture many common design patterns seen in “programming in the small”.  Scan walks across an input collection applying a function to it’s current accumulator argument, and returning a collection of the results that it builds up as it goes.  If you’ve haven’t written code using these before, they can feel uncomfortable at first, but after you’ve used each of these once or twice, you’ll notice them as common patterns that you are currently encoding as for loops, if statements and local state in your programs.  “scan” and it’s related functions give us the ability to name that design pattern and then re-use it – resulting in more declarative code.

“Perfect!  But my actual stream of data (stock ticker prices) is long running, and will provide new data points periodically over minutes, days and weeks.”

Another fun twist on the problem!  Sequences (IEnumerables) have the property that the producer of the data offers up a way to ask for more, and the consumer is in charge of pulling it out.  While it’s doing so, it’s busy the whole time, so if the producer  doesn’t have the data yet, it will have to block waiting for it to be available.  This is not what you want in the long running case, where instead we want to think about the data stream as a sequence of events.  With events, the consumer offers up a way to deal with new data whenever it’s available, and the producer is in charge of pushing the new data points out.  For the same reasons that there is so much interest in smart phones with push email, we should all want our long-running data streams to be event based instead of sequence based.

let boundedMouseMax = 
    form.MouseMove 
    |> Event.filter (fun mea -> mea.X > 10 && mea.Y > 10 && mea.X < 90 && mea.Y < 90)
    |> Event.map (fun mea -> max mea.X mea.Y)
 
do boundedMouseMax |> Event.add (fun maxCoord -> printfn "max is: %A" maxCoord)

Events in F# are just standard .NET events.  But F# allows you to program with these events in a “first-class” way, meaning you can pass them around as data, and there are functions in the “Event” module for taking events and generating new events from them.  For instance, we could take a mouse move event and call “Event.filter” to filter out any time it is outside a bounding box, then “Event.map” to pick the larger of the X or Y coordinate of the mouse.  The result is then a new event which will fire only when the mouse is in the bounding box, and will produce the larger of it’s X and Y coordinate as it’s value. We can then hook up a listener to print out the values whenever this new event fires.

But how do we use this to solve the original problem?

/// Compute a running mean and standard deviation over the 
/// input event stream of floating point numbers
let meanAndStdDev nums = 
    // A function which updates the on-line computation of n, mean and M2
    let newValues (n, mean, M2) x = 
        let n' = n+1.
        let delta = x - mean
        let mean' = mean + delta/n'
        let M2' = M2 + delta*(x - mean')
        (n', mean', M2')
    // The initial vaues of n, mean and M2
    let startValues = (0., 0., 0.)
    // The resulting event stream is a scan over the input events 
    nums 
    |> Event.scan newValues startValues
    |> Event.map (fun (n, mean, M2) -> (mean, sqrt(M2/(n))))

It’s almost identical to the “on-line” algorithm over sequences! This is one of the important benefits of writing more declarative code using constructs like “scan”.  Although the implementations of “scan” over sequences and over events are quite different under the hood – the design pattern of scanning makes sense over both, and is the concept we can program with to tackle the problem at a higher-level.  It really makes you feel that we’ve succeeded in expressing the “what” in place of the “how”.

Here’s what it looks like to use this new function we’ve created:

// Create a new event – we would hook up to the stock ticker events if they were available
let numEvent = new Event<float>()
let numEventStream, fireNewNum = numEvent.Publish, numEvent.Trigger

// Derive a new event that computes the running mean and stddev 
// over the original event stream
let stddevEvents = meanAndStdDev numEventStream

// Hook up an event handler to print out the new mean and standard deviation 
do stddevEvents |> Event.add (fun (mean, stddev) -> printfn "Mean = %A, StdDev = %A" mean stddev)

do fireNewNum 3.
do fireNewNum 7.
do fireNewNum 7.
do fireNewNum 19.

Conclusion

This is a fun example that really shows off why it’s so valuable to use higher-level design patterns for programming in the small – like “map”, “average”, “averageBy” and “scan”.  First class events in F# are a unifying features which allows you to do the same kind of programming you use with sequences and apply it also to events.  This is really compelling, because it’s a case where two data sources which are conceptually very similar are often programmed in very different ways – but with higher-level programming models that functional programming and F# provide, you can program against both using the same techniques.

Known Issues

1. Ctrl+. keybinding doesn't invoke View.ShowSmartTag in C# and VB editors
2. Saving an F# project can fail when 'Save As' is invoked or VB profile is used
3. Cannot add a reference to a project in a Solution folder
4. Cannot add reference from an F# project to a C++/CLI project in Visual Studio
5. Error message about a missing reference may get 'stuck' in the Error List 
6. F# on 64bit
7. Build errors are not reported in the Error List
8. Dynamically evaluating a quotation which uses the "pown" function fails at runtime

1. Ctrl+. keybinding doesn't invoke View.ShowSmartTag in C# and VB editors

Issue:

The F# CTP uses the ctrl+. keybinding to break execution in the F# Interactive.  However, this keybinding is incorrectly applied globally, which prevents it from working in it's standard Visual Studio use of activating an editor SmartTag in the C# and VB editors.  The result is that installing the F# CTP prevents this keybinding from working in the C# and VB editors.

Workaround: 

To set the keybinding for ctrl+. back to it's original binding to the SmartTag, do the following.
In Tools->Options->Environment->Keyboard do the following:
1) Type "View.ShowSmartTag" in the "Show commands containing:" entry field
2) Hold down "ctrl" and press "." (dot) in the "Press shortcut keys" entry field
3) Click "Assign"

You can optionally re-assign the "OtherContextMenus.FSIConsoleContext.CancelEvaluation" command another keybinding, using similar steps.  Alternatively you can use the right-click menu option to cancel evaluation in the F# Interactive when needed.

2. Saving an F# project can fail when 'Save As' is invoked or VB profile is used

Issue:

The 'Save As' command in Visual Studio does not work on F# project files in the CTP release.  Projects can be saved with the name they were created with, but cannot be saved with a different name.  Related to this, if the 'Save new projects when created' option in Visual Studio is turned off, it is not possible to save an F# project after creating it.  Note that this setting is off by default in the VB profile, but on in all other Visual Studio profiles.  In both cases, an error message saying "The operation could not be completed.  No such interface supported" will be raised.

Workaround:

In Tools->Options->Projects and Solutions->General ensure that 'Save new projects when created' is checked.  To move a project file to another location, save the project and close Visual Studio, then move the project file manually and then reopen it in it's new location.

3. Cannot add a reference to a project in a Solution folder

Issue:

Attempting to add a reference from an F# project to a project which is inside a Solution Folder results in "Object not set to an instance of an object." error message, and fails to add the reference.

Workaround:

You can add an assembly reference to the output of the other project's build by browsing to the assembly in the Add Reference... dialog.  In some cases, it may be important to pick up the correct version of the referenced assembly depending on the build configuration, in these cases, rules such as the following can be added manually to the .fsproj file:

    <Reference Include="..\SolutionFolder\Library2\bin\Debug\Library2.dll" Condition ="'$(Configuration)' == 'Debug'" />

    <Reference Include="..\SolutionFolder\Library2\bin\Release\Library2.dll" Condition ="'$(Configuration)' == 'Release'" />

4. Cannot add reference from an F# project to a C++/CLI project in Visual Studio

Issue: 

Attempting to add a project reference from an F# project to a C++/CLI project results in "Object not set to an instance of an object." error message, and fails to add the reference.

Workaround: 

You can add an assembly reference to the output of the C++/CLI build by browsing to the assembly in the Add Reference... dialog.  In some cases, it may be important to pick up the correct version of the referenced assembly depending on the build configuration, in these cases, rules such as the following can be added manually to the .fsproj file:

    <Reference Include="..\ClassLibrary2\bin\Debug\ClassLibrary2.dll" Condition ="'$(Configuration)' == 'Debug'" />

    <Reference Include="..\ClassLibrary2\bin\Release\ClassLibrary2.dll" Condition ="'$(Configuration)' == 'Release'" />

5. Error message about a missing reference may get 'stuck' in the Error List

Issue: 

Adding a project reference to a project which has never been built will result in an error message appearing in the referencing project.  This error message may not go away when the referenced project is built.

Workaround: 

Shutting down and restarting Visual Studio will cause the error message to go away.  

6. F# on 64bit

Issue: 

Broadly speaking, F# applications will work well on 64bit machines and operating systems.  However, there are two possible issues to be aware of.

1. The F# compiler and interactive are both 32bit processes, and will run in the WoW64 on 64-bit operating systems.  This may prevent loading native 64bit code into the F# interactive and may cause registry reads/writes to behave differently than in a true 64bit process. 
2. F# code may be compiled as either 32bit, 64bit, or platform agnostic (AnyCPU).  Note that when F# applications are run on 64bit, they use the CLR's 64bit Just-In-Time compiler, which may not do tail call optimization in all cases, possibly resulting in StackOverflowExceptions in 64bit environments that were not seen in 32bit environments.

Workaround: 

It is possible to make F# Interactive load as 64-bit by modifying fsi.exe using corflags.exe. There are details on this .NEt Framework SDK tool at https://msdn.microsoft.com/en-us/library/ms164699(VS.80).aspx.

7. Build errors are not reported in the Error List

Issue: 

The error list in Visual Studio shows errors reported by the F# background compiler, but does not show errors reported by the F# compiler during an explicit build.  These are in most cases the same, so that this is not always an issue.  However, if some files in the project are not open, or if an error is reported during generation of the assembly, it may not appear in the error list.  Additionally, errors will appear in the error list for any open files, even ones which are not part of the current project.

Workaround: 

The Output Window in Visual Studio will always show the full results of a build.

8. Dynamically evaluating a quotation which uses the "pown" function fails at runtime

Issue:

Attempting to use the LINQ evaluation of a quotation involving the "pown" function will result in a DynamicInvocationNotSupportedException.

Workaround:

Consider using the ** exponentiation operator, a custom integer exponentiation function or repeated multiplication instead.

The task for the contest changes quite a bit each year, but tends to incorporate themese like implementing interpreters/compilers for simple languages/runtimes and solving optimization or state-space exploration problems related to these interperpreters or programs written to run on these interpreters.  In each of the last two years' contests, some really fun puzzle elements have been incoporated into the contest, along with richer storytelling.

"Universal Machine" in F#

Of the years I've done the ICFP programming contest, my favorite so far has been the 2006 contest.  It was truly an inspired creation, and I am rather in awe of the group from CMU who put it together.  If you didn't do the contest in 2006, you can still go back to the site and try it out - which I can definitely recommend.

The contest started seemingly simply - pointing to a spec for a virtual machine (the "UM"), a binary to run on the virtual machine (the "Codex") and a "decryption key" to use: '(\b.bb)(\v.vv)06FHPVboundvarHRAk'. 

Earlier this week, to re-live some of the fun of the 2006 contest, and to get mentally ready for this years contest, I re-implemented the UM, this time in F#.  Here's what it looks like (take a look at the spec linked above for an expalanation of what it's doing):

open System
open System.IO

type UM(p0 : byte[], input : Stream,  output : Stream) = 

    // Read byte[] in as little-endian uint[]
    let p0AsUInts =
        [| for i in 0..4..(p0.Length-4) ->
            ((uint32 p0.[i])<<<24) + 
            ((uint32 p0.[i+1])<<<16) + 
            ((uint32 p0.[i+2])<<<8) + 
            ((uint32 p0.[i+3])) |]
    // The resizable array of memory pages starts with just the input
    let mem = ResizeArray.create 1 p0AsUInts
    // The registers are initialized to 0
    let reg = [|0u;0u;0u;0u;0u;0u;0u;0u|]

    // The main loop of the interpreter.  Should be *fast*.
    let rec cycle finger =
        let arr = mem.[0]
        let op = arr.[finger]
        let code = op >>> 28
        let (a,b,c) = (int ((op >>> 6) &&& 0b111u), int ((op >>> 3) &&& 0b111u), int ((op) &&& 0b111u))
        let (a2, value) = (int (op >>> 25) &&& 0b111, op &&& 0b1111111111111111111111111u)
        match code with
        | 0u -> (if reg.[c] <> 0u then reg.[a] <- reg.[b]); cycle (finger+1)
        | 1u -> (reg.[a] <- mem.[int reg.[b]].[int reg.[c]]); cycle (finger+1)
        | 2u -> (mem.[int reg.[a]].[int reg.[b]] <- reg.[c]); cycle (finger+1)
        | 3u -> (reg.[a] <- reg.[b] + reg.[c]); cycle (finger+1)
        | 4u -> (reg.[a] <- reg.[b] * reg.[c]); cycle (finger+1)
        | 5u -> (reg.[a] <- reg.[b] / reg.[c]); cycle (finger+1)
        | 6u -> (reg.[a] <- ~~~ (reg.[b] &&& reg.[c])); cycle (finger+1)
        | 7u -> ()
        | 8u -> (mem.Add(Array.zeroCreate (int reg.[c])); reg.[b] <- uint32 (mem.Count - 1)); cycle (finger+1)
        | 9u -> (mem.[int reg.[c]] <- [||]); cycle (finger+1)
        | 10u -> (output.WriteByte(byte reg.[c])); cycle (finger+1)
        | 11u -> (reg.[c] <- uint32 (input.ReadByte() % 255)); cycle (finger+1)
        | 12u -> (if reg.[b] <> 0u then mem.[0] <- Array.copy mem.[int reg.[b]]); cycle (int reg.[c])
        | 13u -> (reg.[a2] <- value); cycle (finger+1)
        | _ -> failwith "Did not understand %A" (op, code, a, b, c)

    /// Start the Universal Machine running
    member this.Run() =
        cycle 0

let program = File.ReadAllBytes("codex.umz");
let ee = UM(program, Console.OpenStandardInput(), Console.OpenStandardOutput())

//Try the following if too much is printed to the screen 
//let ee = UM(program, Console.OpenStandardInput(), new FileStream("umix.um", FileMode.Create))ee.Run()

Performance

This code is doing a lot of bit-level operations in F#.  The meat of the code is the tight loop that runs executing the "cycle" function.  Since this is the runtime that the provided binary will run on, it needs to be fast.  And, fortunately, it is.  On my laptop, this loop runs about 15million iterations per second.  Of course, there are opportunities to improve on the performance of this code, but this turns out to be fast enough for the problem, and it's convenient that you get this performance profile by default.

That Doesn't Look Very Functional?

This code is not particularly 'functional'.  But it is totally reasonable F# code.  We often talk about F# as a hybrid functional, imperative and OO programming language.  This is an example where the problem domain really suggests the use of some imperative programming ideas.  The problem defines some global state: the memory, and the registers.  Since we're going to be modifying these constantly during the execution of the runtime, it makes sense to use mutable arrays to store the contents.  F# makes this nice and easy to do.

Wrapping it up with a little OO

As I menionted in a previous blog post, it's really easy to transition between prototyping and component design with F#.  The code above wraps up the state and algorithm for the UM in a simple type definition.  This provides nice encapsulation and abstraction over the specific binary to run, and the input and output streams to use.  And it also wraps up the F# code in a type which could just as easily be consumed from C# or any other .NET langauge.