We ended Part 2 with a set of general purpose parsers that we will use here to, finally, build the JSON parser.

Make sure to read Part 1 and Part 2 for a refresher if you need it.

First things first

We all know how JSON looks like, but for this exercise point your browser to json.org, and check the details of the structures it is built of. Let’s focus on the EBNF-like grammar on the right of the page: the values that form the JSON structured are strings, numbers, objects, arrays, booleans and null – it is pretty simple to represent that in F# using an algebraic type:

type JSValue = 
    | JSString of string
    | JSNumber of float
    | JSObject of (JSValue * JSValue) list
    | JSArray of JSValue list
    | JSBool of bool
    | JSNull
F# Web Snippets

This will be our JSON abstract syntax tree and our parser will “fill up” this structure as it parsers the input string.

There is this simple trick I often use when I’m prototyping with some code, I don’t know yet how to implement some of the functionality of I need, but I want to rely on F#’s type system to make sure I’m on the right way. The idea is use Unchecked.defaultof to generate a default value for the return of your function while you don’t have an implementation for it (I guess this is just like the NotImplementedException VS throws in when generating class member stubs for you, just that we not throwing any exception in this case... we could, but we are not...). I always keep an undefined function handy, like this:

let undefined<'a> = Unchecked.defaultof<'a>

For example, let’s write the functions of our JSON parser based on the grammar from json.org. The parsers for JSString, JSNumber, JSBool and JSNull are straight forward, they are just JSValue Parsers:

let jsString = undefined<JSValue Parser>
let jsNumber = undefined<JSValue Parser>
let jsBool  = undefined<JSValue Parser>
let jsNull = undefined<JSValue Parser>

Note that that JSObject and JSArray both depend on JSValue, and JSValue on them, so these guys will need to be recursive:

let rec jsValue = jsString <|> jsNumber <|> jsArray <|> jsBool <|> jsNull <|> jsObject
and jsElements = undefined<JSValue list Parser>
and jsArray = undefined<JSValue Parser>

and jsMembers = undefined<(JSValue * JSValue) list Parser>
and jsPair = undefined<(JSValue * JSValue) Parser>

and jsObject = undefined<JSValue Parser>

Finally we define the overall JSValue parser as a combination of parsers using the “or” parser combinator (<|>) form last post, meaning a JSValue parser is a jsString parser, or a jsNumber parser, or a jsArray parser, …, and so on.

I really like how this representation and types mimic the JSON grammar, for example: a JSON object is a value (<JSValue Parser>), composed of members (<(JSValue * JSValue) list Parser>) of pairs of string : values (<(JSValue * JSValue) Parser>).

You can actually send the code above to F# Interactive and it will “run” just fine, showing that so far we've got the types correct:

val jsString : JSValue Parser = null
val jsNumber : JSValue Parser = null
val jsBool : JSValue Parser = null
val jsNull : JSValue Parser = null
val jsObject : JSValue Parser = null
val jsMembers : (JSValue * JSValue) list Parser = null
val jsPair : (JSValue * JSValue) Parser = null
val jsArray : JSValue Parser = null
val jsElements : JSValue list Parser = null
val jsValue : JSValue Parser = Parser <fun:op_LessBarGreater@32>

A parser at a time

Dividing to conquer… Let’s build a parser a time, substituting the default values by real implementations.

jsNull Parser

jsNull seems pretty simple to implement, whenever we see a null, we need to return JSNull. We can simply use the symb parser for that:

let jsNull =
    let nullLiteral = s2cs "null"
    parser { let! _ = symb nullLiteral in return JSNull}

Remember, symb takes a char list as input and returns a parser that is able to parse that string. In this case we don’t really need the result of parsing “null”, we only need that it succeeds… and if it succeeds, we return JSNull . Let’s run it on Interactive:

> s2cs "null" |> apply jsNull;;
val it : (JSValue * char list) list = [(JSNull, [])]

jsBool Parser

jsBool is very similar to jsNull, however in this case we can return either JSBool(true) or JSBool(false): a perfect case for our “or” parser:

let jsBool = 
    let trueLit = "true" |> s2cs
    let falseLit = "false" |> s2cs
    parser { let! _ = symb trueLit in return JSBool(true) } 
    <|> parser { let! _ = symb falseLit in return JSBool(false) }

Simple, right? jsBool is a parser that sequences two other parsers, one that parses “true” and the other that parses “false”; its return will depend on the parser that succeeds:

> s2cs "true" |> apply jsBool;;
val it : (JSValue * char list) list = [(JSBool true, [])]
> s2cs "false" |> apply jsBool;;
val it : (JSValue * char list) list = [(JSBool false, [])]

> s2cs "false true false true" |> apply (many jsBool);;
val it : (JSValue list * char list) list =
  [([JSBool false; JSBool true; JSBool false; JSBool true], [])]

jsNumber Parser

jsNumber is a bit more complex, check out the number grammar on json.org… We could use the same approach as before to slowly build it:

let jsNumber = 
    let digitsParser = undefined<char list Parser>
    let intParser = undefined<char list Parser>
    let fracParser = undefined<char list Parser>
    let expParser = undefined<char list Parser>
    undefined<JSValue Parser>

digitsParser has to parse one to many digits; if you remember last post, we can accomplish it with the sat and many1 parsers, like this:

let jsNumber = 
    let digitsParser = many1 (sat System.Char.IsDigit)
    let intParser = (zeroOrOne (text ['-'])) <@> digitsParser
    let fracParser = text ['.'] <@> digitsParser
    let expParser = 
        (text ['e'] <|> text ['E']) <@> 
        (zeroOrOne (text ['+'] <|> text ['-'])) <@>
        digitsParser
    parser {
        let! result = intParser <@> zeroOrOne fracParser <@> zeroOrOne expParser 
        let! _ = space
        return (result |> cs2s |> System.Double.Parse |> JSNumber)
    }     

intParser is quite interesting… it has to sequence zero or one applications of a ‘-‘ parser with the digitsParser. We don’t have a “one or zero” combinator in our bag of parsers from last post, and it seems a good candidate to be added to the list, so let’s define it:

let zeroOrOne p = parser { let! ret = p in return ret } <|> parser { return [] }

When applied, this parser will try to parse the input string using parser p, and if that fails it returns an empty list Parser. Hover over the definition, this is an ‘a list Parser, so p also has to be an ‘a list Parser too, like the text parser defined in the last post:

> [ '-' ] |> apply (zeroOrOne (text ['-']));;
val it : (char list * char list) list = [(['-'], [])]

Let’s then build intParser using it:

let intParser = (zeroOrOne (text ['-'])) <@> digitsParser

We are basically “appending” two parsers here: zeroOrOne (text ['-']) and digitsParser, this can also be quite common, so let’s build a parser combinator for that:

let (<@>) p q = parser {
    let! rp = p
    let! rq= q
    return (rp @ rq)
}

Now intParser, fracParser and expParser can all be written using that:

let intParser = (zeroOrOne (text ['-'])) <@> digitsParser
    let fracParser = text ['.'] <@> digitsParser
    let expParser = 
        (text ['e'] <|> text ['E']) <@> 
        (zeroOrOne (text ['+'] <|> text ['-'])) <@>
        digitsParser

All we need to do now is sequence all these parsers, one by one, concatenate the results and return the equivalent JSNumber:

let jsNumber = 
    let digitsParser = many1 (sat System.Char.IsDigit)
    let intParser = (zeroOrOne (text ['-'])) <@> digitsParser
    let fracParser = text ['.'] <@> digitsParser
    let expParser = 
        (text ['e'] <|> text ['E']) <@> 
        (zeroOrOne (text ['+'] <|> text ['-'])) <@>
        digitsParser
    parser {
        let! result = intParser <@> zeroOrOne fracParser <@> zeroOrOne expParser 
        let! _ = space
        return (result |> cs2s |> System.Double.Parse |> JSNumber)
    }

Two comments here: because we are not using any of the combinators that take care of cleaning up spaces, we need to make sure to sequence a space parser in there. Second, all these parsers are char list parsers, so we can append them together in the end to come up with a string representation of the float number. Here it is in Interactive:

> "-1234.5678e-8" |> s2cs |> apply jsNumber;;
val it : (JSValue * char list) list = [(JSNumber -1.2345678e-05, [])]
> "-1234.5678e8" |> s2cs |> apply jsNumber;;
val it : (JSValue * char list) list = [(JSNumber -1.2345678e+11, [])]
> "-1234.5678e+8" |> s2cs |> apply jsNumber;;
val it : (JSValue * char list) list = [(JSNumber -1.2345678e+11, [])]
> "-1234.5678E+8" |> s2cs |> apply jsNumber;;
val it : (JSValue * char list) list = [(JSNumber -1.2345678e+11, [])]
> "1234.5678E+8" |> s2cs |> apply jsNumber;;
val it : (JSValue * char list) list = [(JSNumber 1.2345678e+11, [])]
> "1234E+8" |> s2cs |> apply jsNumber;;
val it : (JSValue * char list) list = [(JSNumber 1.234e+11, [])]
> "1234E-8" |> s2cs |> apply jsNumber;;
val it : (JSValue * char list) list = [(JSNumber 1.234e-05, [])]
> "1234.5678" |> s2cs |> apply jsNumber;;
val it : (JSValue * char list) list = [(JSNumber 1234.5678, [])]
> "1234" |> s2cs |> apply jsNumber;;
val it : (JSValue * char list) list = [(JSNumber 1234.0, [])]

How cool is that?!

jsString Parser

jsString is bit complex due to all the different  “production rules”. Let’s start by building a couple predicates based on those rules, and a helper function – this is straight out of JSON’s grammar:

let isChar c = (c <> '\"') && (c <> '\\')
    let isEscChar = 
        let cs = "\"\\/bfnrt" |> List.ofSeq |> Set.ofList
        cs.Contains
    let replaceEscChar = function 'b' -> '\b' | 'f' -> '\f' | 'n' -> '\n'
                                | 'r' -> '\r' | 't' -> '\t' | other -> other

replaceEscChar will replace a JSON escape character, identified by isEscChar, by its backslash representation. Let’s build the escChar Parser with these guys’ help:

let escChars = parser {
        let! _ = tChar '\\'
        let! c = sat isEscChar
        return (replaceEscChar c)
    }

We will also need a unicode parser, which will need to parse 4 hexadecimals, how about starting by putting together a hexDigit Parser?

let isHexDigit = 
    let ds = ['A'..'F'] @ ['a'..'f'] @ ['0'..'9'] |> Set.ofList
    ds.Contains
let hexDigit = sat isHexDigit

Nothing new here… based on that, uniChars becomes:

let uniChars = parser {
        let! _ = text [ '\\'; 'u' ]
        let! d1 = hexDigit
        let! d2 = hexDigit
        let! d3 = hexDigit
        let! d4 = hexDigit
        let r = 
            let s = new String [|d1; d2; d3; d4|]
            Byte.Parse(s, Globalization.NumberStyles.HexNumber)
            |> char
        return r
    }

There is one more pattern that will repeat on this and next parsers, for example, a JSON string is any char between ‘ “ ‘, objects are members between “{ }”, arrays are values between “[ ]”… It seems a good idea to build a “between chars” parser, no? Here it goes:

let charToken = tChar >> token

let betweenChars c1 c2 f = parser {
    let! _ = charToken c1
    let! r = f()
    let! _ = charToken c2
    return r
}

betweenChars takes 2 chars and a function (that returns a parser), it binds then together using the let! syntax, and returns the result of applying the parser returned by the function f. And by the way, it comes with a freebie! The charToken Parser: it cleans up spaces after sequencing the tChar Parser. jsString is then defined as:

let jsString = 
    let isChar c = (c <> '\"') && (c <> '\\')
    let isEscChar = 
        let cs = "\"\\/bfnrt" |> List.ofSeq |> Set.ofList
        cs.Contains
    let replaceEscChar = function 'b' -> '\b' | 'f' -> '\f' | 'n' -> '\n'
                                | 'r' -> '\r' | 't' -> '\t' | other -> other
    let escChars = parser {
        let! _ = tChar '\\'
        let! c = sat isEscChar
        return (replaceEscChar c)
    }
    let uniChars = parser {
        let! _ = text [ '\\'; 'u' ]
        let! d1 = hexDigit
        let! d2 = hexDigit
        let! d3 = hexDigit
        let! d4 = hexDigit
        let r = 
            let s = new String [|d1; d2; d3; d4|]
            Byte.Parse(s, Globalization.NumberStyles.HexNumber)
            |> char
        return r
    }
    let chars = many ((sat isChar) <|> escChars <|> uniChars)
    parser {
        let! cs = betweenChars '\"' '\"' (fun () -> chars)
        return (cs |> cs2s |> JSString)
    }

This is probably the biggest parser we have ever built!!! The chars Parser is a combination of a parser that parsers any JSON character (sat isChar), or an escChar or a uniChars. We sequence it using our new betweenChars parser combinator and return a JString of the parsed string. Kind of cool...

The recursive parsers

We already know that the jsValue parser is a combination of all the individual value parsers:

let rec jsValue = jsString <|> jsNumber <|> jsArray <|> jsBool <|> jsNull <|> jsObject

Let’s build the array parser first. Per the JSON grammar, an array is a list of elements between “[ ]”, and elements are JSValues separated by ‘,’. This definition can be easily translated into:

and jsElements = sepby jsValue (charToken ',')
and jsArray = parser {
    let! values = betweenChars '[' ']' (fun () -> jsElements)
    return (JSArray values)
    }

Here we are using the sepby parser combinator we learned last time to build the parser for the array elements. We then put it between ‘[‘ ‘]’ with betweenChars. It ends up being quite clean.

How about the objects? Objects are members between “{ }”, and members are pairs separated by “,”, which are in the format JString : JSValue. This definition also translates easily into F# with our parser combinators:

and jsMembers = sepby jsPair (charToken ',')
and jsPair = parser {
    let! key = jsString
    let! _ = charToken ':'
    let! value = jsValue
    return (key, value)
    }
and jsObject = parser {
    let! members = betweenChars '{' '}' (fun () -> jsMembers)
    return (JSObject members)
    }

One last thing, let’s define a function to easily parse a json string; let’s require that any JSON string to be a JSObject, and use runParser with the jsObject to kick off the parsing sequence: given a string, return the JSON AST (which is a JSValue):

let parseJson : (string -> JSValue) = runParser jsObject

The moment of truth

This is the moment we have been waiting so long for… does all this craziness really work? Go fetch some JSON data and let’s try it out, I’m grabbing some from http://json.org/example:

{"widget": {
    "debug": "on",
    "window": {
        "title": "Sample Konfabulator Widget",
        "name": "main_window",
        "width": 500,
        "height": 500
    },
    "image": {
        "src": "Images/Sun.png",
        "name": "sun1",
        "hOffset": 250,
        "vOffset": 250,
        "alignment": "center"
    },
    "text": {
        "data": "Click Here",
        "size": 36,
        "style": "bold",
        "name": "text1",
        "hOffset": 250,
        "vOffset": 100,
        "alignment": "center",
        "onMouseUp": "sun1.opacity = (sun1.opacity / 100) * 90;"
    }
}}

let widgetJson = "{\"widget\": {
 \"debug\": \"on\",
 \"window\": {
 \"title\": \"Sample Konfabulator Widget\", 
 \"name\": \"main_window\", 
 \"width\": 500, 
 \"height\": 500
 }, \"image\": { 
 \"src\": \"Images/Sun.png\",
 \"name\": \"sun1\", 
 \"hOffset\": 250, 
 \"vOffset\": 250, 
 \"alignment\": \"center\"
 }, \"text\": {
 \"data\": \"Click Here\",
 \"size\": 36,
 \"style\": \"bold\", 
 \"name\": \"text1\", 
 \"hOffset\": 250, 
 \"vOffset\": 100, 
 \"alignment\": \"center\",
 \"onMouseUp\": \"sun1.opacity = (sun1.opacity / 100) * 90;\"
 }
}}"

> parseJson widgetJson;;
val it : JSValue =
  JSObject
    [(JSString "widget",
      JSObject
        [(JSString "debug", JSString "on");
         (JSString "window",
          JSObject
            [(JSString "title", JSString "Sample Konfabulator Widget");
             (JSString "name", JSString "main_window");
             (JSString "width", JSNumber 500.0);
             (JSString "height", JSNumber 500.0)]);
         (JSString "image",
          JSObject
            [(JSString "src", JSString "Images/Sun.png");
             (JSString "name", JSString "sun1");
             (JSString "hOffset", JSNumber 250.0);
             (JSString "vOffset", JSNumber 250.0);
             (JSString "alignment", JSString "center")]);
         (JSString "text",
          JSObject
            [(JSString "data", JSString "Click Here");
             (JSString "size", JSNumber 36.0);
             (JSString "style", JSString "bold");
             (JSString "name", JSString "text1");
             (JSString "hOffset", JSNumber 250.0);
             (JSString "vOffset", JSNumber 100.0);
             (JSString "alignment", JSString "center");
             (JSString "onMouseUp",
              JSString "sun1.opacity = (sun1.opacity / 100) * 90;")])])]

Ladies and gentlemen, it works!!! I can see some watery eyes out there, and it is ok, building monadic parsers is most definitely an emotional experience!

Make sure to get all the parsers and play with them at http://fssnip.net/
• Parser Monad and combinators
• JSON Parser

Wrapping up

This is just like Lego for functional programmers: from parsing a single character to parsing full JSON strings by “simply” composing pieces of parsers together! Now you understand why I completely forgot why I needed a JSON parser in the first place :-).

That said, and now that we are calming down a little bit, it seems we are still missing an piece – if you compare the result above with the "boring parsing" from Part 1, you will notice we here have just a JSON AST, while there, we ended up with a “real” F# type… I guess we will need to look at that in the next post.

Stay tuned and let me know your comments!

Parsing JSON the fun way series

• Part 1: Building a parser from scratch with Dr. Seuss help (part 1)
• Part 2: Tricking you into Parser Monads (it's not going to hurt, I promise)
• Part 3: The JSON Parser Monad

We will stick to the one line programming language once more for part 2 (last time, promise), and will slowly get to the magical Parser Monads. By the end of this post, we should have all parsers from the Monadic Parsing in Haskell paper up and running.

Sugar Love

You probably noticed we kept using expressions like “a thing Parser”, “an ‘a Parser”, “satisfy this predicate Parser”, etc. everywhere in part 1 of the series. It seems logical, at this point, to embrace this concept and code it accordingly in F#, doesn’t it? Remembering again Dr. Seuss’ parser definition using F# notation:

‘a Parser : char list -> (‘a * char list) list

Let’s create a “container” for our parsers using an algebraic type – we will simply take the parser definition as-is, and enclose it into a Parser container:

type 'a Parser = Parser of (char list -> ('a * char list) list)
F# Web Snippets

Got it? We didn’t really do much, literally just copied and pasted Dr. Seuss definition… Any of the parsers we built in part 1 can be encapsulated into this Parser container, for example, remember our char Parser called cParser?

let cParser = function [] -> [] | c::cs -> [c, cs]

Let’s wrap it into the Parser type and call it item (adopting the name convention from Erik's paper):

let item = Parser (function [] -> [] | c::cs -> [c, cs])

Hover your mouse over item and let me know what you see… Correct! As expected item is now officially a char Parser! Let’s test it and see if it works (s2cs was defined in the last post):

> "Test" |> s2cs |> item;;

  "Test" |> s2cs |> item;;
  ------------------^^^^

C:\Users\fzandona\AppData\Local\Temp\stdin(3,19): error FS0001: This expression was expected to have type
    char list -> 'a   
but here has type
    char Parser 

Oops! Clearly we cannot use the same approach we've been using here; we need to first extract the function form inside the Parser type before applying it. Thanks to F#’s pattern matching, we can write:

let parse (Parser p) = p 

And use it like this:

> "Test" |> s2cs |> parse item;;
val it : (char * char list) list = [('T', ['e'; 's'; 't'])]

Good, we are back in business!

Digressing a little bit here, you’ve probably already noticed that I really like to chain functions together by “pipelining” them using the pipeline operator ( |> ); it looks very expressive to me, for example, the F# line above can be read as: take “Text” and send it to s2cs, take the result (a char list) and send it to (parse item). Compare it to, where you need to read the code backwards:

> parse item (s2cs "Text");;
val it : (char * char list) list = [('T', ['e'; 'x'; 't'])]

Back on track, we will continue wrapping all parsers we’ve built so far with our Parser container in a minute, but let’s focus initially at these 3 functions we worked on:

let returnParser thing = fun cs -> [thing, cs]
let zeroParser () = fun _ -> []
let bindParser (p:(char list -> ('a * char list) list), (f : 'a -> (char list -> ('b * char list) list))) = 
    fun cs -> 
        match p cs with
        | (c', cs')::_ -> (f c') cs'
        | [] -> []

We use these guys a lot so let’s give them some TLC. Not sure if you realized, but they all comply with Dr. Seuss definition, so it is going to be a good thing to encapsulate them into the new Parser type, giving us:

let returnParser' thing = Parser (fun cs -> [thing, cs])
let zeroParser' () = Parser (fun _ -> [])
let bindParser' (p : ('a Parser), f : ('a -> 'b Parser)) = 
    Parser(
        fun cs -> 
            match parse p cs with 
            | (c', cs')::_ -> parse (f c') cs'
            | [] -> []
    )

Note that the only difference from this snippet and the previous one is the existence of the parser container, and the use of our parse function to extract the parser from the container before applying it. By the way, hover your mouse over these functions and note how cleaner the type definitions are, we got rid of all that “char list” noise thanks to the Parser type!

Second, because these function/parsers are so important, and now that we have them beautifully wrapped, how about putting them into a type class for “organizational reasons”?

type ParserBuilder () =
    member x.Return a = Parser (fun cs -> [a, cs])
    member x.Bind (p, f) = Parser (fun cs -> 
        match parse p cs with
        | (c', cs')::_ -> parse (f c') cs'
        | [] -> []
    )
    member x.Zero () = Parser (fun _ -> [])
    member x.ReturnFrom a = a

let parser = ParserBuilder()

No changes here, just a class with our previous functions as members, simple and clean (well, I actually sneaked in another member called ReturnFrom, which is just an id function, meaning it returns whatever it receives as input; you will see why we need it later). We also defined a parser value, which is just an instance of our ParserBuilder class.

The next two parsers we built last time were satisfyParser and the tcParser (“this specific char Parser”):

let satisfyParser pred = 
    bindParser(cParser, fun c ->
        if pred c then returnParser c else zeroParser()
    )

let tcParser c = satisfyParser ((=) c)

We should build them in the new neater way by using the parser instance (let parser = ParserBuilder()) and the item parser:

let satisfyParser' pred = 
    parser.Bind(item, fun c ->
        if pred c then parser.Return c else parser.Zero()
    )

let tcParser' c =  satisfyParser' ((=) c)

No news here, just a plain straight rewrite using the new function notations.

Let the magic begin

So far so good, but it can get tiring very quickly if we need to keep calling all those binds, returns and zeros, to build more capable parsers - so let’s let the F# compiler do it for us!

I actually tricked you into creating the ParserBuilder class for “organization purposes” only, I did have a hidden goal… It turns out that, whenever the compiler sees a class type with some especial methods like the ones in ParserBuilder, it does some under-the-covers magic and provides us with some well-deserved syntactic sugar (interestingly enough, you will see that this new syntax allows us to write code that looks very similar to the imperative style, go figure...). For example, satisfyParser’  and tcParser’ can now be written as (renaming them to sat and tChar respectively):

let sat pred = parser {
    let! c = item
    if pred c then return c
}

let tChar c = sat ((=) c)

This is very neat, isn’t it? They call it computation expressions, but I prefer to call it "the magical syntactic sugar that allows me to easily sequence and compose computations". It is exactly the same behavior as before, but with some magical sugar from the compiler: let! c = item is replaced by parser.Bind(item, fun c -> ...); everything after the let! line is added to the continuation function of the Bind call, and finally return c is replaced by the call to parser.Return c. Note how “sweet” this notation is: we don’t even need an else branch in the conditional test above, the compiler is attaching the call to the Zero() for us! You can find all special methods for the "builder types" (like our ParserBuilder class) on this MSDN entry.

If you reached this far, I have a suprise for you! You may not have noticed, but you’ve just built your very first Parser Monad! Congrats! It didn’t even hurt, did it?

No kidding! F#’s computation expression is F#’s syntax for monads, so applying some logic here: a parser using computation expression is a parser using monad, which is a Parser Monad! Cool, now go and tell everyone you’ve conquered the “Monadic Parser Badge” :-).

Using this new notation, let’s port to F# the remaining parsers from Erik’s paper: a couple of “choice combinators”, the "recursion combinators" and finally the "lexical combinators".

Choice Combinators

Plus and Or Parsers

/// Concatenates the results of applying parser p and parser q
let (<+>) p q = Parser (fun cs -> (parse p cs) @ (parse q cs))
/// Applies parser p or parser q and returns at most one result
let (<|>) p q = Parser (fun cs -> 
    match (parse (p <+> q) cs) with
    | []    -> []
    | x::xs -> [x]
)

The “plus” parser (<+>) sequences two parsers (p and q) on the same input string and concatenates the resulting lists. We use this “plus” parser to build an “or” parser which returns at most one result: the result of applying p if it succeeds, or the result of applying q if p fails (or empty list if both fail).

The somewhat weird <+>  and <|> function names allow us to use them in an infix manner, so we can write things like: aParser <+> bParser  or cParser <|> dParser.

Let’s run them on F# interactive to better understand their behavior:

> "Test" |> s2cs |> parse (tChar 'T' <+> tChar 'T');;
val it : (char * char list) list =
  [('T', ['e'; 's'; 't']); ('T', ['e'; 's'; 't'])]

> "Test" |> s2cs |> parse (tChar 'T' <|> tChar 'T');;
val it : (char * char list) list = [('T', ['e'; 's'; 't'])]

> "Test" |> s2cs |> parse (tChar 'T' <+> tChar 'Z');;
val it : (char * char list) list = [('T', ['e'; 's'; 't'])]

> "Test" |> s2cs |> parse (tChar 'T' <|> tChar 'Z');;
val it : (char * char list) list = [('T', ['e'; 's'; 't'])]

> "Test" |> s2cs |> parse (tChar 'Z' <|> tChar 'T');;
val it : (char * char list) list = [('T', ['e'; 's'; 't'])]

> "Test" |> s2cs |> parse (tChar 'Z' <+> tChar 'T');;
val it : (char * char list) list = [('T', ['e'; 's'; 't'])]

> "Test" |> s2cs |> parse (tChar 'Z' <+> tChar 'Z');;
val it : (char * char list) list = []

Recursion Combinators

Text Parser

/// Given a char list, returns a parser that parsers it
let rec text = function
    | []  -> parser { return [] }
    | c::cs -> parser { 
        let! _ = tChar c
        let! _ = text cs
        return c::cs 
    }

The text parser is very similar to the stringParser we built in the last post: it takes a char list as parameter and builds a parser that is capable of parsing that specific char list:

> let thisParser = text (s2cs "This");;

val thisParser : char list Parser = Parser <fun:Bind@40>

> "This is a test" |> s2cs |> parse thisParser;;
val it : (char list * char list) list =
  [(['T'; 'h'; 'i'; 's'], [' '; 'i'; 's'; ' '; 'a'; ' '; 't'; 'e'; 's'; 't'])]

Many and Many1 Parsers

/// Combines many (0 or more) applications of parser p
let rec many p = (many1 p) <|> parser { return [] }
/// Combines at least one (1 or more) applications of parser p
and many1 p = 
    parser { 
        let! r = p
        let! rs = many p
        return r::rs
    } 

Many and its brother many1 are really useful parsers when you need to sequence zero to many, or one to many, applications of a parser. For example, remember the homework from the last post, how to parse “fun      z     =    7”? We can now use many/many1 to easily consume all those white spaces from the input string:

type FunAST = Fun of (char * int)
let funParser = parser {
    let! _ = text (s2cs "fun")
    let! _ = many1 (tChar ' ')
    return Fun
}
let identParser = parser {
    let! ident = item
    let! _ = many (tChar ' ')
    return ident
}
let digitParser = parser {
    let! d = sat Char.IsDigit
    let! _ = many (tChar ' ')
    return (d |> string |> Int32.Parse)
}
let equalParser = parser {
    let! c = tChar '='
    let! _ = many (tChar ' ')
    return c
}
let funLangParser = parser {
    let! _ = many (tChar ' ')
    let! funFunc = funParser
    let! ident = identParser
    let! _ = equalParser
    let! digit = digitParser
    return funFunc(ident, digit)
}

> "fun      z     =    7" |> s2cs |> parse funLangParser;;
val it : (FunAST * char list) list = [(Fun ('z', 7), [])]
> "fun      z     =    7          " |> s2cs |> parse funLangParser;;
val it : (FunAST * char list) list = [(Fun ('z', 7), [])]
> "fail      z     =    7          " |> s2cs |> parse funLangParser;;
val it : (FunAST * char list) list = []

Sepby and Sepby1 Parsers

/// Combines 0 or more applications of parser p separated by parser sep
let rec sepby p sep =  (sepby1 p sep) <|> parser { return [] }
/// Combines 1 or more applications of parser p separated by parser sep
and sepby1 p sep = 
    parser {
        let! r = p
        let! rs = many (parser { 
            let! _ = sep
            return! p
        })
        return r::rs
    }

The next pair of recursion combinators is sepby and sepby1 – they are also very useful when you have a parsing pattern, e.g. things “separated” by other things. They both take two parsers as parameters, a parser to be repeatedly sequenced, and a “separation” parser, that is thrown away. Note how they rely on many and <|> in their implementation. To test them, let’s pretend we can now have entries like this in our fun language:

fun x = 1;
fun z = 9;

We could easily parse them by using the sepby/sepby1 parsers:

let sepParser = parser {
    let! _ = many (tChar ' ')
    let! c = tChar ';'
    return c
}

let improvedLangParser = sepby1 funLangParser sepParser

> "fun z = 1;" |> s2cs |> parse improvedLangParser;;
val it : (FunAST list * char list) list = [([Fun ('z', 1)], [';'])]
> "fun z = 1;   fun y = 2; fun     w        =     3" |> s2cs |> parse improvedLangParser;;
val it : (FunAST list * char list) list =
  [([Fun ('z', 1); Fun ('y', 2); Fun ('w', 3)], [])]

Chainl and Chainl1 Parsers

/// Chain 0 or more applications of parser p separated by applications of parser op
let rec chainl p op a = (chainl1  p op) <|> parser { return a }
/// Chain 1 or more applications of parser p separated by applications of parser op
and chainl1 p op =  
    let rec rest r = 
        parser {
            let! f = op
            let! r' = p
            return! rest (f r r')
        } <|> parser {return r}

    parser { let! a = p in return! rest a }

Finally the last pair of recursion combinators is chainl and chainl1 – they are a little bit trickier though, they parse repeated applications of parser p separated by applications of parser op, similar to sepby; however, op is a left associative operator parser (ohh… I’ve got to use that phrase tonight at the dinner table!). If p is an ‘a Parser, then op will be an (‘a -> ‘a -> ‘a) Parser…

Improving even more our fun language, let’s now support code like “fun x = 1 + 1” by using the chainl and <|> parsers:

let sumParser = parser {
    let! _ = tChar '+'
    let! _ = many (tChar ' ')
    return (+)
}
let digitParser' = 
    chainl1 digitParser sumParser 
    <|> digitParser

let funLangParser' = parser {
    let! _ = many (tChar ' ')
    let! funFunc = funParser
    let! ident = identParser
    let! _ = equalParser
    let! digit = digitParser'
    return funFunc(ident, digit)
}

let improvedLangParser' = sepby1 funLangParser' sepParser

> "fun x=1+ 1; fun z = 9 + 1 + 2 + 3; fun w = 5" |> s2cs |> parse improvedLangParser';;
val it : (FunAST list * char list) list =
  [([Fun ('x', 2); Fun ('z', 15); Fun ('w', 5)], [])]

Hover over the new sumParser parser and you will notice it is a (int -> int -> int) Parser and it returns a left associative operator (+). We use it in the new digitParser’ by chaining digitParser with it. We then apply <|> to try both options, e.g., a digit in our language can be a sum of digits (chainl1 digitParser sumParser), or (<|>) a single digit (digitParser): at most one of these two parsers must return a value. It is also important to note the order used to in the <|> call, had digitParser been used first (e.g. “digitParser <|> chainl1 digitParser sumParser”), chainl1 would never have a chance to be applied, because digitParser would always return the first digit of the sum.

Lexical Combinators

These are very simple parsers but yet very handy. They will implement some of the functionality we’ve been hardcoding so far.

Space Parser

let isSpace =
    // list of "space" chars based on 
    // http://hackage.haskell.org/packages/archive/base/latest/doc/html/Data-Char.html#v:isSpace
    let cs = [' '; '\t'; '\n'; '\r'; '\f'; '\v'] |> Set.ofList
    cs.Contains
let space = many (sat isSpace)

space comes to get us out of the business of parsing whitespaces. We build a predicate function (isSpace) using a set of chars we want to consider as whitespaces (I’m basing mine on the same ones used by Haskell’s Data.Char.isSpace), and then we create a parser that is capable of “parsing many characters that satisfy the isSpace predicate” (I like how the function implementation easily translates to English: many (sat isSpace)).

Token Parser

let token p = parser { 
    let! r = p
    let! _ = space
    return r
}

token takes a parser, sequences it and the space parser, returning the result of the application of the input parser; in other words, it will apply a parser you give it and remove all white space afterwards, returning the parsed thing.

Symbol Parser

let symb = text >> token

symb takes it further by composing a text parser and a token parser (for the less “functional composition” savvy, it could be written as let symb cs = token (text cs) :-). This little guy will allow us to parse a specific string (a literal, for example) with whitespace clean up afterwards… and it will be used a lot.

Apply Parser

let apply p = parse (parser {
    let! _ = space
    let! r = p
    return r
})

Last but not least, apply is used to initiate the parsing computation: it calls parse (the extraction function we defined earlier) on a parser that cleans up the initial whitespace (if any) of the string to be parsed and applies the parser you give it to the rest of the string.

Final fun language parser

Let’s take all these parser combinators and build yet another, but this time really final, version of our fun language parser. Just because we now have more building blocks in our bag, let’s add the capability of accepting more than one digit at time (“fun a = 123 + 321”), by using many1 in the digit parser (see below):

let funP = parser { let! _ = symb (s2cs "fun") in return Fun }
let

identP = parser {
    let! c = token item
    let! _ = symb ['=']
    return c
}

let sumP = parser { let! _ = symb ['+'] in return (+) }

let digitP = 
    let digitP' = parser {
        let! ds = token (many1 (sat Char.IsDigit))
        return  (ds |> cs2s |> Int32.Parse)
    }
    chainl1 digitP' sumP
    <|> digitP'

let funLangP = parser {
    let! f = funP
    let! ident = identP
    let! d = digitP
    return f(ident, d)
}

let finalLangP = sepby1 funLangP (symb [';'])

Do you remember the runParser function?  We can update it to use the apply parser (so white space in the beginning of the string is taken care of), and test our final parser:

let runParser p = 
    s2cs >>
    apply p >>
    function
        | [] -> failwith "Error parsing string"
        | (result,_)::_ -> result
        

let doesItReallyWork () = 
    " fun x=99+ 1; 
 fun z = 9 + 1 + 2 + 3; 
 fun w = 57"
    |> runParser finalLangP

> doesItReallyWork ();;

val it : FunAST list = [Fun ('x', 100); Fun ('z', 15); Fun ('w', 57)]

Success!!!

Wrapping up

We learned that, by moving the handcrafted parsers we built in part 1 to using Computation Expressions, we were able to build parsers in a more succinct and expressive way, thanks to the syntax sugar provided by F#. Using this new trick, we ported all parser combinators in Erik/Graham's paper from Haskell to F#, creating a basic set of general purpose parsers.

We are now ready to move to our next challenge: build the JSON parser (finally!). Enough of the good-for-nothing fun language – I hope it did serve its purpose though.

See you next and let me know your comments!

Parsing JSON the fun way series

• Part 1: Building a parser from scratch with Dr. Seuss help (part 1)
• Part 2: Tricking you into Parser Monads (it's not going to hurt, I promise)
• Part 3: The JSON Parser Monad

The boring parsing approach...

The “boring” way of parsing JSON is, as expected, the simplest approach: you just need to define a Record that will hold the parsed data, decorate it with DataContract/DataMember attributes, and use the DataContractJsonSerializer class from the System.Runtime.Serialization.Json namespace to deserialize your JSON string into an F# Record. For example, enter the following snippet in F# Interactive and you will get a nice “Person” object from the JSON string.

 1: #r "System.Runtime.Serialization.dll"
 2: open System
 3: open System.IO
 4: open System.Runtime.Serialization
 5: open System.Text
 6: 
 7: [<DataContract>]
 8: type Person = {
 9:     [<field: DataMember(Name = "Name")>]
10:     Name : string
11:     [<field: DataMember(Name = "Phone")>]
12:     Phone : int
13: }
14: 
15: let jsonString = "{
16:  \"Name\": \"Fernando\",
17:  \"Phone\": 123456789
18:  }"
19: 
20: let js = Json.DataContractJsonSerializer(typeof<Person>)
21: let buffer = jsonString |> Encoding.UTF8.GetBytes
22: let person = js.ReadObject(new MemoryStream(buffer)) :?> Person
23: 
24: printfn "Name: %s, Phone number: %d" person.Name person.Phone
F# Web Snippets

Quite simple, right? And you will have to agree with me that it is a little bit boring too…

Fun with monadic parsers

What is the fun option then? The fun way of parsing some JSON data (or any data format in fact) involves building your own parser from scratch! Better yet, building it from scratch by using Monadic Parsers! Believe me, it is a lot of fun: by the time I was done writing my parser, I had completely forgotten the reason I needed it in the first place.

The process is quite “easy”: (1) learn all about monads; (2) learn yourself some Haskell so you can understand the next step; (3) read Graham Hutton/Erik Meijer paper Monadic Parsing in Haskell; (4) port the Haskell code to F#; (5) compose the JSON parser using the parser combinators; (6) figure out a way to go from the JSON abstract syntax tree to a F# record, and finally (7) use your JSON data!

Step (1) may take from a few minutes to a few years, depending on how deep you want to dive into it; I am not even going to try to cover it here. Interestingly enough, it seems that everybody is talking about Monads these days and there is a bunch of good tutorials and papers out there – read them all. Monads are called Computation Expressions in F#, and for the purposes of this blog series, this WikiBooks entry should get you started on the main concepts: sequencing of computation/function composition and the syntactic sugar the F# compiler provides us to make life easier.

You should not, but if you prefer, you can actually skip steps (2), (3) and (4), and instead look around for code samples to get you up and running quickly. We will go through similar code on this and next posts, so stick around if you feel like it... However, before we get to the parser monads, let’s wobble through some code to get a glimpse of how they work under the covers.

Disclaimer: the parser we will build here is for education purposes only. If you plan on using monadic parsers in the wild, make sure to use a real one like FParsec, which is the F# port of Haskell's Parsec library.

A simple hardcoded parser

The best Parser Monad definition I’ve found so far comes from Dr. Seuss (who would think he was a functional developer!):

A Parser for Things
is a function from Strings
to Lists of Pairs
of Things and Strings!

We could rewrite Dr. Seuss definition as:

A Thing Parser is a function that takes a String as a parameter, and returns a list of Tuples of Things and Strings.

It turns out that Dr. Seuss uses Haskell for his functional programming, and a Haskell String is just a type synonym for a list of characters. We could use System.String in our parsers, but to stick to Erik’s paper convention, let’s use F# char lists instead. The “Thing” in the definition above is just the type variable for the parser… using the F# type system notation, the definition above becomes:

‘a Parser : char list -> (‘a * char list) list

We will be converting strings to list of chars and vice-versa a lot, so let’s create a couple of simple helper functions:

let s2cs = List.ofSeq
let cs2s cs = new System.String(Array.ofList cs)

Pretend we have this programming language that accepts only one line of code: fun <char> = <single digit integer>. One can’t really do much programming with it, but it will be useful for our parser investigation. We can define its simple abstract syntax tree (AST) as:

// fun x = 1
type FunAST = 
    | Fun of (char * int)

Our task will be, for example, to parse “fun x = 1” and emit Fun(‘x’, 1).

Let’s start by building the simplest parser we can think of, a character parser: given a list of characters (a string), it must return a list containing a tuple with the first character and the remaining ones; it does not get more “building a parser from scratch” than this:

let charParser (chars : char list) = 
    match chars with
    | c::cs -> [(c, cs)]
    | [] -> []

F# being such a great language for writing clean/succinct code, and because functional developers love to write clean/succinct code, we can rewrite the function above simply as:

let cParser = function [] -> [] | c::cs -> [c, cs]

Hover your mouse over the function declaration and you will see that it complies with Dr. Seuss definition: val cParser: ‘a list -> (‘a * ‘a list) list! So here we go, our very first parser! Running it on F# Interactive gives us:

> "Test" |> s2cs |> cParser;;
val it : (char * char list) list = [('T', ['e'; 's'; 't'])]

> "T" |> s2cs |> cParser;;
val it : (char * char list) list = [('T', [])]

> "" |> s2cs |> cParser;;
val it : (char * char list) list = []

Note that, no matter which string you feed this parser, it will return a tuple with the first character and the list of remaining ones, all that inside another list; except, of course, if you feed it the empty string, in which case it returns an empty list. It is important to notice the usage of an empty list here, as it represents an error, or the “end of the parsing computation” because the parser failed to parse the input string.

Think of this result list as a “container”: if it is empty, we are done parsing due to an error; if the result list if not empty, and the second element of the tuple is empty, we have successfully parsed the input string. If the list inside the tuple is not empty, we have successfully parsed part of the input string, but we still have more text to parse.

By the way, we can use List.head to extract the content of our result list:

> "Test" |> s2cs |> cParser |> List.head;;
val it : char * char list = ('T', ['e'; 's'; 't'])

Back to our one-line programming language, how would we go about parsing “fun x = 1” using our char Parser above? If all our parser provides is a char at a time, we will need to parse a char, test it with the language specification, parse another one, test it, etc., etc., something like this:

let uglyParser cs = 
    let (c1, cs1) = cParser cs |> List.head
    if c1 = 'f' then
        let (c2, cs2) = cParser cs1 |> List.head
        if c2 = 'u' then
            let (c3, cs3) = cParser cs2 |> List.head
            if c3 = 'n' then 
                let (c4, cs4) = cParser cs3 |> List.head
                if c4 = ' ' then
                    let (c5, cs5) = cParser cs4 |> List.head
                    if c5 = c5 then
                        let (c6, cs6) = cParser cs5 |> List.head
                        if c6 = ' ' then
                            let (c7, cs7) = cParser cs6 |> List.head
                            if c7 = '=' then
                                let (c8, cs8) = cParser cs7 |> List.head
                                if c8 = ' ' then 
                                    let (c9, cs9) = cParser cs8 |> List.head
                                    if  Char.IsDigit c9 then
                                        let num = System.Int32.Parse(string c9)
                                        [Fun (c5, num), cs9]
                                    else []
                                else []
                            else []
                        else []
                    else []
                else []
            else []
        else []
    else []

If you hover your mouse over uglyParser, you will notice that it does comply with our parser definition, and if you run it on F# Interactive you will see that it actually works just fine… however it is as ugly as ugly can be!

> "fun x = 1" |> s2cs |> uglyParser;;
val it : (FunAST * char list) list = [(Fun ('x', 1), [])]

> "fail x = 1" |> s2cs |> uglyParser;;val it : (FunAST * char list) list = []

It is obvious we are repeating code again and again; we apply the char parser, run a function on the result (if c1 = ‘f’ then…) and return a new parser, we apply that new parser, run another function on the result and return yet another new parser, and so one. We are basically “sequencing” parsers manually; it would be great if we could write something like this instead:

continuationFunction(aResult) = process(aResult) and return(bParser)

sequenceFunction(aParser, continuationFunction) =

let (aResult, remaining) = sequence(aParser)
let bParser = continuationFunction aResult
bParser (remaining)

Analyzing the code pseudo-code above, we can try to infer the types of those two functions, remembering our parser type definition:

‘a Parser: char list -> (‘a * char list) list

the types of our functions above become:

continuationFunction: ‘a -> ‘b Parser
continuationFunction: ‘a -> (char list -> ‘b * char list) list

sequenceFunction: ‘a Parser * (‘a -> ‘b Parser) -> ‘b Parser
sequenceFunction: (char list -> ('a * char list) list) * (‘a -> (char list -> ‘b * char list) list) -> (char list -> ‘b * char list) list

Our sequence function takes an ‘a Parser, a function that knows how to go from ‘a to a ‘b Parser and returns a ‘b Parser… Let’s write some real F# code:

let sequence (aParser:(char list -> ('a * char list) list), (continutationFunc : 'a -> (char list -> ('b * char list) list))) cs = 
    match aParser cs with
    | (c', cs')::_ -> 
        let bParser = continutationFunc c'
        bParser cs'
    | [] -> []

What?! Yes, kind of crazy, we upped the game a little bit here and things got slightly more complex, but if you stare at it long enough it will eventually make sense (or give you some headache). The implementation is actually quite simple, we apply the aParser on the input char list (cs), if there is a parsing error, we return the empty list, otherwise we apply the continuationFunc on the thing parsed (c’), which gives us the bParser, and finally we apply the bParser on the remaining char list (ohhh….).

I know we are not using monads yet, but just to get us closer to that, I’d like to rename and rewrite the sequence function above before using it. I will call it bind and use a different notation, by removing the cs from its declaration, to emphasize that bind returns a function (the behavior continues exactly the same):

let bind (p:(char list -> ('a * char list) list), (f : 'a -> (char list -> ('b * char list) list))) = 
    fun cs -> 
        match p cs with
        | (c', cs')::_ -> (f c') cs'
        | [] -> []

With this new notation, it easier to see that bind is returning a parser, which is the expected behavior: if I sequence, or bind, two parsers, the result has to be another parser, right? With bind in place, let’s write a parser that parses the token “fun” out of “fun x = 1” using it:

let uglyFunParser input = 
    bind (cParser, fun c1 ->
        if c1 = 'f' then
            bind(cParser, fun c2 ->
                if c2 = 'u' then
                    bind(cParser, fun c3 ->
                        if c3 = 'n' then
                            fun cs -> [Fun, cs]
                        else fun _ -> []
                    )
                else fun _ -> []
            )
        else fun _ -> []
    ) input

> "fun x = 1" |> s2cs |> uglyFunParser;;
val it : ((char * int -> FunAST) * char list) list =
  [(<fun:uglyFunParser@73-20>, [' '; 'x'; ' '; '='; ' '; '1'])]

It works perfectly, but still quite ugly... However, we now have in place the basis of sequencing those parsing computations. It is very important that you understand what’s going on here… we are using a lot of anonymous functions instead of explicitly declaring several continuationFuncs, and each of these anonymous functions binds, or sequences, another parser.

“What’s up with all those “fun _ -> []” and “fun cs -> [Fun, cs]”? Remember that any parser we build must comply with Dr. Seuss definition, they are functions that take a char list (cs) and return an empty list (error) or a list with the tuple (thing parsed * remaining char list); and that’s exactly what we are doing, in case of error we return a function that takes one parameter (but we don't care about it, so we throw it away with the "_" notation), and returns the empty list (“fun _ -> []”), and in case of success we return “fun cs -> [(Fun, cs)]”, i.e., a function that takes a char list and returns the tuple (parsed thing (Fun), remaining chars).

These two anonymous functions are so important that we ought to build helper functions for them, let’s call them “returnParser” for the success case, and “zeroParser” for the case of parsing errors:

let returnParser something = fun cs -> [something, cs]
let zeroParser () = fun _ -> []

It is also time to get rid of the if-then-else madness of our ugly function! We will start by defining a “satisfy this predicate Parser”, the idea being, given a predicate function (‘a -> bool), it returns a parser which can only parse things where the predicate returns true – it is almost a direct copy-and-paste of one section of the funUglyParser above:

let satisfyParser pred = 
    bind(cParser, fun c ->
        if pred c then returnParser c else zeroParser()
    )
    //let (c', cs') = cParser cs |> List.head
    //if pred c' then [c', cs'] else []

Make sure to hover your mouse over it and confirm its type complies with our parser definition. With this new parser in place, let’s create an “only parse this specific character Parser”, and call it “tcParser” (as in "this char Parser"):

let tcParser c = satisfyParser ((=) c)

Kind of cool, no? Testing it:

> "fun x = 1" |> s2cs |> tcParser 'f';;
val it : (char * char list) list =
  [('f', ['u'; 'n'; ' '; 'x'; ' '; '='; ' '; '1'])]

And it works… how about sequencing this little guy to improve our funParser?

let funParser (input: char list) = 
    let internalParser = 
        bind (tcParser 'f', fun _ ->
            bind (tcParser 'u', fun _ ->
                bind (tcParser 'n', fun _ ->
                    returnParser Fun
                )
            )
        )
    internalParser input

The input parameter is really not helping much here, by this time we all know that parsers take char lists as input, so let’s remove it:

let funParser' = 
    bind (tcParser 'f', fun _ ->
        bind (tcParser 'u', fun _ ->
            bind (tcParser 'n', fun _ ->
                returnParser Fun
            )
        )
    )

> "fun x = 1" |> s2cs |> funParser';;
val it : ((char * int -> FunAST) * char list) list =
  [(<fun:funParser'@100-3>, [' '; 'x'; ' '; '='; ' '; '1'])]

> "fail x = 1" |> s2cs |> funParser';;
val it : ((char * int -> FunAST) * char list) list = []

You have to agree again that this is much cleaner than our very first uglyParser. We are not even using the zeroParser anymore, since satisfyParser takes care of that for us. But we are now facing another pattern, sequencing the application of the tcParsers. Can we do better than that? Sure thing… let’s create a “parse this specific string Parser”:

let stringParser (tok:string) =
    let rec text = function
        | [] -> returnParser []
        | t::tok' -> 
            bind(tcParser t, fun _ ->
                bind(text tok', fun _ -> returnParser (t::tok') )
            )
    text (s2cs tok)

> "fun x = 1" |> s2cs |> stringParser "fun";;
val it : (char list * char list) list =
  [(['f'; 'u'; 'n'], [' '; 'x'; ' '; '='; ' '; '1'])]
> "fail x = 1" |> s2cs |> stringParser "fun";;
val it : (char list * char list) list = []

Given a token (tok), we iterate through its characters, sequencing parsers that will parse that specific character (t). The result will be a parser (a function) that will parse a specific string out of the input string, or fail if it can't parse the string. How would we use it in our funParser? We just sequence it…

let funParser'' = 
    bind(stringParser "fun", fun _ -> returnParser Fun)

Now we are talking! How about parsing the whole line “fun x = 1”? Let’s put together a few more “parser combinators” to help on that task… Every time we parse a token, we need to remove the white space after the token, so let’s build a “cleanSpaceParser” which will sequence a parser, sequence a “spaceParser”, throwing its result away, and return the result of the first parser (see below) - using cleanSpaceParser, we can now build our final parsers for our single line language:

let singleSpaceParser = tcParser ' ' 

let cleanSpaceParser p = 
    bind(p, fun r ->
        bind(singleSpaceParser, fun _ -> returnParser r)
    )

let finalFunParser = cleanSpaceParser funParser''

let identParser = cleanSpaceParser cParser

let equalParser = cleanSpaceParser <| tcParser '='

let digitParser = 
    bind(satisfyParser System.Char.IsDigit, 
        string >> System.Int32.Parse >> returnParser
    )

If we now put them all together, we end up with the parser for our language:

let funLangParser = 
    bind(finalFunParser, fun funFunc ->
        bind(identParser, fun identifier ->
            bind(equalParser, fun _ ->
                bind(digitParser, fun digit ->
                    returnParser (funFunc(identifier, digit))
                )
            )
        )
    )

Hover your mouse again over the code above to make sure you understand what’s going on! Does it make sense?

We can use a helper function to run the parser and return the results, for example:

let runParser s = 
    match funLangParser (s2cs s) with
    | [] -> failwith "Error parsing string"
    | (result,_)::_ -> result

> "fun x = 1" |> runParser;;
val it : FunAST = Fun ('x', 1)
> "fun y = 2" |> runParser;;
val it : FunAST = Fun ('y', 2)
> "fail x = 1" |> runParser;;
System.Exception: Error parsing string

It works (and looks way better than our first uglyParser)! We have just built a parser from scratch, congrats!

Wrapping up 

“Wow! That was a lot of code (and craziness) to just parse a single line of code”, you might be thinking. And yes, it was indeed; but as I mentioned, we now have the building blocks to create more powerful parsers, including the JSON parser we talked about at the beginning of this post.

We started with a single character parser based on Dr. Seuss definition, and through the “power” computation sequencing/composition, we built more and more capable parsers. The bind, zeroParser and returnParser functions were crucial on making the compositions easier, and the use of lists as the parser’s output provided a way to return both parsing success as well as end of computation due to an error.

The whole concept of functions being passed around can be a bit intimidating at first, especially if you are new to the functional lifestyle; F# Interactive can be a huge help in the learning process here. F# will also make things easier for us when we move to using computation expressions; for example, our funLangParse above could have been written as something like this:

let funLangParser =
 parser {
  let! funFunc = finalFunParser
  let! identifier = identParser
  let! _ = equalParser
  let! digit = digitParser
  return funFunc(identifier, digit)
 }

That is exactly what we will do next: wrap our parsers into computation expressions and then build the JSON parser. Later we will figure out ways to create F# types based on the AST generated by our parsers.

Stay tuned!

Homework

The cleanSpaces parser only parses a single white space out of the input string. Can you rewrite it to parse any number of spaces in a way that a line like this, “fun      z     =    7” would be successfully parsed?

Parsing JSON the fun way series

• Part 1: Building a parser from scratch with Dr. Seuss help (part 1)
• Part 2: Tricking you into Parser Monads (it's not going to hurt, I promise)
• Part 3: The JSON Parser Monad

Here is some information on how to run Cellular Emulator and configure emulator image's network settings to browse to the Internet. Once you configure the emulator images, you can save their states (from Device Emulator) to speed up the process.

Cellular Emulator

Cellular Emulator is a software-based emulator to aid developers and testers in developing and testing their software under the Windows Embedded CE and Windows Mobile platforms. The design goal of Cellular Emulator is to replace the radio module in both development and test environments. The advantage of Cellular Emulator is that it provides not only voice but also data connectivity. Moreover, Cellular Emulator is a powerful tool to test various applications under different wireless network conditions in GSM/GPRS and/or UMTS networks.

Running Cellular Emulator

1. Launch the Cellular Emulator (Start/All Programs / Windows Mobile 6 SDK / Tools / Cellular Emulator ) and the Device Emulator.
2. Read the COM port configuration from the status bar of the Cellular Emulator main window.
3. In the Device Emulator, go to File / Configure and select the Peripherals tab.
4. Map the Serial port 0 of Device Emulator to the specific COM number obtained from step 2
5. Soft reset the Device Emulator (File / Reset / Soft).

Shutting the Cellular Emulator

The close button of the Cellular Emulator main windows will only send the application to the tray panel. To close the application, right click the tray icon and select Exit.

Cellular Emulator Data Connections

The Cellular Emulator will also provide GPRS connection simulation. Users can create simulated GPRS connections just like common ones. The access point name and user name/password are not checked and thus may be anything. Once the data connection is made, the emulator will just act as if it is connected to the network of the host machine. If some proxy settings are needed for the host machine to set up connections to remote machines, they are also needed on the emulator. Assume a network environment where HTTP proxy is needed to access external web sites. Setting samples on Windows Mobile 6 Standard and Professional are provided in the following sections.

1. Enter Start-Settings-Connections-GPRS-Menu-Add. Create a GPRS connection using the settings as follows.
   • Name: PPP
   • Connects to: WAP
   • Network Access Point: Some Access Point Name
   • User Name: Just leave blank for anonymous configuration
2. (Optional) Enter Start-Settings-Connections-Proxy-Menu-Add. Create a proxy using the following settings.
   • Description: Proxy
   • Connects from: WAP Network
   • Connects to: The Internet
   • Address: The proxy in the specified user's network
3. Enter Start-Settings-Connections-Menu-Advanced.
   • Set Internet connection to PPP
   • Set WAP connection to PPP
   • Leave the others as automatic
4. Enter Internet Explorer Mobile, and choose Menu-Tools-Options-Connections, then configure the Internet Explorer Mobile settings:
   • Automatically detect settings: CHECKED
   • Select network: The Internet
5. Choose the Internet Explorer Mobile icon using the arrow key on the keyboard.
6. Browse freely using Internet Explorer Mobile.

1. If the user's computer uses a proxy to access the Internet, then the user needs to setup a Work Connection on the Pocket PC, otherwise the user can setup an Internet Connection.
2. Setup a GPRS connection. Access point and user name can be empty.
3. (Optional) Setup a proxy. This should be the full name of a working proxy required by the corporate network.
4. Open Internet Explorer Mobile and browse to a web site.

Makeover

Out of curiosity, Cellular Emulator went through an extreme makeover from the first time I learned about it as an internal tool, to its release in the Windows Mobile 6 SDK.

I hope you enjoy it and let me know your feedback!

Before:

After:

Could not access network location Common7\IDE\ProjectTemplates

This issue is related to a Custom Action (VBScript) we were using to install the Project Templates for Visual Studio - Vista may be blocking VBScript (through security policy, for example) which then causes our installation to fail. If you log the SDK setup, you will see something like:

DEBUG: Error 2738:  Could not access VBScript runtime for custom action

While we work on that for the SDK Refresh, here is a simple workaround that will allow you to install the SDKs:

1. Click on Start / All Programs / Accessories;
2. Right click on Command Prompt and click on Run as administrator;
3. Click on Continue in the User Account Control pop-up;
4. In the Command Prompt window run the following command: regsvr32 %SystemRoot%\system32\vbscript.dll
5. Close the Command Prompt window and install the SDKs normally.

Please, let me know if that works for you (and yes, we don't like VBScript CAs anymore too! :-)

See you at MEDC 2007!

Please, read carefully the download page details to learn about the new naming and emulator images, requirements and know issues. Make sure to also read the What's New for Developers in Windows Mobile 6 white-paper.

The table shows which SDK must be used when targeting the old Smartphone and Pocket PC categories:

What’s new in the Windows Mobile 6 SDK?
Here is a quick overview of the new SDKs.

Better discoverability
Everything is installed (by default) under C:\Program Files\Windows Mobile 6 SDK. We also “share” the common content from both SDKs, for example, there will be only one help collection, only one set of common samples, tools, etc., even if you install both SDKs. Readme files are also better, go to Start Menu, Programs, Windows Mobile SDK and check the main readme file as well as the tools and samples ones.

Help Collection
The help is now integrated with DocExplorer, but you can still access it through the Start Menu if you want/need to.

Samples
We have samples! A lot of them! Check the Samples readme file to learn about all sample collection.

New Tools
There are also a nice number of new tools with this release. Here is a quick list with a brief explanation of each one:

Cellular Emulator v1
Cellular Emulator is a software-based emulator to aid developers and testers in developing and testing their software under the Windows Mobile platforms. The design goal of Cellular Emulator is to replace the radio module in both development and test environments. The advantage of Cellular Emulator is that it provides not only voice but also data connectivity. Moreover, Cellular Emulator is a powerful tool to test various applications under different wireless network conditions in GSM/GPRS and/or UMTS networks.

In other words: it is your own Mobile Operator running on your desktop :-). You can call your device emulator, send and receive SMS, play with AT commands, SIM cards, etc. How cool is that?

Device Emulator v2
Yes, it is here – install the SDK and have Device Emulator v2 up and running on your machine.

Local Server Framework
The Local Server Framework allows users to develop custom server applications running over localhost. Components of Windows Mobile powered devices must communicate with external services running on external servers. Lack of control over the external server inhibits the ability to comprehensively test client side code. Additionally, varying network conditions in build labs introduce multiple points of failure in test code that are often unrelated to code under test. The server framework allows development of test application servers that run locally over localhost on the device, alleviating these problems. The Local (or Fake) Server Framework allows users to develop either full-featured servers or application-specific servers that communicate with their respective clients.

FakeGPS & GPS Settings
Do you want to develop a GPS-enable application but don’t have a real GPS device? FakeGPS is the answer! Just drop and install the FakeGPS.cab file into your emulator or device and you are ready to go. FakeGPS will set the GPS Intermediate Driver to read the NMEA strings from a text file instead of using a real GPS device.

And what happens if you want to sell your GPS-enabled applications to WM 6 Standard users? You can use GPSSettings to enables Windows Mobile 6 Standard users to configure the GPS intermediate driver, it works in the same way as the built-in GPS settings applet on Windows Mobile 6 Professional and Windows Mobile 6 Classic – better yet, you can even redistribute GPSSettings.exe with your applications!

Hopper
Hopper is a software test tool that simulates input stress on Windows Mobile powered devices. Hopper will stress all applications that are available through the menu system by rapidly sending keystrokes and screen taps in a random fashion. By sending a large number of user inputs very rapidly, Hopper can quickly isolate troublesome scenarios and find bugs in your applications.

It works like this: you setup Hopper to test your application and let it run. It keeps clicking around randomly on your application going through “code paths” that you would not usually test, and of course, it finds bugs!

New Emulator Image
A brand new “Windows Mobile 6 Professional Square QVGA (320x320 pixels - 128 dpi)” emulator image!

CabSignTool
CabSignTool signs a .cab file and all its executable content (.exe, .dll) with the specified certificate – it unpacks the .cab file, signs all .exe and .dll files, pack the .cab file again and sign it, in one single pass.

How do you like it? Stick around as I will continue posting about the SDK, tips and tricks, issues, etc. And let me know your comments and feedback.

Enjoy the new Windows Mobile 6 SDKs!

It gives me great pleasure to announce the following changes that we made in WM6:

• Certificate Installer built into the platform 
• Installs CER, P7B, and PFX files 
• No more Access Denied messages. 
• Installs certs to the ROOT, Intermediate, and MY store
• Wildcard Certificate support for SSL

Full post here.

What’s new for developers?

1. Great application compatibility for Windows Mobile 5 applications
2. .NET Compact Framework v2 SP1 in the ROM!
3. SQL Server Compact Edition in the ROM!
4. New APIs
• New Sound APIs
• WISP Lite (Yes, the same WISP technology from the TabletPC!)
5. Several new tools:
• Cellular Emulator (kind of having your own Mobile Operator ;)
• Local Server Framework (aka FakeServer)
• FakeGPS
• Hopper
• CabToolSigner
• Security Configuration Manager
6. New emulator images
7. Platform improvements
8. Integrated documentation

Overall Windows Mobile 6 SDK makes it easier to develop, build, test and deploy your applications.

I’ll drill down into the SDK details on my next posts – so stay tuned!

I'm proud to announce ScreenLib, a new library for native Windows Mobile developers. It takes away a lot of the pain of designing user interfaces for multiple screen sizes, orientations, form factors etc. It lets you create a user interface once and have it automatically adapt to whatever the device’s screen size is at runtime. By doing this, it offers basic docking & anchoring support for native development and can do a lot of UI plumbing work with just 1 or 2 lines of code.

ScreenLib works on Pocket PC and Smartphone devices, so it's a great step towards creating single binaries that will run on both platforms.

I highly recommend watching a video tutorial that explains how ScreenLib works and why it should be used. To download the video, please visit:
http://www.microsoft.com/downloads/details.aspx?FamilyID=a908dd7f-71c0-4cea-b97d-b9ffe985f903&displaylang=en.

To download ScreenLib for free, please click: ScreenLib.zip

He has presented it here a couple of weeks ago and I was impressed by its simplicity and how it can increase your productivity when dealing with screen orientation/resolution, etc. And as Mel says, we’d like to hear your feedback!

Thanks Mel!

Network Analyzer for Windows Mobile was released today as a PowerToy:

Overview
Network Analyzer for Windows Mobile runs network utilities, for example ping and ipconfig, on a Windows Mobile powered device. Network Analyzer for Windows Mobile facilitates the troubleshooting of network connectivity issues. You can extend the harness. You can add user-defined tests (DLLs) to the list of tests to be executed. An xml input file defines the list of tests to execute. You can use Network Analyzer to send information about network traffic to a .cap file. You can then view the .cap file with the Network Monitor tool or the Ethereal tool. See the readme file below for more information.

You can download it here.

Enjoy it!