Now without further ado:

1. Microsoft Codename "Data Explorer"
   
   Data explorer is a new cool tool (either client or as a preview-service in the cloud) to take in arbitrarily shaped data and mix and match and turn and twirl it around as you please. Do you want to peek on the price of a cool gadget you wish to buy of your beloved online retailer?! Take Data explorer use your query as input source, apply some transformations to it and get a list of providers along with their suggested price as output. Or maybe you want to know what somebody said on twitter about your cool project, well guess what there is Data Explorer to the rescue.     
   
2. Microsoft Codename “Cloud Numerics”
   
   Do you want to be the next cloud data scientist of the year? Well now you can … write and use the analytics models you imagined but weren’t able to get converging because it took so long. Do you want to design extensive experiments for your company leveraging first class cloud power to drive generation of real value from data? Well now you can!
   
3. Microsoft Codename "Social Analytics"
   
   These days, it’s all about social. Whether it is connecting and sharing with the people you know or as a company finding out about your customers. The social revolution is only starting. If you want to get started on developing in the social context, this lab might be for you:
   
   Microsoft Codename "Social Analytics" is an experimental cloud service. It’s aimed at developers who want to integrate social web information into business applications.
   
4. Microsoft Codename "Data Transfer"
   
   How do you actually get data into the cloud and perform this jucy analysis on it that I was writing about above? Don’t worry there is a pair of shoes just right for you:
   
   Microsoft Codename "Data Transfer" is an easy-to-use Web application to move your data into SQL Azure or Windows Azure Blob storage.
   
5. Infer.NET Fun: An F# Library for Probabilistic Programming
   
   F#’s strong suit is the expression of non-trivial data transformation as it is needed in technical and scientific computing. Clearly, you can built abstractions for reasoning on and learning from graphical  models all by yourself. But why not take more advantage of the smart folks over at MSR (yes F# is a lot of MSR love) ?! Well you can with Infer.NET Fun:
   
   FSoft is an F# library for probabilistic programming. It enables you to write generative models as functional programs, run them to obtain samples, and quote them to infer distributions via Infer.NET.
   
6. MySemanticSearch Example on CodePlex
   
   SQL Server 2012 is almost here (in RC form it is) and with it it’s new semantic search capabilities will arrive as well. Just imagine the vector-space model on stereoids instead of stemming and matching it’s all about taking care of natural language’s fuzziness using statistics and similarities.

With that, I’ll leave you to your experimentation – read you soon.

This blog post aims much lower in that I want to use a tool chain, i.e. F# and Wordle to aggregate a document into something that can be analysed by the (currently available) best processor – the human brain. So, assuming a text is made up of words that represent the topic(s) of the text, those words that appear frequently might help us to hypothesise about the topic(s) of the text.

Now, the problem is phrased in such a way that it actually gives away the simple and unrealistic solution:

1. Extract words from text.
   
   A word is a sequence of letters of an alphabet.
   For simplicity’s sake, an english word is made up of only letters. That is, we do not accept contractions, clitics, named entities (like R2D2, etc.), numbers, etc.. Words are delivered as a sequence line by line.
   
2. Remove words that are invalid (with respect to point (1.)).
3. Transform words into a common format, i.e. lower case without whitespace at pre/post-fix position.
4. Stop words, i.e. frequent words increasing noise in a text, will be removed.
5. Count each word, i.e. whenever a word occurs we give it a one (remember how we used to count as a kid …).
6. Output the count of each word, i.e. “<word> 1”.

If you feel like you have seen this a lot recently – Yes, it looks and feels like the Hadoop “Hello World” example. Consequently, the code is similar and laid out in such a way that you can use it in a Hadoop streaming scenario, i.e. the code is put into a container/ program "called FMapper, which takes as input from stdin:

   1:    let invalidWordsReg = Regex( "^\d+$|^\d+\w+$|^\w+\d+$|^\d+.+$|\s+|(�|©|@|®)+" )

   2:    let mutable line = ""

   3:    while line <> null do

   4:        

   5:      line <- Console.ReadLine()

   6:   

   7:      if (String.IsNullOrEmpty(line) = false) then

   8:   

   9:        let words = line.Split([|" "; "/"; "\\"; ","; "."; ";"; "?"; "!"; ":"; 

  10:          ")"; "("; "<"; ">"; "["; "]"; "}"; "{"; 

  11:          "+"; "-"; "*"; "%"; "_"; "#"; "&";

  12:          "§"; "$"; "~"; "|"; "^";

  13:          "\""; "'"; "="; "”"; "“"|], StringSplitOptions.RemoveEmptyEntries)

  14:   

  15:        words

  16:        |> Seq.map(fun word -> word.Trim() )

  17:        |> Seq.map(fun word -> word.Replace("”", "").Replace("“", ""))

  18:        |> Seq.map(fun word -> word.ToLowerInvariant())

  19:        |> Seq.filter(fun word -> (invalidWordsReg.IsMatch word ) = false)

  20:        |> Seq.filter(fun word -> ( StopWords.isEnglishStopWord word ) = false)

  21:        |> Seq.map(fun w -> (w, 1))

  22:        |> Seq.iter(fun (w, c) -> printfn "%s\t%d" w c )

  23:   

  24:    done

As you can see, the sample follows the imperative style (while loop). However, you might use a recursive function peeking on stdin and calling the word processing pipeline whenever there is a line of word(s) available to be processed. Is it necessary to output a 1 for each word? For the simple example, that is discussed here – NO. However, instead of outputting each single word, you might think about computing some summary statistic from that word, group of words, sentence, etc. For example, we might group the word sequence using a word as a key. Furthermore, imagine the distributed (Hadoop streaming) case, where local knowledge (each map step) has to be integrated to form global knowledge (the reduce step).  Consequently, the simple mapper step serves two points:

• Performing analysis (a very simple one in this case)
• Imposing a schema, i.e. from a non-relational text input to relational output “<word>, <count>”

Now, the above program executed on a sample text taken from a SQL Server 2008 help page yields the following output, which illustrates the point of stop words. Yes, we did remove some of them but depending on your application domain you might even remove different ones or more.

What are we left with? We need to integrate the individual word counts. The following “Reducer” program takes care of this using a Dictionary. Global word counts are printed to standard output so that they can be used for further computation.

   1:    let mutable line = ""

   2:    let xmap = Dictionary<string,int>()

   3:   

   4:    while ( line <> null ) do

   5:        

   6:      line <- Console.ReadLine()    

   7:   

   8:      if String.IsNullOrEmpty(line) = false then

   9:        let vLine = line.Split([|"\t"|], StringSplitOptions.RemoveEmptyEntries)

  10:   

  11:        if Array.length vLine <> 2 then failwith "invalid format of input data"

  12:   

  13:        let key = vLine.[0]

  14:        let cnt = Int32.Parse(vLine.[1])

  15:   

  16:        if ( xmap.ContainsKey key ) then

  17:          let v = xmap.Item( key )

  18:          let newv = v + cnt

  19:          xmap.Item(key) <- newv

  20:        else

  21:          xmap.Add( key, cnt )

  22:   

  23:    done

  24:   

  25:    xmap

  26:    |> Seq.iter(fun kvp -> printfn "%s:%d" kvp.Key kvp.Value )

Now, we can test the tool chain by invoking the following command line:

FMap.exe < "C:\TestData\SQLTestDocument.txt" | FReduce.exe

The result of this computation can be seen in the following:

The last step left from our discussion is Wordle. That is, we feed those <word,count> pairs to Wordle. Either by going to the web site or by using the code snippet I presented previously. Either way - our result looks something like this:

Now, simply by looking at the word cloud, we might conclude that the (unknown) document we analysed, might contain information about a “server” from “microsoft”, which has something to do with “sql”, “reporting” among other “services”. Maybe we could use this to query our search engine of choice, leading to maybe this result (or read the document ):

Closing remarks:

The method presented is far from a real world example of automatic summarisation, customer sentiment analysis and other computational problems, which aim to bring structure and/or sense to natural language documents. The meaning of and relationship between words was totally ignored. For example:

1. SQL is an acronym for structured query language but if both occur in a document, we have 4 words (SQL, structured, query, and language) instead of just on representative for SQL.
2. No handling of synonyms. Assuming the text contained some word and a lot of (different) synonyms for that word, depending on the frequency of the word and its synonyms, we might or might not have noticed the topic the word and its synonyms represent.
3. To be or not to be. That is, the small word not (or no, etc.) reverses a statement (negation) or is used to emphasise a statement ( rhetorical question ). Now assuming, each time our high frequency words occurred with a negation, we would still have gotten high counts for those words but the documents topic might be opposite to our impression.
4. A more realistic solution would make use of machinery, such as LSA, PCA, tf-idf and the vector space model (as a common metaphore).

Takeaway:

Natural language processing, F# and Wordle are fun and go together excellently - Read you soon!

Wordle is a toy for generating “word clouds” from text that you provide. The clouds give greater prominence to words that appear more frequently in the source text. You can tweak your clouds with different fonts, layouts, and color schemes. The images you create with Wordle are yours to use however you like. You can print them out, or save them to the Wordle gallery to share with your friends.

But since (often) a picture says  more than … See the following (small) image of what Wordle might do for you:

Now, Wordle may generate word clouds by poiting it to your word source or by posting a collection of "(frequency-weighted) words to the wordle site. Consequently, this is what the following snippet is all about. It is based on the MSDN article “How to: Send Data Using the WebRequest Class”. I followed the imperative blue print and did not use F# heavyweights like async.

 1: open System.Net

 2:  

 3: let words = ""

 4:  

 5: let worddleUrl = "https://www.wordle.net/advanced"

 6:  

 7: let r = System.Net.WebRequest.Create(worddleUrl)

 8: r.Credentials <- CredentialCache.DefaultCredentials

 9: r.Method <- "POST"

 10: r.ContentType <- "application/x-www-form-urlencoded"

 11: let s = r.GetRequestStream()

 12: let wordsEncoded = System.Text.Encoding.UTF8.GetBytes ("text="+ words);

 13: let wordsLen = ( Array.length wordsEncoded )

 14: s.Write(wordsEncoded, 0, wordsLen)

 15: s.Close()

 16:  

 17: let response = r.GetResponse()

 18: let responseStream = response.GetResponseStream()

 19: using ( new System.IO.StreamReader(responseStream)) (fun sr ->

 20:   let responseText = sr.ReadToEnd()

 21:   using (System.IO.File.CreateText("/response.htm")) (fun wr ->

 22:     wr.Write(responseText)

 23:   )

 24: )

The code is quite simple, supply space-delimited words in the words variable (e.g., "hello world”) or you might even use frequency-weighted words using the syntax “word:frequency” where frequency is a natural number like 3. The rest of the code follows the mentioned MSDN sample, i.e. using a WebRequest to contact the Wordle service, waiting for a response and writing the response to a local html file. Now, you might have expected that the result from a Wordle call is an image representation of a word cloud. This is not the case, instead what you get is an HTML with the Wordle Java applet, which when started will actually render your word cloud. Now, what you get is a “suggestion” or one version to render your words as a word cloud. However, the applet provides some level of customisability, i.e. changing word-placement strategy, color, font, etc. Lastly, you might want to print your word cloud, for example, as a pdf. Unfortunately, you cannot export the word cloud into an image container but you may use a screen-shotting device to achieve a similar result.

With that happy word clouding with the excellent Wordle.

Now, MSR and more specifically the Research in Software Engineering Team (RiSE) of MSR made some of the known (and some of the lesser known ones) applications and libraries available in the same convenient, easily accessible Pex style, i.e. via a web interface.

Just head over to RiSE4fun and try your favorites: pex, rex, spec# or z3. The site has a twitter-like real-time feed called live and more importantly supports a REST API. Using this API you may, much like Pex which already exists on Windows Phone 7, build your WP7 variant of a RiSE app or you may use a RiSE tool like the Stanford Parser - online.

With that, enjoy and put the fun in exploring RiSE4fun.

For example, the following output (html file) was obtained by querying my own machine:

As mentioned, the script itself uses simple Powershell techniques like WMI, html and csv handling. Beware that actual script comments and output is in German. Invoking the script is a simple matter, since default arguments are provided.

   1: #param:

   2: #    machineName     - einzelner Rechnername, oder kommaseparierte List von Rechnern => X oder X,Y

   3: #    usageTreshold    - Wert zwischen 0 und 1, Schwellwert für Auslastung des Festplattenspeichers in %

   4: #    htmlFile        - HTML Datei, die zur Dokumentation eingesetzt werden kann

   5:  

   6: param(

   7:     [string]$machineName = "NONE",

   8:     [float]$usageTreshold = 0.11,

   9:     [string]$htmlFile = "C:\environment.html"

  10: );

  11:  

  12: function CheckParam(){

  13:     if ([System.String]::IsNullOrEmpty($machineName) -eq $true) { throw "Rechnername darf nicht null oder leer sein!" ; }

  14: }

  15:  

  16: function GetSpaceOnMachine([string]$machine, [ref]$HTMLRepository) {

  17:     write-host "WMI - GetSpace on `"$machine`" ... " -foregroundcolor gray; 

  18:     $private:message = "";

  19:     $private:MBFrac = 0.00000095367431640625 ; 

  20:     

  21:     (get-wmiobject win32_logicaldisk -namespace "root\cimv2" -computername "$machine" ) | ? { $_.DriveType -eq 3 ; } | % {

  22:     

  23:         $private:name = $_.Name;

  24:         $private:size = $_.Size;

  25:         $private:unused = $_.FreeSpace;

  26:         $private:sizeMB = [System.Math]::Round($size * $MBFrac);

  27:         $private:unusedMB = [System.Math]::Round($unused * $MBFrac);

  28:          

  29:         $private:usage = [double]$unused / [double]$size ;

  30:         $private:usagePercent = [System.Math]::Round($usage * 100.0);

  31:         if ($usage -lt $usageTreshold) {

  32:             $message = "Festplatte $name auf $machine ... weniger als $usagePercent% frei ($unusedMB MB von $sizeMB MB frei) !" ; 

  33:             write-host $message -foregroundcolor red ;

  34:         }

  35:         else {

  36:             $message = "Festplatte $name auf $machine ... mehr als $usagePercent% frei ($unusedMB MB von $sizeMB MB frei) !" ;

  37:             write-host $message -foregroundcolor green ; 

  38:         }

  39:         

  40:         $HTMLRepository.Value.Add( ($machine + " (" + $name + ")" ), $message);

  41:     }    

  42: }

  43:  

  44: function Main(){

  45:     cls ;

  46:     

  47:     CheckParam ;

  48:     

  49:     if ( ($machineName.ToUpperInvariant() -eq 'NONE') -eq $true) {

  50:         write-host "Rechnername `"NONE`" wird auf hostname gesetzt ... "; 

  51:         

  52:         $machineName = hostname ;

  53:     }

  54:     

  55:     [hashtable]$private:HTMLRepository = @{};

  56:     

  57:     if ( $machineName.Contains(",") -eq $true ) {

  58:         write-host "Collection von Rechnern wird abgefragt ... " -foregroundcolor white ;

  59:         

  60:         ([System.Text.RegularExpressions.Regex]::Split($machineName, ",") ) | % {

  61:             GetSpaceOnMachine $_ ([ref]$HTMLRepository);

  62:         }

  63:     }

  64:     else {    GetSpaceOnMachine $machineName ([ref]$HTMLRepository) ;  }

  65:     

  66:     if ( [System.String]::IsNullOrEmpty($htmlFile) -eq $false ) {

  67:     

  68:         if ( (test-path $htmlFile) -eq $true ) { del $htmlFile -force ; }

  69:     

  70:         write-host "Exportiere Ergebnis in HTML Datei ... " ; 

  71:         add-content ("`"Rechner (Festplatte)`", `"Status (" + [System.DateTime]::Now + ")`"") -path "C:\temp.csv" ; 

  72:         $HTMLRepository.Keys  | % { 

  73:             $private:val = $HTMLRepository[$_];

  74:             $private:key = $_ ; 

  75:             add-content "`"$key`", `"$val`" " -path "C:\temp.csv" ; 

  76:         }

  77:         import-csv "C:\temp.csv" | convertto-html -title ("Freier Festplattenplatz in Umgebung (" + [System.DateTime]::Now + ")" )  > $htmlFile ;

  78:         

  79:         $x = (get-content $htmlFile)

  80:         del -force $htmlFile ;

  81:         $x | % { $_ -replace "</head>", "<style type=`"text/css`"> td { text-align:left; border:1px solid #000; vertical-align:top; overflow:hidden; } th { background-color: black; color: white; text-align:left; } table { width:100%; }</style></head>" } | add-content $htmlFile ;

  82:     

  83:         

  84:         del "C:\temp.csv" ;

  85:     }

  86: }

  87:  

  88: Main ;

pushOnto(elem, stack)

I opted for something more readable. Luckily, hjgher order F# functions, i.e. functions that take parameters, which are themselves functions, come to the rescue, As you may see, from the snippet below:

module Test

let onto s = s

type 'a Stack = 'a List

let EmptyStack = List.empty

let push a (ontostack:'a list -> 'a list) (st:'a list) =
    let pushx stack elem = 
        elem::stack
    pushx (ontostack st) a

let mutable myStack = EmptyStack

myStack <- push 'L' onto myStack 
myStack <- push 'O' onto myStack 
myStack <- push 'O' onto myStack 
myStack <- push 'C' onto myStack 

printfn "%A" myStack

printfn "press <any> key to exit"
System.Console.ReadLine() |> ignore

A call sequence like push <Element> onto <Stack>

reads much more natural than the function call presented above, or a variant like:

elem |> push stack

Clearly, the stack example is trivial, but imagine cases like the ones covered by FsUnit or applications to scientific or any domain-specific computing. Instead of letting the technicality of the implementation language shine through, domain-specific concepts and abstractions can be implemented to such a degree that domain-savy experts my (unknowingly) use F# to solve complex business problems.

With that … read you later.

Today’s post is merely a link dump, of things I found while digging through my rss archive:

1. Lync Conversation Translator
   
   Lync-Guistic, also known as Microsoft Lync 2010 Conversation Translator, allows you to send and receive instant messages with Microsoft Lync using the Microsoft Bing translation service. Bilingual users can provide translation feedback to help improve the service.
   
2. Dev Labs: Debugger Canvas:
   Debugger Canvas is a new user experience for the debugger in Visual Studio Ultimate. It pulls together the code you’re exploring onto a single pan-and-zoom display. As you hit breakpoints or step into code, Debugger Canvas shows just the methods that you’re debugging, with call lines and local variables, to help you see the bigger picture.
   
   
3. Multiplayer Gaming for Smartphones:
   
   Gaming on a smartphone comes with compromises, though. Most smartphone-based games are single-player games—Angry Birds is a perfect example—or turn-based, in which two or more players queue up on a single phone to play games.

Bahl and a team of Microsoft Research scientists and interns created Switchboard, a cloud service that enables smartphones to link to each other for multiplayer gaming. Joining Bahl on the project were Justin Manweiler, an intern, and Romit Roy Choudhury, a visiting researcher, both from Duke University; and Sharad Agarwal and Ming Zhangfrom Microsoft Research Redmond. Agarwal was the project leader for Switchboard.

Happy reading, lyncing or debugging.

Citing from the introduction to Martin Porter’s work [1]:

Removing suffixes by automatic means is an operation which is especially
useful in the field of information retrieval.

Consequently, a stemmer is a (software) program to reduce inflected words to their root form (word stem). To derive a word’s root form a stemming program uses stemming algorithm(s). For example, a stemming algorithm might reduce the word catering to the stem cater. Notice, that different words may be reduced to the same stem, because they may be rooted in the same origin.

Potential stemming algorithms include dictionary-based approaches, i.e. looking up a word’s stem, or rule- and stochastic-based approaches [2]. Among the many stemming approaches, Porter’s variant is probably among of the most known ones. Reasons for this may be its availability in documentation and implementation format, and it’s natural appeal, i.e. one can immediately understand what the stemmer does by following Porter’s rules and provided examples.

For example, a rule dictates that any English word, which ends in “SSES” should be contracted to “SS”, i.e. the suffix commonly indicating plural, should be removed. Porter gives a handful of rules (30 or so) nicely arranged in a pipeline processing scheme, i.e. some rules are executed in a batch (named step) whenever the rules pre-condition is matched. Broadly there are 5 different steps, taking care of for example, plural, participle and noun suffixes.

Rule conditions govern whether or not a rule should be executed on some word(-fragment). Rules include:

• Does a word contain vowels
• Does a word end with a certain double consonant
• Does a word end with a particular suffix
• Does a word contain a sequence of vowel and consonant of length l

Whenever a rule is executed it either leads to a word contraction (removing a suffix), composition with an additional “e” or substitution of word parts (letters).

Equipped with this knowledge, I followed Porter’s paper and implemented the basic Porter stemming algorithm in F#. Since Porter did not only give the basic algorithm (rules), but also tests for it, I used a method resembling test-driven development to implement the stemmer. That is, using FsUnit, I implemented rule tests like:

"caresses" |> porterStem |> should equal "caress"

to drive implementation. Clearly, in the beginning, such a test expecting the word “caresses” to be stemmed to the stem “caress”, would fail. However, after having implemented step 1 (the previously mentioned batch of processing rules) given by Porter the stemmer would pass the test. Following this methodology, I implemented each step after I took and coded Porter’s tests for that step. Since tests are localised, i.e. a test tests a particular rule or batch of rules, tests might interfere with one another, i.e. produce some result deemed incorrect by a test but perfectly correct with regard to the implementation.

The F# implementation makes use of pattern matching, active patterns and pipelining. For example, avoiding the actual implementation of steps at the moment, the stemming process a word goes through is defined by:

word
|> step1A 
|> step1B
|> step1C
|> step2
|> step3
|> step4
|> step5A
|> step5B

Here, a word is simply a sequence of letters, which can be either consonant or vowel:

type Symbol = char
///<Summary>
///defines a letter, which can be the empty letter
///or a vowel (Vow) or a consonant (Cons)
///</Summary>
type Letter = 
    | EmptyLetter
    | Vow of Symbol
    | Cons of Symbol

let isVowel (v:Symbol) = 
    v = 'a' || v = 'A' || 
    v = 'e' || v = 'E' || 
    v = 'i' || v = 'I' || 
    v = 'o' || v = 'O' || 
    v = 'u' || v = 'U'

let (|Consonant|Vowel|) (s:Symbol) = 
    match isVowel(s) with
    | true -> Vowel(s)
    | false -> Consonant(s)

Since the actual implementation is rather large, you may find the stemmers implementation on SkyDrive. I created a folder specifically for my F# explorations. Notice, that this is not a full-blown project but rather a dump for (F#) source files. As such, I do not take responsibility for correctness, while I may answer question, the code is free – in lieu of Porters FAQ statement.

In order to compile the stemmer, simply create a standard F# project. In order to run the tests, you need NUnit as advised for FsUnit. The F# project was created using Visual Studio 2010 and F# 2.0. I zipped up all necessary files, including:

• The stemmer’s implementation
• The tests
• FsUnit
• The research paper in txt form (see References)
• A todo list

With that, happy stemming and read you soon.

F# Porter Stemmer SkyDrive link

Update:

• 2011-06-25 (v0.2): bug fixing and additional rules specified by Porter

Todo:

• Optimisations already given by Porter
• Documentation
• F#isation

References

1. M.F.Porter (1980): An Algorithm for suffix stripping
2. Wikipedia: Stemming

Todays post focuses on using F# to query Princeton’s WordNet Online service for information about some word. According to WordNet’s home:

WordNet® is a large lexical database of English. Nouns, verbs, adjectives and adverbs are grouped into sets of cognitive synonyms (synsets), each expressing a distinct concept. Synsets are interlinked by means of conceptual-semantic and lexical relations. The resulting network of meaningfully related words and concepts can be navigated with the browser. WordNet is also freely and publicly available for download. WordNet's structure makes it a useful tool for computational linguistics and natural language processing.

Traditionally, WordNet could be accessed from Prolog or via API abstractions of Prolog’s database database means. However, (more or less) recently the ability to query WordNet online was provided. Since I am aiming at gaining confidence with F#, I tried to make the wonderful world of SynSet approachable via F#.

In order to use WordNet Online there will be the following steps:

1. Initialize and query user about what word she wants to ask WordNet
2. Ask WordNet about the user word
3. Process WordNet’s answer

In order to realise points 1 – 3, we’ll develop a F# console application, which reads the console to obtain a word, make a web request to WordNet online to ask about the word’s SynSets. The result will be a HTML page, which will be processed using the HTMLAgilityPack, which is a CodePlex hosted project, with the following purpose:

This is an agile HTML parser that builds a read/write DOM and supports plain XPATH or XSLT (you actually don't HAVE to understand XPATH nor XSLT to use it, don't worry...). It is a .NET code library that allows you to parse "out of the web" HTML files. The parser is very tolerant with "real world" malformed HTML. The object model is very similar to what proposes System.Xml, but for HTML documents (or streams).

Within the realm of html parsing, we’ll use F# pattern and regular expression matching. Notice that, this is merely a prototypical solution and as such, many improvements can potentially be made to it, including:

• Stability and Robustness
  I do not claim, that the solution handles all WordNet answers correctly. I will do some more testing and tweaking but for now, it seams to work.
• Performance
  I implemented a synchronous pipeline model. So you might take more advantage of asynchronous workflows.
• Completeness
  There are part of speech (adverb) which are currently not handled, by I intent to add those.
• Understandability
  I tried to stay within the F# idioms but, I assume some processing still bears imperative looks.

Now, to give you a feel, for what I intent to do, the following is a picture of WordNet Online being asked about the word “dog”.

What I intent to do is to represent this information using an object SynSet having a SynType (its part of speech) like noun, etc. associated example sentences, and so on. Also, as you can see, WordNet Online allows us to specify certain query options, such as whether or not we want to be shown example sentences (e.g., “the dog barked all night”). Clearly, representing option information will be helpful too. For that, I defined the following types and some pattern match expression within their own module FWordNetTypes.

module FWordNetTypes

open System.Text
open System.Text.RegularExpressions

open HtmlAgilityPack

type WordNetOptions = 
    { 
        ShowExampleSentences:bool;
        ShowGlosses:bool;
        ShowFrequencyCounts:bool;
        ShowDatabaseLocations:bool;
        ShowLexicalFileInfo:bool;
        ShowLexicalFileNumbers:bool;
        ShowSenseKeys:bool; 
        ShowSenseNumbers:bool
    }
    member this.BuildWordNetOptionsString() = 
        let optionString o = ( if o = true then "1" else System.String.Empty )
        let x = ref ( StringBuilder() )
        x := (!x).Append(
            System.String.Format("o0={0}&o1={1}&o2={2}&o3={3}&o4={4}&o5={5}&o6={6}&o7={7}&o8=1"
                , (optionString this.ShowExampleSentences)
                , (optionString this.ShowGlosses)
                , (optionString this.ShowFrequencyCounts)
                , (optionString this.ShowDatabaseLocations)
                , (optionString this.ShowLexicalFileInfo)
                , (optionString this.ShowLexicalFileNumbers)
                , (optionString this.ShowSenseKeys)
                , (optionString this.ShowSenseNumbers)
            )
        )
        (!x).ToString()

type SynType =
    | SynNoun
    | SynVerb
    | SynAdjective 
    | SynNone

type SynSetWord = 
    { 
        Word:string; 
        SenseKey:string; 
        SenseNumber:string
    }
    override this.ToString() = 
        this.Word + "#" + this.SenseNumber + "("  + this.SenseKey + ")"

let SynTypeToString s =
    match s with
        | SynNoun -> "Noun"
        | SynVerb -> "Verb"
        | SynAdjective -> "Adjective"
        | _ -> "None" 

type SynSet = 
    { 
        SType:SynType; 
        LexicalFileInfo:string;
        LexicalFileNumber:string;
        SynWords:seq<SynSetWord>; 
        SynGlos:seq<string>; 
        SynExampleSentences:seq<string>; 
        FrequencyCount:int; 
        DatabaseLocation:string; 
    }
    override this.ToString() = 
        let sb = ref (StringBuilder())
        sb := (!sb).Append("SynType          : " + (SynTypeToString this.SType) + "\r\n")
        sb := (!sb).Append("SynWord(s)       :\r\n")
        this.SynWords |> Seq.iter(fun s -> sb := (!sb).Append("[+]" + s.ToString() + ";\r\n"))

        sb := (!sb).Append("SynGlos:\r\n")
        this.SynGlos |> Seq.iter(fun s -> sb := (!sb).Append("[+]" + s + ";\r\n"))
        
        sb := (!sb).Append("SynExample(s)    :\r\n")
        this.SynExampleSentences |> Seq.iter(fun s -> sb := (!sb).Append("[+]" + s + ";\r\n"))

        sb := (!sb).Append("Frequency        : " + this.FrequencyCount.ToString() + "\r\n")
        sb := (!sb).Append("DB Location      : " + this.DatabaseLocation + "\r\n")
        sb := (!sb).Append("Lexical File Info: " + this.LexicalFileInfo + "\r\n")
        sb := (!sb).Append("Lexical File No. : " + this.LexicalFileNumber)

        (!sb).ToString()

let EmptySynSet = 
    { 
        SType = SynNone; 
        SynWords = Seq.empty ; 
        SynGlos = Seq.empty ; 
        SynExampleSentences = Seq.empty ; 
        FrequencyCount = -1; 
        DatabaseLocation = System.String.Empty; 
        LexicalFileInfo = System.String.Empty; 
        LexicalFileNumber = System.String.Empty; 
    }

let (|Noun|Verb|None|Adjective|) (v:string) =
    match v.ToLowerInvariant().Trim() with
        | "(n)" -> Noun
        | "(v)" -> Verb
        | "(adj)" -> Adjective
        | _ -> None

let (|POS|SynWord|SynWordDesc|SynIndicator|SynExample|SynGlos|) (v:HtmlNode, word:string) =
    if (v.Name.ToLowerInvariant() = "a" && 
        v.Attributes.Contains("class") && 
        v.Attributes.Item("class").Value.ToLowerInvariant() = "pos") then 
        POS(v.InnerHtml.Trim())
    elif (v.Name.ToLowerInvariant() = "b" || 
         (v.InnerHtml.Contains(word) && v.Name.ToLowerInvariant() <> "i" ) ) || 
         (
            v.Name.ToLowerInvariant() = "a" && 
            v.Attributes.Contains("href") && 
            v.InnerHtml.Contains("S:") <> true
         ) then 
        let processedInner = v.InnerHtml.Trim()
        let senseKeyNumberRegEx = Regex("(?<SenseWord>.+)\#(?<SenseNumber>\d+).*\((?<SenseKey>.*)\)")
        let matches = senseKeyNumberRegEx.Match(processedInner)
        if matches.Success then
            let senseWord  = matches.Groups.Item("SenseWord").Value;
            let senseKey = matches.Groups.Item("SenseKey").Value;
            let senseNumber = matches.Groups.Item("SenseNumber").Value;
            
            SynWord( 
                { 
                    Word = senseWord; 
                    SenseKey = senseKey; 
                    SenseNumber = senseNumber; 
                }
            )
        else
            SynWord( 
                { 
                    Word = processedInner ; 
                    SenseKey = System.String.Empty; 
                    SenseNumber = System.String.Empty; 
                } 
            )
    elif ( v.Name.ToLowerInvariant() = "a" && 
           v.Attributes.Contains("href") && 
           v.InnerHtml.Contains("S:")) then 
        SynIndicator(v.InnerHtml.Trim())
    elif (v.Name.ToLowerInvariant() = "i" &&
          v.InnerHtml.StartsWith("\"") && 
          v.InnerHtml.EndsWith("\"")) then 
        SynExample(v.InnerHtml.Trim())
    elif (v.InnerHtml.Trim().StartsWith("(") && 
          v.InnerHtml.Trim().EndsWith(")") ) then 
        SynGlos(v.InnerHtml.Replace("(", "").Replace(")", "").Trim())
    else 
        SynWordDesc(v.InnerHtml.Trim())

As you can see, WordNetOptions, SynSetWord and SynType are the main types to handle WordNet Online options, SynSets and words as part of a SynSet. The rest is F# patterns to make the parsing of WordNet HTML answers more intuitive and readable. For example, instead of dealing with strings of the form “(v)”, I want to be able to ask about part of speech. Or instead of decomposing the answer in terms of HTML nodes, I want to derive decision logic on the basis of words, glosses and example sentences, i.e. problem domain entities. As mentioned, I do not claim this to be the ideal solution, however, I want to stress the importance of decomposing a problem within the problem domain, which is more often than not more suitable then expressing it within the realm of technical terms.

Clearly, patterns abstracting HTML node patterns were even more powerful, i.e. instead of evaluating node names and attributes all over the place, why not have a pattern such as BoldNode or AnchorNode. I plan to incorporate this in the near term.

Equipped with the types, we want to fill via instantiation, we can decompose the problem flow into:

1. Get the user’s word from command-line
2. Check whether the word is a word (not done here) and it is not a stop word, i.e. a word which has not entry in the WordNet database, because it does not express a concept, has no SynSet, etc.
3. Set query options, such a retrieve example sentences or not
4. Ask WordNet
5. Process WordNet’s answer
6. Display potential answer

These 6 steps are shown in the following:

WriteAsHeader "Ask WordNet ..."

printfn "enter a word: "

let stopwords = [|"a"; "the"; "these"; "this"; "those"; "them"; "their" |]
let isStopWord = 
    fun word -> 
        stopwords |> Seq.exists(fun s -> s.ToLowerInvariant() = word)

let word = System.Console.ReadLine().ToLowerInvariant()

if (isStopWord word) then printfn "word is a stop word ... exit"
elif (System.String.IsNullOrEmpty(word)) then printfn "Cannot look for empty word ... exit"
else 
    let options = { 
        ShowExampleSentences    = true; 
        ShowGlosses             = true; 
        ShowFrequencyCounts     = true; 
        ShowDatabaseLocations   = true;
        ShowLexicalFileInfo     = true;
        ShowLexicalFileNumbers  = true;
        ShowSenseKeys           = true;
        ShowSenseNumbers        = true; 
    }
    let showErrors = false
    let answer = askWordNet word options 
    if (System.String.IsNullOrEmpty(answer) <> true) then 
        printfn "obtained answer from WordNet ..."
        answer |> saveAnswerHTML word
        let processedAnswer = processWordNetAnswer word answer options showErrors
        match processedAnswer with 
            | Some(answer)  ->  
                printfn "WordNet said ... :"
                answer  
                    |> Seq.iter( fun k -> WriteAsHeader (k.ToString())) 
            | _ -> ()
    else
        printfn "failed to obtain answer from WordNet ..."

printfn "press <any> key to exit" 
System.Console.ReadLine() |> ignore

As you can see, the main processing is rather small. The stopwords array would clearly not be constant but expanded and stored elsewhere. WriteAsHeader is a simple function, which encloses some string within two horizontal bars (see bottom FWordNetHelper module). saveAnswerHTML will simply dump and HTML response in a file in the local file system, which can be helpful in offline testing mode.

The askWordNet function is a simple wrapper around:

1. Constructing an HTTP request URL from the user’s word and WordNet’s processing options
2. Invoking a web request via .NET means (i.e., System.Net.HttpWebRequest)
3. Getting the request’s response
4. Outputting the response as a string

let askWordNet word (option:WordNetOptions) =
    let url = "https://wordnetweb.princeton.edu/perl/webwn"
    if (System.String.IsNullOrEmpty(word) = true) then failwith "word cannot be null or empty"
    let wordneturl = url + (buildRequestString word option)

    printfn "requesting: %s" wordneturl

    let rq = System.Net.HttpWebRequest.Create(wordneturl)
    let resp = ref ""
    
    try
        using (new System.IO.StreamReader(rq.GetResponse().GetResponseStream())) ( fun s -> 
            resp := s.ReadToEnd()
        )
    with
        | :? System.Net.WebException -> ( printfn "failed to connect to \"%s\"\r\n" url )

    !resp

The programs heavy-lifting is done inside the processWordNetAnswer , which takes the user’s word, the WordNet options and the request’s response (as a string), to do the following:

1. Create an HTML document using the HtmlAgilityPack
2. Scrape the HTML document, so that a list of SynSet entries will be constructed for the user’s word
3. Return the constructed list

let processWordNetAnswer (word:string) answer options showErrors = 
    let doc = new HtmlDocument()
    
    doc.LoadHtml(answer)
    printfn "Html Parse Errors ... "
    if showErrors then 
        doc.ParseErrors 
            |> Seq.iter(fun err -> 
                (printfn "%d %d %s %s" err.Line err.LinePosition err.Reason err.SourceText) 
            )
    
    let synList = List<SynSet>()

    (FindListItemNode doc.DocumentNode "li")
        |> Seq.iter(fun liNode -> 
            let freqRegex = Regex("^(\((?<Frequency>\d+)\))?" + 
                                  "(\{(?<DatabaseLocation>\d+)\})?" + 
                                  "(\x20&lt;(?<FileInfo>.+\..+)&gt;)?" +
                                  "(\[(?<FileNumber>\d+)\])?(\x20)?<a")

            let rMatch = freqRegex.Match(liNode.InnerHtml)
            let mutable freq = -1;
            let mutable dbloc = System.String.Empty
            let mutable fileInfo = System.String.Empty
            let mutable fileNumber = System.String.Empty

            if rMatch.Success then 

                let freqStr = rMatch.Groups.Item("Frequency").Value
                let locStr = rMatch.Groups.Item("DatabaseLocation").Value
                let fileInfoStr = rMatch.Groups.Item("FileInfo").Value
                let fileNoStr = rMatch.Groups.Item("FileNumber").Value

                if (System.String.IsNullOrEmpty(freqStr) = false) then  freq <- System.Int32.Parse(freqStr) ;
                if (System.String.IsNullOrEmpty(locStr) = false) then dbloc <- locStr
                if (System.String.IsNullOrEmpty(fileInfoStr) = false) then fileInfo <- fileInfoStr
                if (System.String.IsNullOrEmpty(fileNoStr) = false) then fileNumber <- fileNoStr

            let xSyn = ref { 
                EmptySynSet with 
                    FrequencyCount = freq; 
                    DatabaseLocation = dbloc; 
                    LexicalFileInfo = fileInfo ; 
                    LexicalFileNumber = fileNumber; 
            }
            
            liNode.ChildNodes 
                |> Seq.iter(fun node -> xSyn := ( FillSynSet node !xSyn word) )
            
            synList.Add( sanitiseSynSet !xSyn )
        )
    
    if synList.Count <> 0 then Some(synList)
    else None

Using HtmlAgilityPack proves extremely beneficial to decompose the HTML response, into WordNet relevant atoms. FindListItemNode is a simple recursive HTML document traversing function which seeks to find the list item nodes within the html document. FillSynSet uses the power of F# pattern matching to fill a SynSet from information found in the reponses list items.

let FillSynSet (node:HtmlNode) (syn:SynSet) (word:string) = 
    match (node, word) with
        | POS(n) -> 
            match n with
                | Noun -> { syn with SType = SynNoun; }
                | Verb -> { syn with SType = SynVerb; }
                | Adjective -> { syn with SType = SynAdjective; }
                | _ -> ( failwith ("Error: Unrecognized POS \"" + n.ToString() + "\"!") )
        | SynExample(e) -> 
            (
                let exampleSentences = 
                    ( e.Split([|";"|], System.StringSplitOptions.RemoveEmptyEntries) 
                        |> Seq.map( fun entry -> entry.Trim() ) 
                    )
                { syn with SynExampleSentences = ( Seq.append syn.SynExampleSentences exampleSentences ) ; } 
            )
        | SynWord(n) -> { syn with SynWords = (Seq.append syn.SynWords [n]) ; }
        | SynGlos(g) -> { syn with SynGlos = ( Seq.append syn.SynGlos [g] ); }
        | SynIndicator(n) | SynWordDesc (n) -> syn

let FindListItemNode (startNode:HtmlNode) soughtNodeName = 
    let col = new List<HtmlNode>()

    let rec FindListItemNode_t (startNode:HtmlNode) soughtNodeName currentCollection = 
        if startNode = null then ()
        else
            if startNode.Name = soughtNodeName then ( col.Add(startNode) ) 
            else 
                startNode.ChildNodes 
                    |> Seq.iter(fun n -> (FindListItemNode_t n soughtNodeName col))

    FindListItemNode_t startNode soughtNodeName col
    col

In order to have some data cleansing, the sanitiseSynSet function takes a SynSet and cleans it from potentially unwanted artefacts (not done extensively). As you can see, having a simple WordNet scraper is indeed not that hard, all left are mentioned helper functions, which depending on your visualisation needs can be dropped:

module FWordNetHelper

open FWordNetTypes

let saveAnswerHTML (word:string) (answer:string) = 
    let time = System.DateTime.Now.Ticks.ToString()
    let fileName = System.String.Format(@".\{0}_{1}.html"
        , word
        , time
    )

    using (System.IO.File.CreateText(fileName)) (fun f ->
        f.Write(answer)
        f.Flush()
    )

let buildRequestString (word:string) (option:WordNetOptions) =
    let str = System.String.Format("?s={0}&sub=Search+WordNet&{1}"
        , word
        , ( option.BuildWordNetOptionsString() ) 
    )
    str

let WriteAsHeader s = 
    printfn "______________________________________________________\r\n"
    printfn "%s" s
    printfn "______________________________________________________\r\n"

From that, we might actually finish our little program by adding necessary namespace and module declarations:

module FWordNetOnline

open System.Text
open System.Text.RegularExpressions
open System.Net
open System.IO
open System.Collections.Generic

open FWordNetTypes
open FWordNetHelper

open HtmlAgilityPack

and ask WordNet about the known word “Dog”:

which, if all goes well, will answer with the following (partial) processed response:

Equipped with that dear reader, a happy exploration of the wonderful world of words using WordNet Online. In case of comments, improvements, please do not hesitate to comment – as only the ignorant is immune to learning.

Today’s post is merely a little code fragment, that gets you started with using the TFS2010 API from F#.

Assuming you have referenced the following Assemblies:

1. System.Windows.Forms
2. Microsoft.TeamFoundation.Client
3. Microsoft.TeamFoundation.Common
4. Microsoft.TeamFoundation.WorkItemTracking.Client

the following code connects to a TFS Server (cbtfs, you might want to change the URI to your needs) on the default port (8080) and loads up information about the TeamProjectCollection DefaultCollection. This will succeed under the assumption that valid credentials are supplied to the UICredentialsProvider. The UICredentialsProvider is also the reason why the assembly reference to System.Windows.Forms is demanded.

After successful authentication, a simple work item query (wiql) is constructed retrieving all work items available for that TeamProjectCollection. Finally, each work item’s id and title information is displayed on the console.

Edit 2011-06-02

I corrected the flow to account for previously checked project collection exception and incorporated a use of sequence expressions to make it feel more F#-ish

// Learn more about F# at https://fsharp.net
module TFSTest

open System.Net
open Microsoft.TeamFoundation
open Microsoft.TeamFoundation.Common
open Microsoft.TeamFoundation.Client
open Microsoft.TeamFoundation.WorkItemTracking.Client

let projectCollectionUri = System.Uri("https://cbtfs:8080/tfs/DefaultCollection")
let mutable projectCollection:TfsTeamProjectCollection = null

try
    
    projectCollection <- TfsTeamProjectCollectionFactory.GetTeamProjectCollection(
        projectCollectionUri
        , new UICredentialsProvider()
    )
    
    projectCollection.EnsureAuthenticated()

    printfn "authenticated with your tfs ... "
with
    | :? TeamFoundationServerUnauthorizedException -> 
        ( printfn "error: failed to authorize with TFS service!")
    | :? TeamFoundationServiceUnavailableException -> 
        ( printfn "error: failed to connect to TFS service - service unavailable!")
    | :? WebException -> 
        (printfn "error: Network problem!" )

let mutable workItemStoreService = null

if (projectCollection <> null) then
    try
        workItemStoreService <- projectCollection.GetService<WorkItemStore>()
        printfn "Connected to WorkItemStoreService"
    with
        | :? System.Exception -> 
            (printfn "error: failed to connect to workitemstore")

if (workItemStoreService <> null) then 

    let workItems = workItemStoreService.Query(
        "SELECT [System.Id], [System.Title] FROM WorkItems"
    )

    if (workItems <> null && workItems.Count > 0) then
        let workItemSequence = seq { for i in 1 .. (workItems.Count - 1) do yield workItems.Item(i) }
        workItemSequence 
            |> Seq.iter(fun item -> (printfn "[%d] %s" (item.Id) (item.Title)))
 
printfn "press any key to exit"
System.Console.ReadLine() |> ignore