Some of you may be wondering why certain design choices were made in the process of designing this challenge.  I hope to address some of these today.  Some additional FAQs are available on the official site.

Why a web service?

At the heart of this, much like it is for the Web N-Gram project, we're believers in data-driven discovery.  We believe that the programmatically-accessed Web is very much the present and the future. 

We also don't want to be in the business of hosting your code.  Aside from the obvious security implications of us running third-party code here, we don't want to dictate what platform, tools, etc., that you can use.  You decide, you host.

Why REST?

In a word, expedience.  Because our input and output specifications are so simple, using a more formal structure such as SOAP would only serve to slow everything down, both in terms of development and runtime.

Why the restrictions on the web service?

There are two main restrictions on the web service.  No redirects, and that MIME type must be text/plain.  Both are motivated by our desire to prevent a malicious participant from making us inadvertently spamming an unsuspecting web site.  Requiring the MIME type to be text/plain prevents you from pointing our code to a regular Web site (these generally return text/html as the MIME type, as they should.)  Similarly, if you could redirect the request with a 301/302 HTTP response, you could easily appear to be a proper site at the time you're registering your web service URI, but redirect us to an unsuspecting third-party URl when it comes time to actually evaluate the speller.  Granted these are not air-tight defenses against malicious intent, but it's better than nothing.

What about the test data?

We've prepared a human-judged corpus to get you started with the process.  There are ~6000 phrases in this dataset, and these are the phrases with which we will call your web service up until the final submission period.  For the final submission period we will obviously have a different corpus.  We caution against tuning your algorithm too closely to the test dataset (a subject for a future blog.)

We're also encouraging folks with training data to share with the community.  For Intellectual Property reasons we won't host these datasets, but we do provide a mechanism through which you can let the community know the availability.

Where can I host my service?

Naturally I would advise you to host your code in Windows Azure but there are no free options as of this writing there unless you already have an MSDN account (which itself is not free.)  For the more budget-challenged, Amazon and Google both offer some hosting for free.  (This is not an endorsement.)  There are surely many others.  Keep in mind that while we don't intend to ever over-access your site, you are ultimately responsible for any charges or other issues with the hosting service of your choice, including rate-limiting.

What if I have more questions?

E-mail is always an option, but we strongly encourage you to use one of the social media options, for two reasons: (a) someone else have the same question and would benefit from a shared response, and (b) there are more of you than there are of us -- and one of you may already know the answer!  This about community, folks!

You might be asking, why not a trie?  The simple answer is size.  If you have on the order of a billion words in your lexicon, a trie would generally be quite large.  And large usually means slow, especially at this scale.  What we need instead is a data structure that allows us to map the tokens to IDs without storing the lexicon itself, which is gigabytes in storage.

A number of techniques exist in this area but a property many of these have, ours included, is the possibility of false-negatives.  This is to say that if you have a known token, the token will reliably map to a particular ID value, but if you have an unknown token, the ability to detect this is not guaranteed.  Specifically, an in-vocabulary and out-of-vocabulary token can map to the same ID value.  This is the trade-off to get the size down.  Folks who study these sorts of things often talk in terms of bits per token,  They often employ a secondary construct known as a fingerprint, to help reduce the odds of a false-positive.  Of course the fingerprint also contributes towards the bits per token. so it comes down to keeping the error rate acceptably low.

Choosing a probabilistic hash instead of a data structure that embodies the lexicon also means that our service cannot do partial word wildcard lookups (i.e. you cannot find the probability of chris* and get the probability of chris, christopher, christmas, etc.).  It does, however, allow us to use a tree-like structure to facilitate backoff lookup (see this post for more on this) as well as our generative-mode API (post) so it isn't all bad.  Note that in order to produce the list of words in generative mode, we do have the ability to map IDs back to the token.  In other words if we check the round-trip value of mapping a token to an ID and then back, we could test for the validity of the token.  For performance reasons, we don't bother.

You think you can do a better job?  Now's your chance to prove it — Introducing the Speller Challenge!  Win cash prizes!

Mosey on to https://spellerchallenge.com and get a head start.  The contest officially opens January 2011.

As you recall, for unigrams, P(x)is simply the probability of encoutering xirrespective of words preceding it.  A naïve (and logical) way to compute this would be to simply take the number of times x is observed and divide it by the number of total words.  This would be fine until you encounter a word you'd never seen before—what should the unigram probability of that term be?

One option is to say P(x) is 0 if x is out-of-vocabulary (OOV).  This would be the right answer if you were dealing with unsmoothed models.  What we have instead are smoothed models, which means that we like to attribute some nonzero probability to words however unlikely the particular words are. If you were computing joint probabilities involving multiple terms, you likely don't want a single term to 'doom' the overall probability to 0 just because it was OOV.  So what we've done here is give the unknown word a small probability mass, and, since the overall probability needs to sum up to unity, we take it out of the denominator.  If the counts of a word x is expressed as C(x) and count of all words expressed as C(*), you'd have:

P(x) = C(x) / [1 + C(*)]
P(<unk>) = 1 / [1 + C(*)]

What the above equation shows is that we've given unknown words the equivalent probability of a word observed just once.

But what happens if you encounter multiple unknown terms while evaluating a joint probability?  Since the probability mass represents all unknown terms, it has to be distributed amongst unknown terms.  Therefore you divide it n-ways; the contribution of the unknown terms to the overall joint probability is thus [P(<UNK>)/n]^n.

1. Check your web proxy
   
   Your web proxy can be a bottleneck, especially in a corporate environment.  Some proxies will limit the number of calls an application can make in a given amount of time, and will return 5xx (HTTP) responses when you exceed this limit.  Oftentimes it's difficult to tell whether the web service is returning the error as opposed to your proxy, so check the error message carefully.  If there's an alternative proxy you can use, try that one instead.  Here's how you'd modify your application configuration file:
   

   1. Find your application configuration file.  Generally, this file will be called app.config or web.config
   2. Find the binding your endpoint is using under <configuration><system.serviceModel><bindings><wsHttpBinding> or equivalent
   3. Set useDefaultWebProy="false", and
   4. Set proxyAddress="https://your-proxy-name-and-maybe-port"
2. Batch your calls
   
   If your task involves getting a bunch of probability values, the use of the batch methods is strongly encouraged.  This is partly due to the amortization of XML parsing, and also partly due to how lookups are distributed in the cluster. So instead of calling GetProbability or GetConditionalProbability, call GetProbabilities or GetConditionalProbabilities whenever possible.
3. Reuse your WCF proxy
   
   There's no need to repeatedly create a client proxy for every call to the service.  This is in fact a great overhead that's usually unnecessary.  You might catch for faults (exceptions) and only recreate a client proxy in such an event.  Bonus points if you have an exponential backoff during retries while dealing with network hiccups.  
4. Choose models wisely
   It turns out that the larger the model, the longer the lookups take.  Body, as you might guess, is the largest; anchor and bquery are next largest, and title is the smallest. You may also notice some performance difference between jun09 models and apr10 models due to some changes in how the data is hosted.

Contribute your own tips!

As noted in earlier blog entries, our data comes straight from Bing.  All tokens are case-folded and with a few exceptions, all punctuation is stripped.  This means words like I'm or didn't are treated as two tokens each, even though we might consider them to be single words.  Here is the aforementioned exceptions in the bing-body:jun09 dataset:

.net
c++
c#
j#
g++
j++
com+
gdi+

The dataset qualification is critical here, because we aren't necessarily committed to keeping that set the same moving forward.  But this could be a problem for the GetConditionalProbability method, because the definition of this method is the probability of the last word in the given context.  Notice I used 'word' instead of token.  Therein lies a compromise — for the purpose of GetConditionalProbability we use whitespace as a word boundary.  An example might help illustrate this point:

GetConditionalProbability("bing-body:apr10:3", "on no, you didn't")

The input phrase has 5 tokens: oh, no, you, didn, and t.  If we were simply using the token boundary we would return the value of P(t|you didn), and ignore the three tokens preceding.  But because we're using whitespace as boundaries, we return P(t|you didn)×P(didn|no you) and ignore the two preceding tokens.  Subtle, yes, but crucial if you want predictable results.

We also have two special tokens, <s> and </s>.  These mean, beginning of segment and end of segment, respectively.  These help you get probabilities at the segment boundaries.  Here's an example:

>>> import MicrosoftNgram
>>> s = MicrosoftNgram.LookupService(model='urn:ngram:bing-body:apr10:2')
>>> for t in s.Generate('<s>', maxgen=6): print t
...
('home', -1.3229439999999999)
('the', -1.389273)
('skip', -1.6477809999999999)
('search', -1.8886769999999999)
('a', -1.938218)
('this', -1.9436990000000001)

 This shows that in the body stream, the most common first word is home, and so on.

The one tricky business with the algorithm as outlined above is that it can be computationally expensive, especially if the pre-word-broken string is long.  But with reasonably aggressive pruning, a breadth-first-search is actually quite usable.  The coding is nevertheless tedious, so we've prepared two modes by which you can experiment with word-breaking: an interactive Silverlight application, and a REST service.  Official write-up is here.

Shown here is a capture from the WordBreaker4Web app, breaking the title of this post:

So what are some applications for such a service?  One obvious example is URL or filename breaking.  In the case of a domain names, spaces are not permitted so unless you want to substitute hyphens, you're only choice is to string the words together.   This leads to rather long domain names such as this [N.B. examples are random, not intended as an endorsement]: insearchoftheworldsmostbeautifulwoman .  Or perhaps this Internet meme: penisland, meant to be parsed as pen island, not some other way.  As for filename breaking, omission of space characters is still quite prevalent, as you're surely aware.

Other applications exist, too.  Hash tags, seen throughout the Internet and quite prominently on Twitter (where we also are, by the way), are by nature words strung together with spaces removed.  Here's an example, picking one of the trending terms at random:

C:\bin\curl>curl https://web-ngram.research.microsoft.com/ngramwordbreaker/break.svc/?p=imoneofthosepeople
i|m|one|of|those|people (-16.02505)
im|one|of|those|people (-16.56622)
i|m|oneof|those|people (-17.15775)
im|oneof|those|people (-17.69892)
imone|of|those|people (-17.93192)

For those with sharp eyes will note that the REST service shown above doesn't provide a means to specify a language model.  This is because we built an artificial model based on ‘bing-title:apr10:2’ (that's right – bigram) with the n-grams drastically pruned for faster performance at the expense of accuracy.  It nevertheless does a fairly decent job.  Compare the speed between the REST service and the Silverlight demo app and the effect of the shrunk datset becomes quite obvious.

Does your Department of Licensing word break?  Some of you may recall the news about a particular plate getting rejected in Colorado:

C:\bin\curl>curl https://web-ngram.research.microsoft.com/ngramwordbreaker/break.svc/?p=iluvtofu
iluv|tofu (-12.75313)
iluv|to|fu (-14.22803)

So we think it was about 30x more likely that they loved bean curds.  But who knows.

We decided to take a closer look at the dataset to how the top 100K lists changed between Jun09 and Apr10.  Our findings are interesting:

• The union of the word set is just shy of 110K.  This means that 10% of the words either fell in or out of the top 100K.  This is a turnover rate higher than I expected.
• Some words that are newly in the top list are what you'd expect (unigram log10 probability difference shown parenthetically):
  • espnlosangeles (21.88993), an ESPN satellite established during 2009
  • debate2010 (21.53613)
• Some words took a predictable jump:
  • ipad (2.560667), a product introduced mid-year
• Quite a few words newly in the mix are not conversational words:
  • childreplyhtml (22.09848)
  • focaladvid (21.76564)

Curious indeed.

To get started, you'll of course need to get the module, which you can download here.

The main class is named LookupService.  An instance of this object encapsulates two crucial pieces of information (a) the user token, and (b) the language model of interest.  The user token is a GUID issued by Microsoft Research.  This is something we use to track the amount of usage; neither the phrases nor models used are tracked in the interest of protecting users' privacy.  The language model is dataset against which you can query probabilities.  Details on language models were covered in last week's post, but in a nutshell there are three properties to a model: source, version, and order.  The following instantiations of the constructor are all functionally equivalent, provided that (i) you use your actual GUID, not xx.., and (ii) for the first case, you've specified an environment variable called NGRAM_TOKEN and its value set to your GUID.

>>> s = MicrosoftNgram.LookupService()
>>> s.GetModel()
'bing-body/jun09/3'
>>> s = MicrosoftNgram.LookupService(token='xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx')
>>> s.GetModel()
'bing-body/jun09/3'
>>> s = MicrosoftNgram.LookupService('xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx')
>>> s.GetModel()
'bing-body/jun09/3'
>>> s = MicrosoftNgram.LookupService(token='xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx',model='bing-body/jun09/3')
>>> s.GetModel()
'bing-body/jun09/3'
>>> s = MicrosoftNgram.LookupService('xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx','bing-body/jun09/3')
>>> s.GetModel()
'bing-body/jun09/3'

 I prefer to set the environment variable, since I can't be bothered to memorize a GUID.  Speaking of environment variables, note that MicrosoftNgram uses urllib under the covers, so if you need to specify a proxy, set HTTP_PROXY appropriately.

Once you have a LookupService object, you can call the various methods. 

>>> s = MicrosoftNgram.LookupService(model='bing-body/apr10/5')
>>> s.GetConditionalProbability('happy cat is happy')
-0.93900499999999998
>>> s.GetConditionalProbability('happy cat is sad')
-4.2167089999999998
>>> s.GetJointProbability('kthxbai')
-7.6080370000000004

 Well it's good to know that happy cat is 1000x more likely to be happy than sad.  What else can happy cat be?

>>> for t in s.Generate('happy cat is', maxgen=5): print t
...
('always', -0.36325089999999999)
('a', -0.89422170000000001)
('happy', -0.93900499999999998)

So we know that happy cat is never sad (well, most likely anyway — bing-body/apr10 has a unigram cutoff of 10.)  We can further infer that when computing the conditional probability above, we must have backed off to a lower-order n-gram.

One of the key insights in studying how documents are composed on the Web is that there are subtle but nevertheless distinct styles used, even in a single document.  You know this intuitively -- when you form a query to your search engine, for example, you often string together keywords to form a phrase, but you wouldn't use the same phraseology in prose.  Even in prose, the style in which we compose paragraphs will differ from how we compose titles or reference other material.

In the Web N-Gram service, we have collected from Bing textual data from four different sources (sometimes referred to as 'streams'): Body, Anchor, Title, and Query.  To get the list of currently supported models, use this handy URL. 

There are two different naming conventions used by our service: the URN form when using XML/SOAP, and the Path form for REST access.

URN: urn:ngram:{source}:{version}:{order}

Path: {source}/{version}/{order}

In this context, order=1 is unigram, order=2 is bigram, and so forth.

A number of papers have been published demonstrating computational tasks using different models.  Here's one that appeared in SIGIR: A Comparative Study of Bing Web N-gram Language Models for Web Search and Natural Language Processing.