I delivered a session on Azure ML at the recent NYC
Microsoft BigData hackathon. I had a great time exploring exciting machine
learning scenarios and ideas with the attendees. Later in the day, while the
teams were heads down working on their projects, I had open time to do some
Azure ML hacking using the New York City restaurant inspection results dataset.

The gist of the idea was to create a model that can predict
the grade of a restaurant from the description of the violation. The results
were remarkably accurate, so I am sharing the approach, and this is the
walkthrough.

Read more here.

Enter Azure Search

Azure Search is a platform-as-a-service offering that handles the hurdles of maintaining a search index while giving you full flexibility to define the index and scale out when you need more resources and scale down when you don’t need them anymore. In my last post, I described how you can enable full text search for an Azure Mobile Services back end. As a continuation of the theme, I thought it would be useful to showcase how you can enable a similar functionality for a database-driven ASP.NET website. So let’s get to it.

Revisiting the Basics

Similar to the previous post, this process consists of creating an index, then enqueueing data changes and having a process that reads from the queue and updates the index.

We will also use the Azure Service Bus’s queue (hence guaranteeing orderly delivery) and an attribute-based implementation to tag the data entities with metadata. The metadata defines the index’s schema and the properties that will be written to the index. In this instance, we will leverage an Azure WebJob to perform the index operations.

Prerequisites

To follow the steps below, you need to download and make a reference to the project that contains the attribute and infrastructure code. You can get it here. Also, I am assuming that you already have an instance of Azure Search and Azure Service Bus running. For more information about getting started with Azure Search, see this article.

Building Our Model and Capturing Changes

In an empty ASP.NET MVC website, let’s create a model and decorate it with the custom Azure Search attribute.

namespace aspnet_mvc_azuresearch.Models
{
    public class RockBand
    {
        [Indexable("Edm.String", false, Key = true, Name = "id")]
        public string Id { get; set; }

        [Indexable("Edm.String", true, Name = "name", Retrievable=true)]        
        public string Name { get; set; }

        [Indexable("Edm.String", true, Name = "genre", Retrievable = true)]
        public string Genre { get; set; }

        [Indexable("Edm.String", true, Name = "description", Retrievable = true)]
        public string Description { get; set;}
    }
}

Next, using the scaffolding capabilities of Visual Studio, let’s wire up a controller with Entity Framework and the corresponding write, edit, delete, and list views.

It’s considered a best practice to implement a view model between your view and your entities. But for simplicity, we will use the scaffolded code as is.

Let’s include the code (highlighted below) in the applicable actions that will send the entity changes to the queue .

namespace aspnet_mvc_azuresearch.Controllers
{
    public class RockBandsController : Controller
    {
...      
        

        ...
        
        // POST: RockBands/Create
        // To protect from overposting attacks, please enable the specific properties you want to bind to, for 
        // more details see http://go.microsoft.com/fwlink/?LinkId=317598.
        [HttpPost]
        [ValidateAntiForgeryToken]
        public ActionResult Create([Bind(Include = "Id,Name,Genre,Description")] RockBand rockBand)
        {
            if (ModelState.IsValid)
            {
                db.RockBands.Add(rockBand);
                db.SaveChanges();
                

                return RedirectToAction("Index");
            }

            return View(rockBand);
        }


        // POST: RockBands/Edit/5
        // To protect from overposting attacks, please enable the specific properties you want to bind to, for 
        // more details see http://go.microsoft.com/fwlink/?LinkId=317598.
        [HttpPost]
        [ValidateAntiForgeryToken]
        public ActionResult Edit([Bind(Include = "Id,Name,Genre,Description")] RockBand rockBand)
        {
            if (ModelState.IsValid)
            {
                db.Entry(rockBand).State = EntityState.Modified;
                db.SaveChanges();
                

                return RedirectToAction("Index");
            }
            return View(rockBand);
        }

        // POST: RockBands/Delete/5
        [HttpPost, ActionName("Delete")]
        [ValidateAntiForgeryToken]
        public ActionResult DeleteConfirmed(string id)
        {
            RockBand rockBand = db.RockBands.Find(id);
            db.RockBands.Remove(rockBand);
            db.SaveChanges();
           

            return RedirectToAction("Index");
        }

        ...

    }
}

In the WebConfig, you need to add the following settings:

<appSettings>
    ...
    <add key="ASEM_SbConnString" value="YOUR SERVICE BUS CONNECTION STRING"/>
    <add key="ASEM_SearchServiceName" value="giventocodems"/>
    <add key="ASEM_IndexName" value="ixrockbands"/>
    <add key="ASEM_ApiVersion" value="2014-10-20-Preview"/>
    <add key="ASEM_ApiKey" value="YOUR AZURE SEARCH API KEY"/>
  </appSettings>

Search Results

Next, let’s create a search user interface and we will start with a new view model.

namespace aspnet_mvc_azuresearch.Models
{

    public class SearchViewModel
    {
        public string Criteria { get; set; }
        public SearchResultViewModel[] Results { get; set; }
    }

    public class SearchResultViewModel
    {
        public string Id { get; set; }
        public string Name { get; set; }
        public string Description { get; set; }
        public string Genre { get; set; }
    }
}

Add a view for our view model.

@model aspnet_mvc_azuresearch.Models.SearchViewModel

@{
    ViewBag.Title = "Index";
    Layout = "~/Views/Shared/_Layout.cshtml";
}

<h2>Search</h2>

<p>
    @using (Html.BeginForm("","Search",FormMethod.Get))
    {
       

        <div class="form-horizontal">
            <h4>RockBand Search</h4>
            <hr />
            <div class="form-group">
                @Html.LabelFor(model => model.Criteria, htmlAttributes: new { @class = "control-label col-md-2" })
                <div class="col-md-10">
                    @Html.EditorFor(model => model.Criteria, new { htmlAttributes = new { @class = "form-control" } })                    
                </div>
            </div>

            <div class="form-group">
                <div class="col-md-offset-2 col-md-10">
                    <input type="submit" value="Search" class="btn btn-default" />
                </div>
            </div>
        </div>
    }

</p>



<ul>
    @foreach (var item in Model.Results)
    {
        <li>
            <strong>@Html.DisplayFor(model => item.Name)</strong>
            <p>
                <small>Genre: @Html.DisplayFor(model => item.Genre)</small>
            </p>
            <p>
                @Html.DisplayFor(model => item.Description)
            </p>
            
        </li>        
    }
</ul>

And the search controller.

namespace aspnet_mvc_azuresearch.Controllers
{
    public class SearchController : Controller
    {
        private AzureSearchManager searchMan = new AzureSearchManager();
        
        [HttpGet]
        public async Task<ActionResult> Index(string criteria) 
        {

            if (string.IsNullOrEmpty(criteria))
            {
                return View(new SearchViewModel() { Results = new SearchResultViewModel[0] });
            }

            var results = await searchMan.SearchAsync(criteria);
            var resultsVM = ((JArray)results["value"])
                            .Select<JToken, SearchResultViewModel>(t => new SearchResultViewModel() { Id = (string)t["id"], Name = (string)t["name"], Genre = (string)t["genre"], Description = (string)t["description"] })
                            .ToArray<SearchResultViewModel>();

            return View(new SearchViewModel() { Results = resultsVM });
        }
    }
}

Processing Changes and Index Operations

Finally, we need to implement the process that will read the messages from the queue and perform the index operations. For this, we will use an Azure WebJob using the WebJobs SDK. By using the SDK, we don’t need to implement polling logic because it provides built-in capabilities to trigger the WebJob when new messages arrive automatically. So let’s add a new Azure WebJob project to our solution.

After adding a reference to the AzureEntityManager project, let’s add a function that receives the message from the queue and performs the corresponding index operation.

Note: The AzureSearchManager, before performing the corresponding index operation, checks if the index exists, and if it doesn’t, it creates one according to the attributes defined in the data entity.

namespace Giventocode.AzureSearchWebJob
{
    public class Functions
    {
        // This function will get triggered/executed when a new message is written 
        // on an Azure Queue called queue.
        
        public static async Task IndexEntity( [ServiceBusTrigger(IndexManager.INDEXER_QUEUE)] string msg ,
             TextWriter log) 
         {
             var searchMan = new AzureSearchManager();              
             
             await searchMan.IndexDocumentAsync(msg);
         } 

    }
}

Azure WebJob require a storage account where execution logs will be stored, and in the configuration file, you must include the connection string to the storage account and the service bus instance that contains the queue.

<connectionStrings>
   <!-- The format of the connection string is "DefaultEndpointsProtocol=https;AccountName=NAME;AccountKey=KEY" -->
   <!-- For local execution, the value can be set either in this config file or through environment variables -->
   <add name="AzureWebJobsDashboard" connectionString="YOUR STORAGE ACCOUNT CONNECTION STRING" />
   <add name="AzureWebJobsStorage" connectionString="YOUR STORAGE ACCOUNT CONNECTION STRING" />
   <add name="AzureWebJobsServiceBus" connectionString="YOUR SERVICE BUS CONNECTION STRING" />

 </connectionStrings>

We also need to add the AzureEntityManager settings to the App.config.

<appSettings>
   <add key="ASEM_SearchServiceName" value="giventocodems"/>
   <add key="ASEM_IndexName" value="ixrockbands"/>
   <add key="ASEM_ApiVersion" value="2014-10-20-Preview"/>
   <add key="ASEM_ApiKey" value="YOUR AZURE SEARCH API KEY"/>
 </appSettings>

Running with It!

We are now ready to run the website. So let’s create a new entry and then edit its description.

After running the web job, we can see that it processed two messages: upload (insert) and then merge (update) operations.

Search for using a keyword from the updated description and results!

If we go to the Azure preview portal, we can see the newly created index and our document.

Final Thoughts

It is worth mentioning that this approach is suitable for on-premises scenarios. So you can have your website hosted outside of Azure—and the WebJob and Azure Search service running on Azure. In addition, you can implement your own queue processing job easily. To facilitate this task, the IndexManager exposes a method that reads from the service bus queue and performs the index operation accordingly.

Also, you can download the full solution from here.

In a previous blog post, I described implementation of full-text search capabilities for your Azure Mobile Services back-end. With the recent availability of Azure Search —search as a service in Azure— I thought it would be useful to revisit the idea and modify it using this cool new Azure capability.

Defining the Index’s Schema

A document is the basic unit of data in Azure Search. Each document contains a set of fields, and each field has a set of attributes that define how the index handles the data associated which each field. When we create an index in Azure Search, we must create a schema that outlines attributes for each field. In the schema, we outline the types of fields available, which fields are searchable, and other interesting field capabilities. For example, we determine whether a field can be used as a search suggestion, for ordering or filtering the results, and as a facet, such as showing a count of results by brand if you search for “sports cars.”

Note: For more information about the schema definition, click here.

Following the spirit of a “code first” approach, let’s create a custom attribute that models the attributes of the index. We can then use it in the properties of an entity to create the metadata that will define the schema of the index.

[AttributeUsage(AttributeTargets.Property)]
    public class IndexableAttribute : Attribute
    {

        private string _name;

        public string Name
        {
            get { return _name; }
            set { _name = value; }
        }

        public readonly string Type;
        public readonly bool Searchable;
        
        private bool _filterable = true;

        public bool Filterable
        {
            get { return _filterable; }
            set { _filterable = value; }
        }

        private bool _sortable = true;

        public bool Sortable
        {
            get { return _sortable; }
            set { _sortable = value; }
        }


        private bool _facetable = true;

        public bool Facetable
        {
            get { return _facetable; }
            set { _facetable = value; }
        }


        private bool _suggestions = false;

        public bool Suggestions
        {
            get { return _suggestions; }
            set { _suggestions = value; }
        }

        private bool _key = false;

        public bool Key
        {
            get { return _key; }
            set { _key = value; }
        }


        private bool _retrievable = false;

        public bool Retrievable
        {
            get { return _retrievable; }
            set { _retrievable = value; }
        }

        public IndexableAttribute(string type, bool searchable )
        {            
            this.Type = type;
            this.Searchable = searchable;
        }
    }

Capturing Changes

Whenever an entity is modified or created, we need to capture and queue these changes. The benefit of queuing these changes is that we can decouple the process of maintaining the index from the request that updates the database.

The process of sending the entity to the queue consists of getting the information for each property’s custom attribute and value. The following code shows the implementation of this process. Full code here. 

public class IndexQueueManager
{
...

public TData EnqueueEntity<TData>(Task<TData> tEntity, string action) where TData : class, ITableData
    {

        if (tEntity.IsFaulted)
        {
            throw new InvalidOperationException("Task in faulted state", tEntity.Exception);
        }
        
        var ixEntity = GetEntityIndexInfo<TData>(tEntity.Result, action);
        
        var msg = new BrokeredMessage(JsonConvert.SerializeObject(ixEntity));
        msg.TimeToLive = new TimeSpan(7, 0, 0, 0);
        _indexerQ.Send(msg);

        return tEntity.Result;
    }
...  
}

Note: The object that goes into the queue is an instance of EntityIndexInfo. You can find the source code here.

For the queue’s implementation, we will use the queuing capabilities of the Azure Service Bus. Service Bus provides orderly delivery of the messages, which is an important consideration because we need to guarantee updates to the index in the same order that changes to an entity occur. The following code shows the constructor of the IndexQueueManager class where we initialize the queue client.

public IndexQueueManager(string connString)
{
    if (connString == null)
    {
        throw new ArgumentNullException("connString");
    }            

    var mgr = NamespaceManager.CreateFromConnectionString(connString);

    if (!mgr.QueueExists("amsindexerqueue"))
    {                
        var q = mgr.CreateQueue("amsindexerqueue");              
    }

    _indexerQ = QueueClient.CreateFromConnectionString(connString,"amsindexerqueue");
    
}

Finally, one elegant way to capture when these changes occurred is by overriding the insert, replace, update, and delete methods of the EntityDomainManager. The following code shows the implementation of a derived domain manager that, after performing a database operation, sends the entity to the queue.

public class IndexableEntityDomainManager<TData> : EntityDomainManager<TData> where TData : class, ITableData, new()
{
    private IndexQueueManager IndexManager;

    public IndexableEntityDomainManager(DbContext context, HttpRequestMessage request, ApiServices services, string ServiceBusConnString)
        : base(context, request, services)
    {
        if (ServiceBusConnString == null)
        {
            throw new ArgumentNullException("ServiceBusConnString");
        }

        this.IndexManager = new IndexQueueManager(ServiceBusConnString);
    }
    public  override Task<TData> ReplaceAsync(string id, TData data)
    {
        return base.ReplaceAsync(id, data)
            .ContinueWith<TData>((t) => IndexManager.EnqueueEntity<TData>(t, "mergeOrUpload"));            
    }

    public  override Task<TData> UpdateAsync(string id, Delta<TData> patch)
    {
        return  base.UpdateAsync(id, patch)
                .ContinueWith<TData>((t) => IndexManager.EnqueueEntity<TData>(t, "merge")); ;
    }

    public  override Task<TData> InsertAsync(TData data)
    {
        return base.InsertAsync(data)
                    .ContinueWith<TData>((t) =>IndexManager.EnqueueEntity<TData>(t, "upload"));            
    }

    public  override Task<bool> DeleteAsync(string id)
    {
        return base.DeleteAsync(id)
            .ContinueWith<bool>((t) => 
                {
                    IndexManager.EnqueueEntity<TData>(Task<TData>.Run(() => new TData() { Id = id }), "delete");
                    return t.Result;
                });
        
    }
}

Maintaining the Index

Now that we have the changes in the queue, we need to read the messages from the queue and update the index. The following code first reads a message and checks whether an index exists. If an index does not exist, the code creates one according to the definition included in the message. Finally, the code creates a document instance containing the changes to the entity and sends a request to update the index with this information. Check out the full implementation here.

public class IndexQueueManager
{
...

    public async Task<Boolean> ReadFromQueueAndIndexAsync()
    {
        try
        {
            var msg = _indexerQ.Receive();

            if (msg == null)
            {
                return false;
            }

            var indexInfo =  JsonConvert.DeserializeObject<EntityIndexInfo>(msg.GetBody<string>());

            await CreateIndexIfNotExistsAsync(indexInfo);

            var indexDoc = GetIndexDocument(indexInfo);
            
            await IndexDocumentAsync(indexDoc);

            _indexerQ.Complete(msg.LockToken);

            return true;
        }
        catch(Exception ex)
        {
            _indexerQ.Abort();

            throw ex;
        }

    }
...
}

After we wrap up this code as a scheduled job, we are ready to enable this functionality in our back-end!

public class IndexerJob : ScheduledJob
    {
        public async override Task ExecuteAsync()
        {
            var mgr = new IndexQueueManager(CloudConfigurationManager.GetSetting("Search_ServiceBusConnString"));
            var maxNumberOfMgs = 1000;
            var processMsgs=0;

            while (await mgr.ReadFromQueueAndIndexAsync() && processMsgs < maxNumberOfMgs)
            {
                processMsgs++;
                Services.Log.Info(string.Format("Processed message. Message count {0}",processMsgs.ToString()));
                
            }
                     
        }
    }

Using What We Built

First things first: let’s create an Azure Search instance using the preview portal.

We are ready to use our custom attribute and domain manager. Let’s create a data model and decorate the properties using the custom attribute.

public class RockBand:EntityData
{
    [Indexable("Edm.String", true, Suggestions = true, Retrievable = true, Facetable = false)]
    public string Name { get; set; }

    [Indexable("Edm.String", true,Facetable=false)]
    public string Description { get; set; }

    [Indexable("Edm.String", true)]
    public string Genre { get; set; }
}

Next, let’s create a table controller for the model that uses our domain manager.

public class RockBandController : TableController<RockBand>
{
  protected override void Initialize(HttpControllerContext controllerContext)
  {
      base.Initialize(controllerContext);
      MobileServiceContext context = new MobileServiceContext();
      DomainManager = new IndexableEntityDomainManager<RockBand>(context, Request, Services, CloudConfigurationManager.GetSetting("Search_ServiceBusConnString"));
  }

  // GET tables/RockBand
  public IQueryable<RockBand> GetAllRockBand()
  {
      return Query(); 
  }

  // GET tables/RockBand/48D68C86-6EA6-4C25-AA33-223FC9A27959
  public SingleResult<RockBand> GetRockBand(string id)
  {
      return Lookup(id);
  }

  // PATCH tables/RockBand/48D68C86-6EA6-4C25-AA33-223FC9A27959
  public Task<RockBand> PatchRockBand(string id, Delta<RockBand> patch)
  {
       return UpdateAsync(id, patch);
  }

  // POST tables/RockBand/48D68C86-6EA6-4C25-AA33-223FC9A27959
  public async Task<IHttpActionResult> PostRockBand(RockBand item)
  {
      RockBand current = await InsertAsync(item);
      return CreatedAtRoute("Tables", new { id = current.Id }, current);
  }

  // DELETE tables/RockBand/48D68C86-6EA6-4C25-AA33-223FC9A27959
  public Task DeleteRockBand(string id)
  {
       return DeleteAsync(id);
  }

}

Now we can insert a record, run the indexer job and check the index’s schema via the portal.

Finally, we can perform a search using a client app and view our results!

You can find the source code for the whole solution here. ]]> https://docs.microsoft.com/archive/blogs/giventocode/how-to-use-azure-documentdb-with-azure-mobile-services Thu, 16 Oct 2014 15:24:22 GMT https://blogs.msdn.microsoft.com/giventocode/2014/10/16/how-to-use-azure-documentdb-with-azure-mobile-services/ With the release of DocumentDB, developers now have access to a scalable, nimble, and robust NoSQL platform as a service in Azure. I thought it would be a good idea to showcase how you can use DocumentDB with Azure Mobile Services. But before we dive into the details, and with the goal of providing some insights into how I implemented this integration, let’s discuss some aspects of the design of Azure Mobile Services with a .NET back end.

Note: This integration could be beneficial if you want to use some features of Azure Mobile Services, such as authentication, notifications, etc., while benefiting from a NoSQL database approach.

To get started, let’s cover some of the basics of Azure Mobile Services. First, it is important to notice that Azure Mobile Services with a .NET back end is built using the WebAPI, so controllers, models, routes, etc., are also the building blocks of the service. Another design consideration is that Azure Mobile Services separates the operations of the API and the mechanism that handles how the data is stored. Because of this separation of concerns, you can change where data comes from or where it is saved, without major changes to the implementation of the APIs. To achieve this, the design of Azure Mobile Services introduces the concept of a domain manager.

The role of the domain manager is to manage the semantics of the underlying repository of data and provide a consistent interface that allows the typical create, read, update, and delete (CRUD) scenarios.  On top of the domain manager, Azure Mobile Services provides a generic controller that exposes this functionality as REST operations. The controller also has a type parameter and is the type of entity representing the data that the client receives and sends to the API. This entity is often referred as a data transfer object (DTO). In the simplest case, the DTO reflects the underlying data model.

Note: There are scenarios in which you will want to use an existing database where it is necessary to expose a subset or a variation of data via the API. In such cases, there is another set of objects that represents the data model, and creating a DTO mapping process of some sort is necessary. The implementation of Azure Mobile Services includes a convenient mapper component (Automap), as well as a mapping domain manager to facilitate these tasks.

Now let’s discuss DocumentDB. Simply put, you can think of DocumentDB as a JSON repository where each element (resource) is addressable via a URI. At the top is your account followed by a set of databases. Each database can have one or more collections of documents. A document is a main data unit, and in the context of Azure Mobile Services, entities are documents. To reflect this I will use the term “document entity” going forward.

Note: You can create a new DocumentDB service instance through the preview portal only.

Following the design considerations discussed, we will create a domain manager that abstracts the nuances of persisting and retrieving documents from the DocumentDB instance and a controller that leverages it. So let’s get to it!

Document Domain Manager

For the implementation of the domain manager for DocumentDB, we will implement the methods shown in the diagram below, and to do this we will leverage the DocumentDB .NET SDK.

The methods are basically the same as those used for the interface IDomainManager<T>. Notice that we are not explicitly implementing this interface due to the constraint that makes instances of <T> implement the ITableData interface. ITableData defines basic properties that are not applicable for a document stored in DocumentDB.

You can check the full implementation here.

Note: This full implementation assumes that you have two settings in your config file: endpoint and authKey. You can get the values for these settings in the Azure portal.

The Controller

Next, we will need to implement a base (abstract) controller that expects the domain manager we just discussed. In addition, we need to define a base type for our document entities. The DocumentDB client has a handy class called Resource, which has some basic properties that reflect key metadata of a document entity as well as an ID property. So let’s require that all document entities derive from this class by placing a constraint in our controller.

The following figure shows the constraint and the methods we will need to implement. You can check the complete implementation here.

Playing with DocumentDB

So now we have everything we need to define a new entity and then easily implement an API that exposes the CRUD operations having DocumentDB as the back end.
Let’s create a Todo item entity.

public class DocumentTodoItem:Resource
    {

        [JsonProperty(PropertyName = "name")]
        public string Name { get; set; }

        [JsonProperty(PropertyName = "desc")]
        public string Description { get; set; }

        [JsonProperty(PropertyName = "isComplete")]
        public bool Completed { get; set; }
    }

public class TodoItemDocumentController:DocumentController<DocumentTodoItem>
    {
        protected override void Initialize(HttpControllerContext controllerContext)
        {
            base.Initialize(controllerContext);
            DomainManager = new DocumentEntityDomainManager<DocumentTodoItem>("AMSDocumentDB", "todolist", Request, Services);
        }

        public IQueryable<DocumentTodoItem> GetAllTodoItems()
        {
            return Query();
        }

        public SingleResult<DocumentTodoItem> GetTodoItem(string id)
        {
            return Lookup(id);
        }

        public Task<DocumentTodoItem> ReplaceTodoItem(string id, DocumentTodoItem item)
        {
            return ReplaceAsync(id, item);
        }

        public async Task<IHttpActionResult> PostTodoItem(DocumentTodoItem item)
        {
            var doc = await InsertAsync(item);     
            
            return CreatedAtRoute("DefaultApis", new { id = doc.Id }, doc);
        }

        public Task DeleteTodoItem(string id)
        {
            return DeleteAsync(id);
        }
    }

Let’s run it locally and create a Todo item.

Let’s check DocumentDB using Document Explorer via the portal—and voila!

Conclusion

In today’s blog post, I showed you how you can use DocumentDB as the back end for Azure Mobile Services. You can find the complete example here.

Maybe you have noticed the panel with a post’s recommendations next to the blog post itself and wondered how it works.

In today’s post I’ll describe the implementation of this feature, which leverages some really cool Azure technology including Azure Machine Learning (Azure ML), Azure Mobile Services and Azure Storage.

But first, let me give you a quick overview of the approach to developing this recommendations mechanism and the rationale behind it. The goal is simply to provide a set of recommendations of similar content directly from a blog post. One way to achieve this goal is to treat this as a categorization problem. Once all of the blog posts have a category assigned to them, the solution provides recommendations from the categories that the blog posts share.

If we go down this path, first we would define these categories and create a set of data (a training dataset) categorized according to our expectations. From this data, we would train a machine learning model for future categorization. In my case, this means that I’d need to categorize some of my data manually and select and train a model. Once the model is trained, I could use it to categorize the rest of the data automatically. If we were dealing with tens of thousands of articles or blog posts, the effort would be justifiable. However, for my blog that has significantly less content, I might just spend time tagging my content and doing the categorization myself.
 

Note: In the context of machine learning, the above-described approach is known as supervised learning.   

Now, say we could categorize the data without having to provide a training data set or the categories up front. The only decision we would make would be the number of groups of data. The similarity of the text would define what would go into each group.

This approach would be analogous to giving a person several blog posts and a number of bins and tasking the person with placing blog posts in bins corresponding to their degree of similarity. As such, each bin would contain a group of the most similar blog posts.

One algorithm that performs this type of analysis is K-Means clustering. K represents the number of centroids (centers of the clusters) that the algorithm will try to find in the data. In my scenario, you can think of the centroids as the bins or groups of data. This is the approach I used, so let’s get to it!

Note: One of the trade-offs with K-Means clustering is that there is not a deterministic way to know the optimal value of K. This is what is known as a NP-hard problem. Clustering is an example of unsupervised learning.

Data In

To get my blog’s data ready for K-Means using Azure ML, there are a few preparatory steps, including removing HTML, CSS, and the stop words. 

Note: “Stop words” are repeating words that are often removed from machine learning analysis due to their high frequency and limited semantic value. For example “the”.

Once the data is ready, I have to present it in a format that Azure ML can read. To implement all of these tasks, I leverage the job-scheduling capabilities of Azure Mobile Services.

The code shows, the relevant part of the implementation of the scheduled job that reads the blog’s data from the RSS feed, cleans it, and persist it in an Azure table (you can get the full code here).

    public class RssMLPrep : ScheduledJob
    {

        public async override Task ExecuteAsync()
        {

            using (var httpClient = new HttpClient())
            {

                var response = await httpClient.GetAsync(new Uri(CloudConfigurationManager.GetSetting("RssSource")));
                var data = await response.Content.ReadAsStringAsync();
                var xnodes = new XPathDocument(new MemoryStream(UTF8Encoding.UTF8.GetBytes(data)))
                              .CreateNavigator()
                              .Select("//item");

                var table = GetCloudTable();


                foreach (XPathNavigator node in xnodes)
                {
                    var title = node.SelectSingleNode("title").ToString();

                    //Include the title in the text and prepare text...
                    var text = PrepareText(string.Format("{0} {1}", title, node.SelectSingleNode("description").ToString()));

                    //hash the title to remove invalid chars and use it as the key
                    var rowKey = ComputeSH1Hash(title);
                    
                    var tableOp = TableOperation.InsertOrMerge(new BlogEntry()
                    {
                        Text = text,
                        PubDate = DateTime.Now,
                        Link = node.SelectSingleNode("link").ToString(),
                        RowKey = rowKey,
                        PartitionKey = "BlogEntries",
                        Title = title
                    });

                    await table.ExecuteAsync(tableOp);

                }

            }         
        }
      ...
}

The Experiment

Next, I will describe the implementation of the clustering process in Azure ML.

1.- Connecting the Modules and Configuring the Reader

First, we sign into Azure ML Studio and create a new experiment. From the list of modules on the left pane, locate and drop into the experiment the following modules: Reader, K-Means Clustering, Train Clustering Model, Feature Hashing, Execute R Script, Metadata Editor and Writer modules. Connect the modules according to the screenshot below.

Click on the Reader module and configure the properties so that it can read from the Azure table.

2.- Feature Hashing and Principal Component Analysis (PCA)

Next, we need to convert the blog’s text data into a format that K-Means can use. K-Means assumes a vector of n-dimensions as the input. Furthermore, since K-Means utilizes a Euclidian distance calculation to determine how close a vector is from the centroid, it is important that these dimensions contain numeric values and that these values are normalized.

Although it sounds a bit complex, fortunately a technique called feature hashing simplifies the implementation of these tasks. Additionally, Azure ML provides a Feature Hashing module that leverages the Vowpal Wabbit library to perform a 32-bit murmurhash v3 hashing.

To use this module, we must indicate the column with the text we want to hash, the hashing bit size, and the length of the N-grams. The figure below shows you the properties of the module.

Note: An N-grams value of 2 indicates that we will hash terms consisting of two words each.

The result of this module will be 4,096 new numeric columns for each row! This implies that K-Means will try to find K number of clusters in a space of 4,096 dimensions. This is a problem that is often referred to as the “curse of dimensionality” and specifically for K-Means is that the concept of distance becomes less precise as the number of dimensions grows. 

Note: Determining the number of columns is the hashing bit size parameter. Because we entered 12, this translates into 2^12 columns, which is 4,096.

Another technique commonly used to reduce the number of dimensions is Principal Component Analysis (PCA). To do this in Azure ML, we will need to implement the following R code using the Execute R Script module.

Note: The script selects the top 10 principal components. Depending on your data, however, you might find it beneficial to use this filter as a tuning parameter and return greater or fewer components

dataset1 <- maml.mapInputPort(1) # class: data.frame

#rowkey, title, link, pubdate, partitionkey, timestamp
reference_data <- dataset1[,1:7]

#pca on the hashed cols
pca_results <- prcomp(dataset1[,8:4103])

#take only the top 10 principal components
top_components <- data.frame(pca_results$x[,1:10])

#bind the results to the reference data
data.set <- cbind(reference_data,top_components)

maml.mapOutputPort("data.set");

At this point, we have the data ready for our model! As I mentioned earlier, there is no exact way to know the optimal value of K. However, one way to approach this is to think about the number of high-level topics or themes that your content covers. For my case, I thought that 5 would be a good starting point. The following image shows the configuration of the K-Means Model and Training Clustering Model modules.

Note: The initialization in K-Means could be a key factor in the quality of the outcome. By default, the algorithm picks the top dimensions as the starting points for the centroids. However, a better initialization that is geared toward avoiding some of the shortcomings of the default approach is K-Means++, and Azure ML provides this initialization optimization!

Important: When configuring the selected columns in Train Clustering Model module, make sure that you select the option Allow duplicates and preserve column order in selection.

4.- Writing Our Results and Running the Experiment

Before writing our results we need to convert the Assignments column to String. For this we will use the Metadata Editor module with the following properties.

Finally, we write our results in an Azure table using the Writer module. So let’s configure the Writer according to the figure below.

Ready, set, run the experiment by clicking on the Run button!

Data Out

We are almost there! What is left is to expose the results table via Azure Mobile services and the client code. The following code shows the relevant part of the implementation via a custom API. (You can get the full code here.)

public class BlogRecommendationsController : ApiController
{
    public ApiServices Services { get; set; }


    // GET api/BlogRecommendations
    [AuthorizeLevel(Microsoft.WindowsAzure.Mobile.Service.Security.AuthorizationLevel.Application)]
    public IEnumerable<BlogItem> Get(string title)
    {
        var resultsTable = GetCloudTable("BlogDataTableResults");

        var cluster =resultsTable.CreateQuery<BlogEntryResult>()
                    .Where<BlogEntryResult>(b => b.Title == title)
                    .Select<BlogEntryResult, string>
                    (r => r.Assignments)
                    .FirstOrDefault<string>();

        if (cluster != null)
        {
            return resultsTable.CreateQuery<BlogEntryResult>()
                    .Where<BlogEntryResult>(b => b.Assignments == cluster && b.Title != title)
                    .Select<BlogEntryResult, BlogItem>
                    (r => new BlogItem() { Title = r.Title, Link = r.Link });
        }

        return null;
    }
...
}

For the client, I am using the Azure Mobile Services HTML SDK. The following code shows the implementation of the client code:

$("#recommendation_panel").hide();

var client = new WindowsAzure.MobileServiceClient(
    "https://<YOUR MOBILE SERVICE>.azure-mobile.net/",
    "<YOUR_APP_KEY>"
);

//We are assuming the value in the HTML title tag is equal to the title of the RSS feed
var rawTitle = $(document).attr('title');

client.invokeApi("BlogRecommendations", {
        body: null,
        parameters: {title:rawTitle},
        method: "get"
    }).done(function (results) {      
        showRecommendations(results.result);        
    }, function(error) {
        //error handling...
    });

function showRecommendations(items){    
    if(items == null){
        return;
    }  

    if (items.length <= 0){
        return;
    }


    $("#recommendation_panel").show();
    var lst = $("#recommendation_body").append('<div class="list-group"/>'); 
    
    items.forEach(function(r,i,a) {
         lst.append('<a class="list-group-item" href="' + r.link + '">'+r.title+'</a>');   
    })

    $("#recommendation_header").append("Recommended Posts"); 
    $("#recommendation_footer").append("Powered by Azure Machine Learning"); 
}

Note: You will need to configure CORS for the Azure Mobile Service. CORS is supported in Azure Mobile Services with a .NET backend starting version 1.0.348. For more information, see this blog.

Final Thoughts!

In today’s post, I showed you how to implement a recommendations feature for your blog or web site quickly by utilizing the K-Means clustering algorithm in Azure ML.

In future blog posts, I will show you some of the optimizations I did, given the results I got from the approach outlined here. 

Stay tuned!

In today’s post, I’ll explore how you can use Azure Machine Learning to perform regression analysis on a dataset.

Regression analysis is a data mining process that identifies a model that depicts the correlation between an outcome and one or more predictors.

But, why is this important or useful?  Well, for one thing,  using a trained model,  you could predict an outcome given a set of known values. In other words, you have the ability to gain insights into a likely outcome that is yet to happen.  Interesting, right?

The Dataset

One very compelling source of data is the NYC Open Data portal. The portal features datasets containing a wide range of topics and categories related to New York City.  Information about schools, restaurants ratings, pot holes, just to mention a few, is available via the portal.

While browsing the data I came across the 311 Service Requests dataset. This dataset contains all the 311 service requests from 2010 to present.  If you are not familiar with 311, think of it as a customer service line for all the non-emergency requests to New York City’s government.  So I decided to  upload the dataset to Azure ML and use the visualization tools of Azure ML (Machine Learning) to find what’s the most frequent complaint of the very vocal New Yorkers.

Note:  To upload a dataset in a CSV format you will need to create a ML workspace via the Azure Portal, sign-in into ML Studio and create a new Experiment. More information here. 

              For more information about how to visualize your data here.

To my surprise, I learned that the most frequent complaints in 2013 were heating related.

The Predictor

I thought it’d be interesting to create a model to predict the number of heating complaints.  For the predictor of the number of heating complaints,  I think you’d agree that there must be a correlation between the temperature and the number of these complaints.  So next, I needed to find the average temperature of when the complaint was created.  And fortunately, there’s a dataset for that! NOAA National Climatic Data Center.

After creating a database on Azure, importing the data and a bit of T-SQL’s aerobics. I ended up with a dataset containing two columns: the number of heating complaints and the average temperature of when they occurred. I added a Reader node (to read data from my Azure SQL table) to my experiment and removed the original dataset. I am ready to go.

Note: In Azure ML you have the ability to train your models from datasets that you upload, but also from SQL Azure, Table/Blob Storage, HTTP endpoints and other sources. For more information see this.

Linear Regression

Let’s start with the simplest the regression: lineal regression.  In short, a linear model (identified from the process of regression) assumes that there’s a proportional relationship between to predictor (AvgDailyTemp)  and the outcome (Complaints).  So let’s train a Linear Model using Azure ML and our dataset.

The modeling process in Azure ML, at a high level consists of finding the parameters of your model from your dataset (Train Model), test it with a dataset, which could be a different dataset to the one you use for training (Score Model)  and then determine some  key quality factors of the results or evaluate against another model (Evaluate Model).

To do this, you need to drop a Train Model, Score Model, Evaluate Model and Linear Regression nodes into the experiment and connect them as depicted below.

Note: You could fine tune the linear regression process by changing the Solution Method (Ordinary Least Squares or Online Gradient Descent), whether to include an intercept and other parameters. For more information see here.

Next you need to configure the Train Model node to select the Complaints column –this how you tell Azure ML what is the predicted outcome of your model.

So let’s see how the model did.  The following image shows Azure ML’s visualization of the predicted values next to the scattered plot of the actual values.

Possion Regression and Log Scale

From the scatter plot of the actual values, you can tell that although the values are very scattered, the plot resembles an exponential function.  However, having an exponential model means that we would loose the ease of regression of the linear model. Fortunately, we can have both. We do this by just transforming a component of our model, in our case the outcome, into the  logarithm of number of complaints. And the transformation means that our model instead of predicting the number of complaints, will predict the log of the number of complaints.  Let’s plot that scenario.

As you can see from the plot the trend is linear, once we do the transformation. So we can either go back to our data and apply the log function to the Complaints column or we can use the Possion regression. The outcome of the Possion regression is precisely the natural log of the outcome variable.  Your experiment should look similar to the picture below.

Now if we plot the scored model vs the actual data the get the following.

By looking at the graph it seems to reflect better the actual shape and scale of values.

Closing notes

In this blog post I showed you how to visualize data,  create linear and Possion regressions using Azure ML.

Next, I’ll show you  how can use decision forests to get a model that better fits the data.

But first, let’s review a few key concepts about geo-locations and calculation of distances.

News: The Earth isn’t flat!

If you think of the values of longitude and latitude as a Cartesian coordinate system (in 2 dimensions) calculating distances between two points is a simple calculation - thank you Pythagoras.

But as early Greeks correctly deducted, the Earth’s shape resembles a sphere –although not a perfect one, it’s more of a flattened sphere. So if we use a trigonometric approach to  calculate the distance between two points, we’ll be off as the distance between two geo points is not a straight line but an arc segment on the Earth’s curvature.  In mathematics this is called a great-circle distance.

A good computational approach to calculate the great-circle distance consists of using the Haversine formula.  This formula provides acceptable accuracy and is less susceptible to  computational rounding errors when compared to other mathematically correct approaches such as the spherical law of cosines. 

Note: The actual Earth’s shape is elliptical, dented at the poles, so although the Haversine formula assumes a perfect sphere, the results are acceptable for most cases. 

For the implementation, I will use the Haversine formula in combination with a trigonometric approach, that I will introduce for optimization reasons.

Enough of Math and Theory, show me the code

First I need to create a GeoPoint entity and associate it to my data entity.  In this example, I will be using the default template for Azure Mobile Services with a .NET backend.  The template has an implementation of a back end for a to-do list.  What this means is that I will be attaching geo-location information to to-do items.

Here’s the code for the GeoPoint and TodoItem entities.

 public class GeoPoint:EntityData
 {
        public double Latitude { get; set; }
        public double Longitude { get; set; }
  }

 public class TodoItem : EntityData
 {
        public string Text { get; set; }

        public bool Complete { get; set; }

        public virtual GeoPoint Location { get; set; }
 }

What’s really cool is that with this code and relaying on the existing TableController for my TodoItem entity , I can have my mobile application send TodoItems to my back end with geo information attached to them and the information will be persisted in the database, without any more coding!

With the data stored in a database, let’s expose an action that allows a client to retrieve the list of items closest to a geo location and within a maximum distance from it.

From the logic’s flow perspective, for each item in the database, I need to calculate the distance to the given geo-location and then check if it’s less than the maximum distance. Doing this for all the items, implies the need to enumerate all of them,  calculate the distance and check if it’s less than the maximum distance. 

Since calculating the actual distance using the Haversine formula is a complex mathematical operation,  in the code below, first I narrowed the list down by using a simpler trigonometric calculation . The trigonometric approach won’t provide an accurate distance so we can’t use it to determine if it’s within the maximum distance requested by the client, however it will provide the list of the closest items to the geo-point nevertheless.

From the narrowed list, I calculate the distance, using the Haversine formula and then determine if it’s within range and then return the items to the client.

Note: An additional benefit of this approach is that it also shares the burden of the calculation between  the application and database servers. If we calculate the Haversine result within the SQL statement then all the heavy lifting will reside on the database.

public IEnumerable<TodoItem> GetTodoItemsNearGeoPoint(double longitude, double latitude, int count, int maxDistance)
{
            var closest = Query()
               .OrderBy<TodoItem, double>(t => Math.Pow(longitude - t.Location.Longitude, 2) + Math.Pow(latitude - t.Location.Latitude, 2))
                .Take<TodoItem>(count);

            var withinMaxDistance = new List<TodoItem>();

            foreach (var item in closest)
            {
                var distance = CaculateGreatCircle(longitude, latitude, item.Location.Longitude, item.Location.Latitude);

                if (distance <= maxDistance)
                {
                    withinMaxDistance.Add(item);
                }

            }

            return withinMaxDistance;
        }

private double CaculateGreatCircle(double long1, double lat1, double long2, double lat2)
        {
            var R = 3959; //In Miles           
            var dlong = ToRadians(long1-long2);
            var dLat = ToRadians(lat1-lat2);
            var arc = Math.Sqrt(
                        Math.Pow(Math.Sin(dLat) / 2, 2)
                        +
                        Math.Cos(ToRadians(lat1)) *
                        Math.Cos(ToRadians(lat2)) *
                        Math.Pow(Math.Sin(dlong) / 2, 2)
                    );
            return R * 2 * Math.Atan2(arc,Math.Sqrt(1- arc));
                                       
        }

private double ToRadians(double value)
        {
            return Math.PI * value / 180;
        }

Finally,  please note that distance is assumed to be in Miles.

Conclusion

In this blog post, I show how you can easily associate geo location information to your objects using Azure Mobile Services as well as the implementation of server side logic to retrieve objects that are closest to a given geo-location.

The CTP also includes the next version of ASP.NET. ASP.NET vNext is a streamlined stack for building modern web applications for both cloud and on premises servers.

Visual Studio “14” continues to push forward the Visual C++ compiler on standards conformance for several features such as C++ 11 user-defined literals, inline namespaces, noexcept  as well as C++ 14 generalized lambda capture and std::user-defined literals.

Please note that these are very early bits and not recommended for production scenarios. For developers is also recommended to install this CTP on a clean machine or VM. Side-by-side installations with Visual Studio 2013 are not allowed.

Learn More: Soma’s Blog

Download: Visual Studio “14” CTP 1

KB: Visual Studio “14” CTP 1 KB]]> https://docs.microsoft.com/archive/blogs/giventocode/the-apps-that-powered-the-lumia-arc-of-wonder Fri, 23 May 2014 20:28:17 GMT https://blogs.msdn.microsoft.com/giventocode/2014/05/23/the-apps-that-powered-the-lumia-arc-of-wonder/ First blog of the series where I will discuss the apps behind the viral Nokia commercial Lumia Moments - Arc of Wonder.

Check it out !

From idea to execution...

Creative Iterations

“Fail fast” was the mantra. An aggressive schedule demanded the fail fast mindset. We had to think simplest way to achieve the goal and try it. Ensuring that all the phones could capture a sequence of pictures with a latency of less than 50ms(1ms is a 1000th of a second) was not a simple goal. We wanted to avoid point to point connections to each device to push the “shutter”. The rationale behind this was that in such cases, the triggering app would have to sequentially enumerate each connection to send data, hence introducing the risk of triggering delays between the first and last devices.

Initially, we considered an approach where we would rely on the device’s clock and schedule when the picture will be taken. The phones would then poll a cloud resource to receive the time they need to take the picture.  We quickly realized that the devices were off by a few seconds despite having clock syncing enabled. We wanted to give this approach another try so we implemented a time adjustment algorithm that uses a central point of reference in the cloud. Even though we brought the latency down to 150ms, we were  still off from our goal.

At this point, we iterated again and began to work on a new solution based on network multicasting. Network multicasting is a form of networking communication where the data is addressed simultaneously to a group of devices.

The new approach showed promising results in our first tests.  Odd discrepancies between certain devices had to be tackled, caused by the camera’s flash and the automatic lighting adjustment of the device. But after a few iterations of tweaks we were able to meet our latency goal!

The iterative process coupled with Visual Studio enabled us to adapt quickly, code with superb productivity and implement these ideas with a few lines of code.

One Controller to Control them All

Each phone runs a custom camera app built from the grounds up for this project that takes advantage of the Lumia’s 1020 maximum picture resolution. The app listen listens for a network “trigger”  that fires the camera and saves the picture in the device.

A key concern was avoiding manipulating the devices once they were mounted on the rig. The rig is a custom built metal arc where each device was purposely placed to take a 180 degrees’ perspective of the same moment in time. Great for making the effect possible, but not for manipulating each device individually, therefore having the ability to set the camera's settings for all the devices from a central location was a chief goal.  So taking advange of the same networking approach we used for the trigger we implemented a mechanism where each device would "listen" for the camera settings as well.

Like any project, the aggressive schedule kept looming while the list of new challenges was growing. One of those challenges was to figure out how the director would trigger the shooting sequence and the broadcasting of the camera settings.This functionality was better achieve with a bigger form factor made sense and a Surface was the natural choice.

From the development’s perspective, this meant that we had to create another app. This app would need to send the execution trigger as well as the camera settings to all the phones. For this purpose, the WinRT networking APIs provided us with the stepping stone that we needed to create a Windows Store App in a very short period of time.

Testing with 50 Devices

But, wait, how do you measure latency? Specially considering that the internal clocks of the devices could be out of sync by a few seconds and this rules out any time-stamp or logging mechanism generated from the phones. An impossible task?  Well, not really. Our solution was quite simple. To test, we pointed the cameras to a large screen displaying a stopwatch counting down to the milliseconds. Once we’ve triggered the capture, we checked the time value captured in each photo.  Latency measured!

With so many individual moving parts, there was no way we could anticipate the number and types of issues we were going to face once the components were integrated and tested as a whole. The first challenge we encountered was the arm which held the phone on the rig obstructed the viewfinder. To fix this issue, we made the viewfinder resizable.

A more pressing concern was that we could not get a good sense of whether the devices were receiving the execution trigger and the settings. With the multicasting mechanism we had low latency to the expense of reliability. So we decided to show on each device a running log of what was received so we check after each shooting.

Simply put, we strived towards finding simple and practical solutions to our challenges.

Shooting Day

At the shooting, the hard work paid off and the applications performed as expected, delivering foundation for the awesome video that Paul Trillo created.

Wrapping up!

In this blog post, I provided you a quick overview into how we approached the creation of software for the Lumia Moments commercial. In my next blog posts, I will dive deeper into some key aspects of the implementation of the software.

Stay tune!

Update: I made some changes to this blog post since I first published it to include some additional details about the phone app and how we tested latency.