Cognitive Services: Emotion API Overview

Overview

BrowserPower is a simple game leveraging the power of Azure Cognitive Services - Vision API.  The premise is simple, given clues, find the best image from the web that matches it.  The clues might be a herd of sheep or a particular celebrity eating a cheeseburger.  The game itself is not as important as the technology used to create it.
This wiki highlights extending the game to support the Emotions API.

Note: SendGrid was highlighted in a previous TechNet Wiki article.
Note: Configuration was highlighted in a previous TechNet Wiki article.

Background

The game grew from the wiki article: Getting started with Cognitive Services - Vision.  Playing the game is simple: An internet search game using Azure Cognitive Services. The project has been uploaded to MSDN as it is an interesting illustration of building an ASP.NET Core Azure Web App from the ASP.NET Core Web Application template and implements several of steps in the ASP.NET Core Tutorials.

The source can be located here.

Adding Emotion API

Adding the Emotion API to a Cognitive Services subscription required creating a new service in the Cognitive Service blade:

When creating the Emotion API, we chose to add the new service into the same resource group as the other resources.  Also illustrated in the picture below is the limit of having only one free service and its limitations:

Emotion API Blade 

As expected from Azure services, the Emotion API provides a rich set of management capabilities including a nice overview showing the total calls/errors as well as an indication of latency:

Calling the Emotion API

Adding a call to the Emotion API was straightforward as the API was designed similarly to the Vision API.  The best example is actually from the documentation so please review for a complete example.  For BrowserPower, this required three main steps.  First, the image is retrieved using an HttpClient as a byte array:

The byte array is then submitted to the Emotion API also using an HttpClient:

private string  CallService(CongitiveServiceOption option, string uri, byte[] byteData)
{
    var client = new  HttpClient();
 
    // Request headers. NOTE: Replace this example key with a valid subscription key.
    client.DefaultRequestHeaders.Add("Ocp-Apim-Subscription-Key", option.ApiKey);
 
    HttpResponseMessage response;
 
    using (var content = new ByteArrayContent(byteData))
    {
        content.Headers.ContentType = new  MediaTypeHeaderValue("application/octet-stream");
        response = client.PostAsync(option.BaseUrl + uri, content).Result;
 
        if (response.Content.Headers.ContentType.MediaType ==  "image/jpeg")
        {
            var base64 = Convert.ToBase64String(response.Content.ReadAsByteArrayAsync().Result);
            return String.Format("data:image/gif;base64,{0}", base64);
        }
        else
            return response.Content.ReadAsStringAsync().Result;
 }

The last step is to parse the JSON result:

private async Task<Emotion[]> GetEmotions(byte[] byteData)
{
    return await Task.Run(() => {
        var emotion = _cognitiveService.GetEmotion(byteData);
        return JsonConvert.DeserializeObject<Emotion[]>(emotion);
    });
}
 
public class  Emotion
{
    public FaceRectangle faceRectangle { get; set; }
    public Scores scores { get; set; }
}
 
public class  FaceRectangle
{
    public int  height { get; set; }
    public int  left { get; set; }
    public int  top { get; set; }
    public int  width { get; set; }
}
 
public class  Scores
{
    public float  anger { get; set; }
    public float  contempt { get; set; }
    public float  disgust { get; set; }
    public float  fear { get; set; }
    public float  happiness { get; set; }
    public float  neutral { get; set; }
    public float  sadness { get; set; }
    public float  surprise { get; set; }
}

Summary

Using the Emotion API added a new dimension to BrowserPower.  Now a score will be awarded depending on if the image supplied matches the emotion.  For example, if the clue requires a sad man around thirty, a matching image containing a sad individual would score higher than a surprised one.