Cognitive Services: Image Text Sentiment Analysis using Vision and Text Analytics Api
Introduction
With the ever growing use of social media platforms like FaceBook, Twitter, Linked In to promote business, analyzing what the users feel and what feedback they share regarding the business is going to be of paramount importance. One such scenario is where the users post there feedback in images on the concerned company's social media pages. It is of great importance that the companies analyze what the users post on their social media pages as it will give them an overall idea of the sentiment of the users towards the services offered by the company, which in turn will help them plan their marketing strategies accordingly.
Microsoft Cognitive Services
As per MicroSoft,
Microsoft Cognitive Services( formerly Project Oxford) are a set of APIs, SDKs and services available to the developers to make their applications more intelligent, engaging and discover able. Microsoft Cognitive Services expands on Microsoft's evolving portfolio of machine learning APIs and enables developers to easily add intelligent features such as emotion and video detection; facial, speech and vision recognition; and speech and language understanding - into their applications. Our vision is for more personal computing experiences and enhanced productivity aided by systems that increasingly can see hear, speak, understand and even begin to reason.
Scope
This article explains the concept of the Sentiment analysis of the text detected from an Image using a Sample Console Application. The article assumes that the user is familiar with the basic concepts of C# and know how to consume the REST Apis in the C# code. In This Sample Application, images from a SampleImage folder are read and then the image is sent to the Computer Vision Api to detect the text from the image. Once the Computer Vision api returns the json payload, the detected text is sent to the Text Analytics Api to detect the sentiment score.
Pre Implementation
Computer Vision Api Account
In order to use the Computer Vision Api in the Sample Application, first an Api account for the Computer Vision Api needs to be created. Once this is done, the Api will be available to integrate the Computer Vision Api in the Sample Application. Following screenshot shows the process to do so.
Once the API account is created, select the account from the dashboard and following window is visible, the access keys and end point are required from this window which will be used to create a connection to the Computer Vision Api.
Text Analytics Api Account
In order to use the Text Analytics Api in the Sample Application, first an Api account for the Text Analytics Api needs to be created. Once this is done, the Api will be available to integrate the Text Analytics in the Sample Application. Following screenshot shows the process to do so.
Once the API account is created, select the account from the dashboard and following window is visible, the access keys and end point are required from this window which will be used to create a connection to the Text Analytics API.
Design
Constants
The constants class is used to store the constant values that are used across the application. Following is the constants class used in the application.
01.using System;
02.using System.Collections.Generic;
03.using System.Linq;
04.using System.Text;
05.using System.Threading.Tasks;
06.
07.namespace ImageTextSentimentDetection
08.{
09. public class Constants
10. {
11. /// <summary>
12. /// Key Name that is used to pass the subscrption Key Value in the Request Headers
13. /// </summary>
14. public static string SubscriptionKeyName = "Ocp-Apim-Subscription-Key";
15.
16. /// <summary>
17. /// Uri for the Computer Vision APi to which the Request Will be routed
18. /// </summary>
19. public static string VisionApiUri = "https://southeastasia.api.cognitive.microsoft.com/vision/v1.0";
20.
21. /// <summary>
22. /// Uri for the Text Analytics Api to which the Request will be routed.
23. /// </summary>
24. public static string TextAnalyticsApiUri = "https://southeastasia.api.cognitive.microsoft.com/text/analytics/v2.0";
25.
26. /// <summary>
27. /// Path to the SampleImages Folder
28. /// </summary>
29. public static string SampleImagesFolderPath = @"..\..\SampleImages";
30.
31. /// <summary>
32. /// Vision APi Subscription Key Value. This needs to be populated before Starting the Sample
33. /// </summary>
34. public static string VisionApiSubcriptionKey = "Enter key here";
35.
36. /// <summary>
37. /// Text Analytics Subscription Key value. This Needs to be Populated before Starting the Sample
38. /// </summary>
39. // public static string TextAnalyticsApiSubscriptionKey = "Enter key here";
40.
41.
42. }
43.}
Computer Vision Api Response Class
This class is used to convert the json payload received from the Computer Vision Api into a C# class which can be used later on for extracting text. The class is generated using the sample json payload received from the Computer Vision Api. The class structure will change if the response received is change,
01.// To parse this JSON data, add NuGet 'Newtonsoft.Json' then do:
02.//
03.// using ImageTextSentimentDetection;
04.//
05.// var data = ComputerVisionSuccessFullResponseClass.FromJson(jsonString);
06.
07.namespace ImageTextSentimentDetection
08.{
09. using System;
10. using System.Net;
11. using System.Collections.Generic;
12.
13. using Newtonsoft.Json;
14.
15. public partial class ComputerVisionSuccessFullResponseClass
16. {
17. [JsonProperty("language")]
18. public string Language { get; set; }
19.
20. [JsonProperty("textAngle")]
21. public double TextAngle { get; set; }
22.
23. [JsonProperty("orientation")]
24. public string Orientation { get; set; }
25.
26. [JsonProperty("regions")]
27. public Region[] Regions { get; set; }
28. }
29.
30. public partial class Region
31. {
32. [JsonProperty("boundingBox")]
33. public string BoundingBox { get; set; }
34.
35. [JsonProperty("lines")]
36. public Line[] Lines { get; set; }
37. }
38.
39. public partial class Line
40. {
41. [JsonProperty("boundingBox")]
42. public string BoundingBox { get; set; }
43.
44. [JsonProperty("words")]
45. public Word[] Words { get; set; }
46. }
47.
48. public partial class Word
49. {
50. [JsonProperty("boundingBox")]
51. public string BoundingBox { get; set; }
52.
53. [JsonProperty("text")]
54. public string Text { get; set; }
55. }
56.
57. public partial class ComputerVisionSuccessFullResponseClass
58. {
59. public static ComputerVisionSuccessFullResponseClass FromJson(string json)
60. {
61. return JsonConvert.DeserializeObject<ComputerVisionSuccessFullResponseClass>(json, Converter.Settings);
62. }
63. }
64.
65. public static class Serialize
66. {
67. public static string ToJson(this ComputerVisionSuccessFullResponseClass self)
68. {
69. return JsonConvert.SerializeObject(self, Converter.Settings);
70. }
71. }
72.
73. public class Converter
74. {
75. public static readonly JsonSerializerSettings Settings = new JsonSerializerSettings
76. {
77. MetadataPropertyHandling = MetadataPropertyHandling.Ignore,
78. DateParseHandling = DateParseHandling.None,
79. };
80. }
81.}
Text Analytics Response Class
This class is used to convert the json payload received from the Text Analytics Api into a C# class which can be used later on for extracting text. The class is generated using the sample json payload received from the Text Analytics Api. The class structure will change if the response received is change,
01.// To parse this JSON data, add NuGet 'Newtonsoft.Json' then do:
02.//
03.// using ImageTextSentimentDetection;
04.//
05.// var data = TextAnalyticsResponseClass.FromJson(jsonString);
06.
07.namespace ImageTextSentimentDetection
08.{
09. using System;
10. using System.Net;
11. using System.Collections.Generic;
12.
13. using Newtonsoft.Json;
14.
15. public partial class TextAnalyticsResponseClass
16. {
17. [JsonProperty("documents")]
18. public Document[] Documents { get; set; }
19.
20. [JsonProperty("errors")]
21. public Error[] Errors { get; set; }
22. }
23.
24. public partial class Document
25. {
26. [JsonProperty("score")]
27. public long Score { get; set; }
28.
29. [JsonProperty("id")]
30. public string Id { get; set; }
31. }
32.
33. public partial class Error
34. {
35. [JsonProperty("id")]
36. public string Id { get; set; }
37.
38. [JsonProperty("message")]
39. public string Message { get; set; }
40. }
41.
42. public partial class TextAnalyticsResponseClass
43. {
44. public static TextAnalyticsResponseClass FromJson(string json)
45. {
46. return JsonConvert.DeserializeObject<TextAnalyticsResponseClass>(json, Converter.Settings);
47. }
48. }
49.
50.}
Helper Classes
Helper classes are created to cater to each of the Cognitive Api calls. One class caters to the Computer Vision Api call while other class caters to the Text Analytics Api call. Each of them is discussed below.
VisionHelper
This class caters to the calls made to the Computer Vision Api. In this current Sample the DetectTextInImage method in the class is used to call the Computer Vision Api. This method passes the image to the api in 'application/octet-stream' format. It deserializes the json payload into the class created earlier and extracts out the detected text. The class is as follows.
01.using System;
02.using System.Collections.Generic;
03.using System.Linq;
04.using System.Text;
05.using System.Threading.Tasks;
06.using Newtonsoft.Json.Serialization;
07.using System.Net.Http;
08.using System.Net.Http.Headers;
09.using System.Web;
10.
11.namespace ImageTextSentimentDetection
12.{
13. public class VisionHelper
14. {
15. public static string DetectTextInImage(byte[] imageBytes)
16. {
17. string detectedText = String.Empty;
18.
19. var queryString = HttpUtility.ParseQueryString(String.Empty);
20.
21. HttpClient client = new HttpClient();
22.
23. using (var content = new ByteArrayContent(imageBytes))
24. {
25. client.DefaultRequestHeaders.Add(Constants.SubscriptionKeyName, Constants.VisionApiSubcriptionKey);
26.
27. queryString["language"] = "unk";
28. queryString["detectOrientation "] = "false";
29. content.Headers.ContentType = new MediaTypeHeaderValue("application/octet-stream");
30. var uriString = Constants.VisionApiUri + "/ocr?" + queryString;
31. HttpResponseMessage response = new HttpResponseMessage();
32. response = client.PostAsync(uriString, content).Result;
33.
34. var responseJson = response.Content.ReadAsStringAsync().Result;
35.
36. if (response.StatusCode == System.Net.HttpStatusCode.OK)
37. {
38. var data = ComputerVisionSuccessFullResponseClass.FromJson(responseJson);
39.
40. foreach (Region region in data.Regions)
41. {
42. foreach (Line line in region.Lines)
43. {
44. foreach (Word word in line.Words)
45. {
46. detectedText = detectedText + " " + word.Text;
47. }
48. }
49. }
50. }
51. else
52. {
53. detectedText = "Error Occured While Calling Computer Vision Api";
54. }
55. return detectedText;
56. }
57.
58. }
59. }
60.}
SentimentHelper
This class caters to the Text Analytics Api call from the Sample Application. In current context, the DetectSentiment method in the class accepts the detected string as input. It then creates the json payload and then invokes the Text Analytics Api. It Deserializes the json payload received from the api call into the response class and extracts out the sentiment score. This method is designed to pass only one detected text at a time to the Text Analytics Api, but it can be modified as required. The class is as follows.
01.using System;
02.using System.Collections.Generic;
03.using System.Linq;
04.using System.Text;
05.using System.Threading.Tasks;
06.using Newtonsoft.Json;
07.using System.Net.Http;
08.using System.Net.Http.Headers;
09.using System.Web;
10.
11.namespace ImageTextSentimentDetection
12.{
13. public class SentimentHelper
14. {
15. public static string DetectSentiment(string detectedText)
16. {
17. string sentimentScore = String.Empty;
18. string jsonText = "{\"documents\":[{\"language\":\"en\",\"id\":\"1\",\"text\":\"" + detectedText + "\"}]}";
19.
20. HttpClient client = new HttpClient();
21. byte[] detectedTextBytes = Encoding.UTF8.GetBytes(jsonText);
22. using (var content = new ByteArrayContent(detectedTextBytes))
23. {
24. var queryString = HttpUtility.ParseQueryString(String.Empty);
25. client.DefaultRequestHeaders.Add(Constants.SubscriptionKeyName, Constants.TextAnalyticsApiSubscriptionKey);
26. var uriString = Constants.TextAnalyticsApiUri + "/sentiment?" + queryString;
27. content.Headers.ContentType = new MediaTypeHeaderValue("application/json");
28. HttpResponseMessage response = new HttpResponseMessage();
29. response = client.PostAsync(uriString, content).Result;
30.
31. if (response.StatusCode == System.Net.HttpStatusCode.OK)
32. {
33. var data = TextAnalyticsResponseClass.FromJson(response.Content.ReadAsStringAsync().Result);
34. foreach (var document in data.Documents)
35. {
36. sentimentScore = Convert.ToString(document.Score);
37. }
38. }
39. else
40. {
41. sentimentScore = "Error";
42. }
43.
44. }
45.
46.
47. return sentimentScore;
48. }
49.
50. }
51.}
Console Application
The console application, greets the user and advises user to ensure that the Subscription keys for the Computewr Vision Api and Text Analytics Api are generated and added to the Constants class. It then asks the user to confirm to proceed by Accepting [Y/N] input. Once done, it reads the files stored in the SampleImages projects and calls the helpers on each of the images. Following is the Console Application class.
01.using System;
02.using System.Collections.Generic;
03.using System.Linq;
04.using System.Text;
05.using System.Threading.Tasks;
06.using System.IO;
07.using System.Reflection;
08.using System.Globalization;
09.namespace ImageTextSentimentDetection
10.{
11. public class ImageTextSentimentDetctionApp
12. {
13. static void Main(string[] args)
14. {
15. string visionApiSubcriptionKey = String.Empty;
16. string textAnalyticsApiSubscriptionKey = String.Empty;
17.
18. Console.WriteLine("Welcome to the Image Text Detection Sample Application");
19. Console.WriteLine("This Application will Get the Images From the SampleImages Folder");
20. Console.WriteLine("Once the Image is read, the Sample APplication Will call Computer Vision Api and Try to Detect Text in the Image"); ;
21. Console.WriteLine("ON SuccessFull Detection of text, the Application will use Text Analystics To Detect The Sentiment of APi");
22. Console.WriteLine("To Run this Sample Application, You need to generate the Subscription Key for the Computer Vision and Text Analytics Api.");
23. Console.WriteLine("Do You Wish To Continue? Press Y for yes and N for No");
24. try
25. {
26. if (Convert.ToString(Console.ReadKey().Key).ToUpper() == "Y")
27. {
28. Console.WriteLine(Environment.NewLine);
29. Console.WriteLine("Starting The Application Flow");
30.
31. foreach (string imageFile in Directory.GetFiles(Constants.SampleImagesFolderPath, "*.jpg"))
32. {
33. var imageFileInfo = new FileInfo(imageFile);
34.
35. byte[] imageBytes = File.ReadAllBytes(imageFileInfo.FullName);
36.
37. Console.WriteLine("File Name of the FileRead :" + imageFileInfo.Name);
38. Console.WriteLine("Initiating the Image TextDetection");
39. string detectedText = VisionHelper.DetectTextInImage(imageBytes);
40. if (detectedText.Contains("Error"))
41. break;
42. Console.WriteLine("Text Detected in Image:" + detectedText);
43. Console.WriteLine("Starting Sentiment Score analysis for the detected text");
44. string sentimentScore = SentimentHelper.DetectSentiment(detectedText);
45. if (sentimentScore.Contains("Error"))
46. break;
47. Console.WriteLine("Sentiment Score for the Detected Text is :" + sentimentScore);
48. }
49. }
50. else
51. {
52. Console.WriteLine("Thank You For Visiting the Application Will Exit Now.");
53. }
54. }
55. catch (Exception ex)
56. {
57. Console.WriteLine("Sorry an Exception Was Caught!!");
58. Console.WriteLine("Following are the Exception Details");
59. Console.WriteLine(ex.Message);
60. }
61. Console.WriteLine("Press Any Key To Exit");
62. Console.ReadKey();
63.
64. }
65. }
66.}
Testing
Following are the images that were used to test the Sample application.
[
](resources/6036.Worst-Experience.JPG)
The Result obtained from the Test Console Application is as follows
Actual result for the text analysis for the text detected from above test messages when done on the Microsoft Demo Page ( For Details See References) are shown below.
Visual result for the sentiment analysis of the text in the first test image is as shown below.
Visual result for the sentiment analysis of the text in the second test image is as shown below.
Conclusion
The test results as observed from the Sample Console Application as well as from the Microsoft Demo Tool are exactly the same proving the successful consumption of the Computer Vision Api and Text Analytics Api.
Source Code
The source code can be found at the TechNet Gallery at Source Code For Image Text Sentiment Analysis using Vision and Text Analytics Api
See Also
Following links can be used for extra reading related to the topic discussed in this article.
- Logic Apps: Optical Character Recognition From Image and Detected Text Analysis Using Microsoft Cognitive Computer Vision Api, Text Analytics Api and Logic Apps
- Getting started with Cognitive Services - Vision
- Logic Apps: Face Verification Using Microsoft Cognitive Services Face Api and Logic Apps
- Cognitive Services: Emotion API Overview
References
Following material was referred to while writing this article.