Connect(); 2017
Volume 32 Number 13
Artificial Intelligence - Getting Started with Microsoft AI
By Joseph Sirosh; 2017
Software developers are quickly adopting Artificial Intelligence (AI) technologies, such as natural language understanding, sentiment analysis, speech recognition, image understanding and machine learning (ML). Across a broad range of industries and sectors, AI-infused software applications and cloud services drive innovative customer experiences, augment human capabilities and transform how we live, work and play. New tools, cloud-hosted APIs and platforms make it even easier to build such applications.
Modern AI applications live at the intersection of cloud computing, data platforms and AI tools. The cloud provides a powerful foundation for elastic compute and storage, while supporting special-purpose hardware such as graphics processing units (GPUs) that accelerate demanding calculations. It also enables connectivity, identity, application monitoring and the Internet of Things (IoT). Data platforms in the cloud can ingest and integrate massive volumes of data, use databases and data lakes to transform and analyze the data, and build real-time data-driven applications. Layered upon these capabilities are AI tools and algorithms that help developers build models from the data for targeted intelligent scenarios, and deploy them in a hosted AI application.
What are some examples of such applications? Here are a few examples:
- Health Care: Doctors in the Microsoft Intelligent Network for Eyecare (MINE) leverage AI to improve patient outcomes for eye surgeries by identifying optimal surgical parameters to personalize treatment and maximize the probability of success. At the Cochrane Transform Project, AI is used to improve and streamline the nonprofit organization’s comprehensive reviews of health care interventions, with the goal of identifying the best treatments and interventions for patients. AI is used to analyze thousands of research studies, sharply reducing the time spent on manual reviews and freeing staff to focus on more urgent tasks.
- Manufacturing: Jabil, one of the world’s leading design and manufacturing companies, uses AI to optically inspect printed circuit boards and detect manufacturing defects. AI is also used in the manufacturing environment to monitor equipment to predict declining efficiency and impending failure of machines.
- Retail: Lowe’s partnered with Microsoft to create a kitchen remodeling design experience driven by AI. The Lowe’s customer shares his or her dream kitchen photos with a design specialist, who uses an AI-powered application to gain deep insight into the style and preferences of the customer. These findings are used to generate a match from the Lowe’s dream kitchen collection, which is then shown in a mixed-reality environment so the customer can interact with the products (such as cabinets, counter tops and appliances). Like Lowe’s, many retailers are turning to AI to re-imagine the retail experience for consumers.
Building these kinds of AI applications requires the integration of numerous components. To help eliminate the heavy lifting involved in building end-to-end systems, Microsoft provides a powerful AI platform, composed of a set of loosely coupled but highly interoperable services on Microsoft Azure. This article provides an overview of this platform and points to valuable resources for getting started developing exciting AI applications.
Microsoft AI Platform
The Microsoft AI platform provides a suite of powerful tools, such as the Bot Framework, Cognitive Services, Azure Machine Learning and many more. These tools allow developers to easily and quickly infuse AI into their applications and scenarios, enabling new, intelligent experiences for their users.
Powered by the enterprise-ready capabilities of Azure, the Microsoft AI platform presents a rich set of interoperable services, APIs, libraries, frameworks and tools that developers can leverage to build smart applications. The Microsoft AI platform consists of three core areas: AI Services, AI Infrastructure and AI Tools. Let’s start with a quick tour of the AI platform, shown in Figure 1.
.png "The Microsoft AI Platform")
Figure 1 The Microsoft AI Platform
Cognitive Services: Trained services like Cognitive Services enable you to jumpstart development of your AI applications, without requiring you or your data science team to develop and train the models. Cognitive Services features a rich set of instant AI capabilities that you can use. These AI capabilities are organized into the following categories: vision, speech, language, knowledge and search.
The Cognitive Services APIs enable you to leverage powerful computer vision algorithms that have been pre-trained to recognize things like different face attributes, landmarks, celebrities, gender, emotion, and printed or written words (Optical Character Recognition, or OCR). Powerful language capabilities can recognize commands from users, analyze key phrases, perform translations and spell check, and more.
Customized Computer Vision Models: As you explore Cognitive Services to develop your AI applications, you may find that you need to further customize the models using your own data. You can do that with Custom Vision services (customvision.ai). Custom Vision lets you bring your own data, and use it to train your computer vision models. Underneath the hood, state-of-the-art transfer learning techniques leverage existing pre-trained computer vision models, and evolve them to learn about the new images you upload to the Custom Vision service, as depicted in Figure 2. This state-of-the-art system allows you to develop a highly performing computer vision model in just minutes.
.png "Using CustomVision.AI to Build a Custom Travel Attractions Computer Vision Model")
Figure 2 Using CustomVision.AI to Build a Custom Travel Attractions Computer Vision Model
Custom Machine Learning and Deep Learning Models: As you work on various use cases, data scientists in your organization might need to develop and customize deep learning models, using various deep learning toolkits. The Microsoft AI platform provides an open and flexible environment for that deep learning. Azure Machine Learning empowers data scientists to build, develop and manage models at scale, while data stores like CosmosDB, SQL DB, SQL Data Warehouse (DW) and Azure Data Lake (ADL) provide access to the structured and unstructured data that inform your ML and deep learning models.
With Azure Machine Learning, you can easily train your models in Spark, run them on Azure Deep Learning Virtual Machines (DLVM), or process them on a managed GPU cluster with Batch AI, and more. Azure Machine Learning experimentation and model services boost productivity by helping you keep track of your projects, enabling you to train on both local and remote compute infrastructures, create containers for model deployment, and manage and monitor the behavior of models.
Bots provide exciting new ways to engage with customers and employees, helping them complete tasks. The Bot Framework provides a rich set of capabilities for conversational AI, so you can develop powerful new bots that interact with your customers and employees via Web sites, applications, text/SMS, Skype and more.
Tools like Visual Studio Tools for AI, Azure Machine Learning Studio and Azure Machine Learning Workbench provide a great starting point to get started building innovative, intelligent AI applications.
Let’s get started with Microsoft AI by using the various services to build an AI application that leverages the intelligent cloud and can be deployed to the intelligent edge. I’ll start with Cognitive Services, then move on to building custom models with Azure Machine Learning. I’ll finish with a dive into the Bot Framework and show how you can turn any bot into an intelligent bot powered by Microsoft AI.
Cognitive Services
You can add AI capabilities to any .NET app you’re developing using Cognitive Services. Let’s get started with the Intelligent Kiosk sample app found on the .NET Machine Learning and AI Web site at bit.ly/2yNtRpF. You can access the source code from bit.ly/2zysSXJ.
The Intelligent Kiosk sample app shows how to use Cognitive Services in different scenarios, including:
- Customizing product recommendations based on detected gender and age of visitors.
- Building a real-time AI pipeline to analyze visitor demographics for retail.
- Performing automatic photo capture and face identification.
Let’s consider the scenario of building a real-time AI pipeline to analyze the demographics of people visiting a retail store called Realtime Crowd Insights. In this scenario, the Realtime Crowd Insights app continuously analyzes key frames from the live video stream from the kiosk camera. It then uses Cognitive Services to assign each visitor an anonymous unique identifier, so it can count the number of unique visitors to the kiosk. The software uses facial recognition to determine the age and gender of visitors, as well as recognize displayed emotion. Figure 3 shows the solution in action.
.png "Realtime Crowd Insights Solution with Cognitive Services")
Figure 3 Realtime Crowd Insights Solution with Cognitive Services
Once you’ve downloaded the sample code, you’ll learn how to extract key frames from the camera at one frame per second (fps), sending each frame to the Cognitive Services Computer Vision APIs to identify the age, gender and emotion of people present in each frame in the file RealTimeDemo.xaml.cs (bit.ly/2hZZDFF). Lines 160 and 161 of this file show how you can invoke the methods for emotion and face identification (specifically, DetectEmotionAsync and DetectFacesAsync). Various face attributes, such as bounding box, age and gender are returned by Cognitive Services.
As you explore the code, you’ll find the image analysis methods in the file ImageAnalyzer.cs. ImageAnalyzer further leverages helper classes (like FaceServiceHelper and EmotionServiceHelper) to leverage Cognitive Services for face and emotion detection. In FaceServiceHelper, you’ll see how you can use the FaceServiceClient provided by Cognitive Services to instantiate the faceClient that’s used for face detection.
private static void InitializeFaceServiceClient() {
faceClient = ApiKeyRegion != null ?
new FaceServiceClient(ApiKey,
string.Format("https://{0}.api.cognitive.microsoft.com/face/v1.0",
ApiKeyRegion)) : new FaceServiceClient(ApiKey);
}
You’ll find that this is a common pattern used when working with Cognitive Services. Each of the Cognitive Services provides REST APIs, as well as an SDK that makes it easy to use Cognitive Services from any application. The SDK lets you quickly get started developing AI applications using C#, Java, JavaScript, PHP, Python and Ruby. You can also use Curl to directly access the Cognitive Services APIs. Learn more about the APIs at aka.ms/msdn/cognitiveservices/restapi and the Cognitive Services SDK at aka.ms/msdn/cognitiveservices/sdk.
The API key for Cognitive Services can be obtained from the Azure portal after the Cognitive Service you need has been created. When you create a Cognitive Service using the Azure portal, you need to specify the API type that you require—for example, Computer Vision. When developing applications, use the API Key to instantiate each service client.
Figure 4 shows how you can create a Computer Vision Cognitive Service using the Azure portal, and then obtain the API key.
.png "Creating a Cognitive Service and Obtaining the API Key Using the Azure Portal")
Figure 4 Creating a Cognitive Service and Obtaining the API Key Using the Azure Portal
Inside Azure Machine Learning
Azure Machine Learning services is an integrated, end-to-end data science and advanced analytics solution for professional data scientists to prepare data, develop experiments and deploy models at cloud scale. The package enables data science teams to have an environment that enables them to be productive, and amplifies the data science work that they do each day.
When developing custom machine learning and deep learning models, there are lots of choices in the toolkits. Azure Machine Learning lets you use the toolkits you’re familiar with—Cognitive Toolkit (CNTK), Tensorflow, Caffe and more. In addition, Azure Machine Learning can deploy, manage and monitor your models at scale. Whether deploying to the cloud, to a data lake, to a database like SQL Server 2017 or to the intelligent edge, Azure Machine Learning provides outstanding flexibility for model deployment.
The main components of Azure Machine Learning are:
- Azure Machine Learning Workbench: Consists of a desktop application and command-line tools that let you manage the entire data science lifecycle, from data ingestion and preparation to model development and deployment. Machine Learning Workbench is available on both Windows and macOS. Figure 5 shows the Machine Learning Workbench UI.
- Experimentation Service: Runs training models across different machine learning environments, ranging from a local machine to a Docker container on a remote virtual machine (VM) to a scaled-out Spark cluster on Azure in the cloud. Experimentation Service integrates with Machine Learning Workbench and supports Git integration, access control and sharing among co-workers.
- Model Management Service: Enables management and deployment of machine learning workflows and models. The Model Management Service keeps track of different model versions, and can package and deploy machine learning models as REST APIs served from a Docker container.
.png "Azure Machine Learning Workbench")
Figure 5 Azure Machine Learning Workbench
Once you’ve created the Azure Machine Learning Experimentation and Model Management Service, you can download the Machine Learning Workbench to help with your data preparation tasks, as well as manage all your machine learning projects.
Building Bots
Companies have started using intelligent bots to enable users to interact with their services. Bots provide tremendous value in a host of scenarios, including customer support engagements, answering product-related questions, providing tourists with navigation instructions or responding to HR-related questions. We see bots being used in exciting new ways and in different industries, including retail, health care, manufacturing, telecommunications and the public sector.
For example, the AzureBot lets you manage your Azure services (for instance, starting or stopping Azure VMs) using a bot that’s available on Skype, Microsoft Teams and other platforms. The Summarize bot uses Bing to present the main points from any Web page. Figure 6 shows some of the top bots currently being used in Skype (bing.com/search?q=top+bots).
.png "Top Bots Being Used in Skype")
Figure 6 Top Bots Being Used in Skype
You can build an intelligent bot in minutes using the Azure Bot Service, then deploy it to reach customers on multiple channels, like Skype, Messenger, Microsoft Teams or via a Web site.
To jumpstart development of your first bot, leverage the QnA Maker, a service that makes it easy to create a question and answer bot from an existing frequently asked questions (FAQ) page. Underneath the hood, QnA maker uses state-of-the-art machine learning algorithms and natural language processing (NLP) to distill the information found on FAQ pages to create question-and-answer pairs, as depicted in Figure 7. Visit qnamaker.ai to find out more.
.png "QnA Maker Architecture")
Figure 7 QnA Maker Architecture
Let’s look at the architecture for building an intelligent bot that’s infused with a range of AI capabilities. These include Cognitive Services like OCR, Text Analytics, Computer Vision and the Language Understanding Intelligent Service (LUIS), as well as program synthesis capabilities from the PROSE SDK (microsoft.github.io/prose) that enables developers to add intelligence to bots. These services are used to power the Stack Overflow bot, which enables developers to get answers to questions within their development environment. You can see the Stack Overflow bot in action at bit.ly/2hqnEst, and access the sample code at bit.ly/2zIDbZ9.
The Stack Overflow bot leverages Azure Functions to power its dialog analysis. An intelligent Azure Function orchestrates the components used to analyze a user-uploaded screenshot, calling on OCR to extract text in the image and Text Analytics to identify key phrases. Figure 8 shows the Stack Overflow Bot architecture.
.png "Stack Overflow Bot Architecture")
Figure 8 Stack Overflow Bot Architecture
Program Synthesis by Example (PROSE) enables the bot to generate code by learning from examples provided by the user. This lets developers quickly use generated code to perform transformations of input and output strings. For example, the developer provides an input string like “Jane Doe” and specifies the output string as “Doe, J.”. PROSE learns from this and generates the code that then performs the transformation.
In addition, data scientists can build custom deep learning models using Azure Machine Learning to automatically predict which Stack Overflow tags will be most relevant to the questions asked. This enables the Stack Overflow bot to expand the keywords used to perform keyword searches with Bing Search, helping developers find the answer they need faster.
Wrapping Up
In this article, you learned about the exciting capabilities offered by the Microsoft AI platform. From Cognitive Services that enable you to jumpstart using AI for building intelligent applications, to customizing state-of-the-state computer vision deep learning models, to building deep learning models of your own with Azure Machine Learning, the Microsoft AI platform equips developers with the tools they need. The AI platform is open and flexible, and empowers developers to choose the technology and deep learning framework best suited for their scenarios and skills.
This is the beginning of an intelligent revolution, and the Microsoft AI platform empowers you with exciting enterprise-ready services, infrastructure and tools to build intelligent, innovative applications. Get started today with microsoft.com/ai!
Resources
You can learn more about what’s new with Azure Machine Learning at bit.ly/2zVdpUs. Also, there are lots of resources available to help you get started with Azure Machine Learning. Kick things off with the setup and installation guide at aka.ms/aml-blog-setup and a quick start sample from aka.ms/aml-blog-iris. As you explore Azure Machine Learning dive deeper into more in-depth tutorials. Check out a three-part Iris classification tutorial at bit.ly/2AwjSSN, as well as a detailed data wrangling tutorial at bit.ly/2yOrJxQ.
In addition, you can find sample code and detailed scenario walk-throughs on several very interesting use cases. These include:
- Aerial Image Classification: Distributed training and operationalization of a land-usage classification model (bit.ly/2jf7y5o).
- Document Collection Analysis: Developing a robust model for text analytics (bit.ly/2ypziXr).
- Predictive Maintenance: Building an end-to-end predictive maintenance solution using PySpark (bit.ly/2zyBW0N).
- Energy Demand Time Series Forecasting: Forecasting energy demands to predict future loads on an electrical grid (bit.ly/2hps9mL).
The possibilities of using AI to solve exciting real-world problems are endless! I can’t wait to see how you use the Microsoft AI platform to create the next breakthrough intelligent solution.
Joseph Sirosh is the corporate vice president of the Cloud AI Platform at Microsoft, where he leads the company’s enterprise AI strategy and products such as Azure Machine Learning, Azure Cognitive Services, Azure Search and the Bot Framework. He’s passionate about machine learning and its applications and has been active in the field since 1990. Sirosh holds a doctorate in computer science from the University of Texas at Austin and a BTech in computer science and engineering from the Indian Institute of Technology Chennai.
Wee Hyong Tok is a principal data science manager of the Cloud AI Platform at Microsoft, where he leads AI Prototyping and Innovation. He has advised many Fortune 500 companies on data platform architectures, and using AI for their strategic initiatives. Wee Hyong holds a doctorate in computer science from the National University of Singapore.
Thanks to the following Microsoft technical expert for reviewing this article: Anand Raman, chief of staff of the Cloud AI Platform at Microsoft