Edit

Vision classifier model with Azure Custom Vision Cognitive Service

Azure
GitHub

Solution ideas

This article is a solution idea. If you'd like us to expand the content with more information, such as potential use cases, alternative services, implementation considerations, or pricing guidance, let us know by providing GitHub feedback.

This architecture uses Custom Vision to classify images taken by a simulated drone. It provides a way to combine AI and the Internet of Things (IoT). Azure Custom Vision can also be used for object detection purpose.

Architecture

Diagram of the Search and Rescue Lab architecture to create an image classifier model.

Download a Visio file of this architecture.

Workflow

  1. Use AirSim's 3D-rendered environment to take images taken with the drone. Use the images as the training dataset.
  2. Import and tag the dataset in a Custom Vision project. The cognitive service trains and tests the model.
  3. Export the model into TensorFlow format so you can use it locally.
  4. The model can also be deployed to a container or to mobile devices.

Components

Microsoft AirSim Drone simulator

Microsoft AirSim Drone simulator is built on the Unreal Engine. The simulator is open-source, cross-platform, and developed to help AI research. In this architecture, it creates the dataset of images used to train the model.

Azure Custom Vision

Azure Custom Vision is part of Azure Cognitive Services. In this architecture, it creates an image classifier model.

TensorFlow

TensorFlow is an open-source platform for machine learning (ML). It's a tool that helps you develop and train ML models. When you export your model to TensorFlow format, you'll have a protocol buffer file with the Custom Vision model that you can use locally in your script.

Scenario details

Azure Cognitive Services offers many possibilities for Artificial Intelligence (AI) solutions. One of them is Azure Custom Vision, which allows you to build, deploy, and improve your image classifiers. This architecture uses Custom Vision to classify images taken by a simulated drone. It provides a way to combine AI and the Internet of Things (IoT). Azure Custom Vision can also be used for object detection purpose.

Potential use case

This solution is ideal for the rescue, simulation, robotics, aircraft, aerospace, and aviation industries.

Microsoft Search and Rescue Lab suggests a hypothetical use case for Custom Vision. In the lab, you fly a Microsoft AirSim simulated drone around in a 3D-rendered environment. You use the simulated drone to capture synthetic images of the animals in that environment. After creating a dataset of images, you use the dataset to train a Custom Vision classifier model. To train the model, you tag the images with the names of the animals. When you fly the drone again, take new images of the animals. This solution identifies the name of the animal in each new image.

In a practical application of the lab, an actual drone replaces the Microsoft AirSim simulated drone. If a pet is lost, the owner provides images of the pet to the Custom Vision model trainer. Just like in the simulation, the images are used to train the model to recognize the pet. Then, the drone pilot searches an area where the lost pet might be. As it finds animals along the way, the drone's camera can capture images and determine if the animal is the lost pet.

Deploy this scenario

To deploy this reference architecture, follow the steps described in the GitHub repo of the Search and Rescue Lab.

Contributors

This article is maintained by Microsoft. It was originally written by the following contributors.

Principal authors:

Next steps

Read other Azure Architecture Center articles: