Use cases for Azure Face service

Article
09/28/2022

What is a Transparency Note?

An AI system includes not only the technology, but also the people who will use it, the people who will be affected by it, and the environment in which it is deployed. Creating a system that is fit for its intended purpose requires an understanding of how the technology works, its capabilities and limitations, and how to achieve the best performance.

Microsoft's Transparency Notes are intended to help you understand how our AI technology works, the choices system owners can make that influence system performance and behavior, and the importance of thinking about the whole system, including the technology, the people, and the environment. You can use Transparency Notes when developing or deploying your own system, or share them with the people who will use or be affected by your system.

Microsoft's Transparency Notes are part of a broader effort at Microsoft to put our AI principles into practice. To find out more, see Microsoft AI principles.

This Transparency Note is part of our effort at Microsoft to implement our Facial Recognition Principles, which set out how we approach the development and deployment of facial recognition technology. We encourage you to use the principles to guide your development efforts as you use this technology.

The basics of Azure Face service

Accessible through Azure Cognitive Services, Face detects, recognizes, and analyzes human faces in images using pre-trained machine learning models that have been developed by Microsoft. Developers can integrate Face functions into their systems without creating their own models.

When used responsibly, facial recognition is an important and useful technology that can improve efficiency, security, and customer experiences. Face is a building block for creating a facial recognition system.

Key terms

Facial verification: A "one-to-one" matching of a face in an image to a single face from a secure repository or photo to verify they are the same individual, using unmanipulated images. An example is a banking app that enables users to open a credit account remotely by taking a selfie and taking a picture of a photo ID to verify their identity.
Facial identification: "One-to-many" matching of a face in an unmanipulated image to a set of faces in a secure repository. An example is a touchless access control system in a building that replaces or augments physical cards and badges in which a smart camera captures the face of one person entering a secured door and attempts to find a match from a set of images of faces of individuals who are approved to access the building.
Facial recognition: A term that captures both Face Identification and Face Verification scenarios.
Facial detection: Finds human faces in an image and returns bounding boxes indicating their locations. Face detection models alone do not find individually identifying features, only a bounding box marking the entire face.
Facial redaction: Redaction enables modification of images or videos in order to blur or block faces of individuals.
Bounding box: A box drawn around the location of a face in the photo in response to Face Detection calls.
Template: A face template is a unique set of numbers generated from an image that represents the distinctive features of a face. The images themselves – whether enrollment or probe images (see below) – are not stored by Face, and the original images cannot be reconstructed based on a template.
Enrollment: Enrollment is the process of enrolling photos of individuals for template creation so they can be recognized. High-quality photos yield high-quality enrollment templates.
Person ID: When a person is enrolled to a verification system used for authentication, their template is also associated with a primary, randomly-generated identifier[^1] called the Person ID that will be used to determine which template to compare with the probe image (see below).
Probe image: A probe image is an image submitted to a facial recognition system to be compared to enrolled individuals. Probe images are also converted to probe templates to compare to enrolled templates. All images are immediately deleted once they are converted to templates.
Recognition confidence score: When a probe image is queried to the Face Verification or Identification API, the API will return a recognition confidence score of whether two faces match in the range of [0, 1], such as 0.6. This is not the same as the percent likelihood that two faces match (i.e., a 0.9 recognition confidence score does not mean there is a 90% chance that the two faces match).
Recognition confidence threshold: The minimum confidence score required to determine whether two faces belong to the same person based on the recognition confidence score. For example, if the confidence threshold is 0.5 and the recognition confidence score returned from a probe image query is 0.6, then the two faces are considered a match.
Candidate list: For Face Identification scenarios, a candidate list is the list of faces with scores above the recognition confidence threshold. [^1]: Face does not store primary identifiers, such as customer IDs, alongside facial templates. Instead, Microsoft associates stored facial templates with random GUIDs or globally unique identifiers. System developers can associate the GUID generated by Microsoft with an individual’s primary identifier to support verification of that individual.

Face Functions

Face Detection (“Detection” / “Detect”) answers the question, “Are there one or more human faces in this image?” Detection finds human faces in an image and returns bounding boxes indicating their locations. Face detection models alone do not find individually identifying features, only a bounding box. All other functions are dependent on Detection: before Face can identify or verify a person (see below), it must know the locations of the faces to be recognized.

Face Detection Attributes: The Detect API can also optionally be used to analyze attributes about each face using additional AI models, such as pose and facial landmarks like eye or nose position. The attribute functionality is completely separate from the verification and identification functionality of Face. The full list of attributes is described in the service concepts. The values returned by the API for each attribute are predictions of the perceived attributes and are best used to make aggregated approximations of attribute representation rather than an individual assessment.

Face Verification ("Verification" / "Verify") builds on Detect and addresses the question, "Are these two images of the same person?" Verification is also called "one-to-one" matching because the probe image is compared to only one enrolled template. Verification can be used in identity verification or access control scenarios to verify a picture matches a previously captured image (such as from a photo from a government issued ID card). For more information, see the Verify API reference documentation.

Face Identification ("Identification" / "Identify") also starts with Detect and answers the question, "Can this detected face be matched to any enrolled face in a database?" For this reason, identification is also called "one-to-many" matching. Candidate matches are returned based on how closely the probe template of the detected face matches each of the enrolled templates. For more information about facial identification, see the Identify API reference documentation.

Face Find Similar ("Find Similar") also builds on Detect and searches for similar looking faces from all enrollment templates. For more information, see the Find Similar API reference documentation.

Face Group ("Group") also builds on Detect and creates smaller groups of faces that look similar to each other from all enrollment templates. For more information, see the Group API reference documentation.

For more information on functions of Azure Face service, see the Face documentation.

Limited Access to Azure Face service

Azure Face service is a Limited Access service, and registration is required for access to some features. To learn more about Microsoft’s Limited Access policy visit aka.ms/limitedaccesscogservices. Certain features are only available to Microsoft managed customers and partners, and only for certain use cases selected at the time of registration. Note that facial detection and facial redaction use cases do not require registration:

Facial detection: Detect the locations and attributes of faces for accessibility (such as Seeing AI) or modern productivity.

Facial redaction: Redact or blur detected faces of people recorded in a video to protect their privacy.

Limited Access commercial use cases

The following use cases are approved for commercial contexts: Facial verification for identity verification to grant access to digital or physical services or spaces. Such verification may be used for opening a new account, verifying a worker, or authenticating to participate in an online assessment. Identity verification can be done once during onboarding, and repeatedly as someone accesses a digital or physical service or space.

Facial identification for touchless access control to enable an enhanced experience using facial recognition, as opposed to methods like cards and tickets. This can reduce hygiene and security risks from card/ticket sharing/handling, loss, or theft. Facial recognition can assist the check-in process for accessing sites and buildings, such as airports, stadiums, offices, and hospitals.

Facial identification for personalization to enable ambient environment personalization with consent-based facial recognition that enriches experiences on shared devices. For example, hot desk screens and kiosks in the workplace and home can recognize you as you approach to provide directions to your destination or jumpstart hands-free interaction with smart meetings devices.

Facial identification to detect duplicate or blocked users to control or prevent unauthorized access to digital or physical services or spaces. For example, such identification may be used at account creation or sign-in or at access to a work site.

Limited Access public sector use cases

The following use cases are approved for the public sector:

Facial verification for identity verification to grant access to digital or physical services or spaces. Such verification may be used for opening a new account, verifying a worker, or authenticating to participate in an online assessment. Identity verification can be done once during onboarding, and repeatedly as someone accesses a digital or physical service or space.

Facial identification for touchless access control to enable an enhanced experience using facial recognition, as opposed to methods like cards and tickets. This can help reduce hygiene and security risks from card/ticket sharing/handling, loss, or theft. Facial recognition can assist the check-in process for accessing sites and buildings, such as airports, stadiums, offices, and hospitals.

Facial identification for preservation and enrichment of public media archives to identify individuals in public media or entertainment video archives for the purposes of preserving and enriching public media only. Examples of public media enrichment include identifying historical figures in video archives or generating descriptive metadata.

Facial identification to assist law enforcement or court officials in prosecution or defense of a criminal suspect who has already been apprehended, to the extent specifically authorized by a duly empowered government authority in a jurisdiction that maintains a fair and independent judiciary, OR to assist officials of duly empowered international organizations in the prosecution of abuses of international criminal law, international human rights law, or international humanitarian law.

Facial identification to respond to an emergency involving imminent danger or risk of death or serious physical injury to an individual.

Facial identification for purposes of providing humanitarian aid or identifying missing persons, deceased persons, or victims of crimes.

Considerations when using Azure Face Service

The use of Azure Face by or for state or local police in the U.S. is prohibited by Microsoft policy.

The use of real-time facial recognition technology on mobile cameras used by law enforcement to attempt to identify individuals in uncontrolled, "in the wild" environments is prohibited by Microsoft policy. This includes where police officers on patrol use body-worn or dash-mounted cameras using facial recognition technology to attempt to identify individuals present in a database of suspects or prior inmates. This policy applies globally.

Avoid use of facial recognition or detection technology to attempt to infer emotional states, gender identity, or age. Microsoft has retired general-purpose facial detection capabilities that were used to classify emotion, gender, age, smile, hair, facial hair, and makeup. General-purpose use of these capabilities poses a risk of misuse that could subject people to stereotyping, discrimination, or unfair denial of services. These capabilities will be carefully restricted to select accessibility scenarios such as those provided by Seeing AI.

Avoid use for ongoing surveillance of real-time or near real-time identification or persistent tracking of an individual. Ongoing surveillance is defined as the tracking of movements of an identified individual on a persistent basis. Persistent tracking is defined as the tracking of movements of an individual on a persistent basis without identification or verification of that individual. Face was not designed for ongoing surveillance or persistent tracking of an individual and does not work on large-scale real-time camera streams. In accordance with our Six Principles for Developing and Deploying Facial Recognition Technology, the use of facial recognition technology for the ongoing surveillance of individuals by law enforcement should be prohibited except in narrow circumstances and only with adequate protections for individual civil liberties and human rights.

Avoid use for task-monitoring systems that can interfere with privacy. Face's probabilistic AI models were not designed to monitor individual patterns to infer intimate personal information, such as an individual's sexual or political orientation.

Avoid use in protected spaces. Protect individuals' privacy by evaluating camera locations and positions, adjusting angles and regions of interest so they do not overlook protected areas such as restrooms.

Avoid use in environments where enrollment in identification or verification is not optional. Protect individuals' autonomy by not planning enrollment in situations where there's pressure to consent.

Avoid use where a human in the loop or secondary verification method is not available. Failsafe mechanisms, e.g., a secondary method being available to the end user if the technology fails, helps to prevent denial of essential services or other harms due to false negatives.

Carefully consider use in schools or facilities for older adults. Face has not been heavily tested with data containing minors under the age of 18 or adults over age 65. We recommend that customers thoroughly evaluate error rates for any scenario in environments where there is a predominance of these age groups.

Carefully consider use for healthcare-related decisions. Face provides probabilistic results like face detections, attributes, and recognitions. The data may not be suitable for making healthcare-related decisions.

Carefully consider use in public spaces. Evaluate camera locations and positions, adjusting angles and regions of interest to minimize collection from public spaces. Lighting and weather in public spaces such as streets and parks will significantly impact the performance of the spatial analysis system, and it is extremely difficult to provide effective disclosure in public spaces.