Observed people tracking and matched faces

Article
03/22/2024

Important

Due to the Azure Media Services retirement announcement, Azure AI Video Indexer announces Azure AI Video Indexer features adjustments. See Changes related to Azure Media Service (AMS) retirement to understand what this means for your Azure AI Video Indexer account. See the Preparing for AMS retirement: VI update and migration guide.

Important

Face identification, customization and celebrity recognition features access is limited based on eligibility and usage criteria in order to support our Responsible AI principles. Face identification, customization and celebrity recognition features are only available to Microsoft managed customers and partners. Use the Face Recognition intake form to apply for access.

Observed people tracking and matched faces are Azure AI Video Indexer AI features that automatically detect and match people in media files. Observed people tracking and matched faces can be set to display insights on people, their clothing, and the exact timeframe of their appearance.

The resulting insights are displayed in a categorized list in the Insights tab, the tab includes a thumbnail of each person and their ID. Clicking the thumbnail of a person displays the matched person (the corresponding face in the People insight). Insights are also generated in a categorized list in a JSON file that includes the thumbnail ID of the person, the percentage of time appearing in the file, Wiki link (if they're a celebrity) and confidence level.

Prerequisites

Review transparency note overview

General principles

This article discusses observed people tracking and matched faces and the key considerations for making use of this technology responsibly. There are many things you need to consider when deciding how to use and implement an AI-powered feature:

Will this feature perform well in my scenario? Before deploying observed people tracking and matched faces into your scenario, test how it performs using real-life data and make sure it can deliver the accuracy you need.
Are we equipped to identify and respond to errors? AI-powered products and features will not be 100% accurate, so consider how you'll identify and respond to any errors that may occur.

View the insight

When uploading the media file, go to Video + Audio Indexing and select Advanced.

To display observed people tracking and matched faces insight on the website, do the following:

After the file has been indexed, go to Insights and then scroll to observed people.

To see the insights in a JSON file, do the following:

Click Download and then Insights (JSON).

Copy the observedPeople text and paste it into your JSON viewer.

The following section shows observed people and clothing. For the person with id 4 ("id": 4) there's also a matching face.

 "observedPeople": [
   {
     "id": 1,
     "thumbnailId": "4addcebf-6c51-42cd-b8e0-aedefc9d8f6b",
     "clothing": [
       {
         "id": 1,
         "type": "sleeve",
         "properties": {
           "length": "long"
         }
       },
       {
         "id": 2,
         "type": "pants",
         "properties": {
           "length": "long"
         }
       }
     ],
     "instances": [
       {
         "adjustedStart": "0:00:00.0667333",
         "adjustedEnd": "0:00:12.012",
         "start": "0:00:00.0667333",
         "end": "0:00:12.012"
       }
     ]
   },
   {
     "id": 2,
     "thumbnailId": "858903a7-254a-438e-92fd-69f8bdb2ac88",
     "clothing": [
       {
           "id": 1,
           "type": "sleeve",
           "properties": {
               "length": "short"
           }
       }
     ],
     "instances": [
       {
           "adjustedStart": "0:00:23.2565666",
           "adjustedEnd": "0:00:25.4921333",
           "start": "0:00:23.2565666",
           "end": "0:00:25.4921333"
       },
       {
           "adjustedStart": "0:00:25.8925333",
           "adjustedEnd": "0:00:25.9926333",
           "start": "0:00:25.8925333",
           "end": "0:00:25.9926333"
       },
       {
           "adjustedStart": "0:00:26.3930333",
           "adjustedEnd": "0:00:28.5618666",
           "start": "0:00:26.3930333",
           "end": "0:00:28.5618666"
       }
     ]
   },
   {
     "id": 3,
     "thumbnailId": "1406252d-e7f5-43dc-852d-853f652b39b6",
     "clothing": [
       {
         "id": 1,
         "type": "sleeve",
         "properties": {
             "length": "short"
         }
       },
       {
         "id": 2,
         "type": "pants",
         "properties": {
             "length": "long"
         }
       },
       {
         "id": 3,
         "type": "skirtAndDress"
       }
     ],
     "instances": [
       {
         "adjustedStart": "0:00:31.9652666",
         "adjustedEnd": "0:00:34.4010333",
         "start": "0:00:31.9652666",
         "end": "0:00:34.4010333"
       }
     ]
   },
   {
     "id": 4,
     "thumbnailId": "d09ad62e-e0a4-42e5-8ca9-9a640c686596",
     "clothing": [
       {
         "id": 1,
         "type": "sleeve",
         "properties": {
             "length": "short"
         }
       },
       {
         "id": 2,
         "type": "pants",
         "properties": {
             "length": "short"
         }
       }
     ],
     "matchingFace": {
       "id": 1310,
       "confidence": 0.3819
     },
     "instances": [
       {
           "adjustedStart": "0:00:34.8681666",
           "adjustedEnd": "0:00:36.0026333",
           "start": "0:00:34.8681666",
           "end": "0:00:36.0026333"
       },
       {
           "adjustedStart": "0:00:36.6699666",
           "adjustedEnd": "0:00:36.7367",
           "start": "0:00:36.6699666",
           "end": "0:00:36.7367"
       },
       {
           "adjustedStart": "0:00:37.2038333",
           "adjustedEnd": "0:00:39.6729666",
           "start": "0:00:37.2038333",
           "end": "0:00:39.6729666"
       }
     ]
   }
 ]

To download the JSON file via the API, use the Azure AI Video Indexer developer portal.