Use the custom categories (rapid) API (preview)

Article
29/01/2025

The custom categories (rapid) API lets you quickly respond to emerging harmful content incidents. You can define an incident with a few examples in a specific topic, and the service will start detecting similar content.

Follow these steps to define an incident with a few examples of text content and then analyze new text content to see if it matches the incident.

Important

This new feature is only available in select Azure regions. See Region availability.

Caution

The sample data in this guide might contain offensive content. User discretion is advised.

Prerequisites

An Azure subscription - Create one for free
Once you have your Azure subscription, create a Content Safety resource in the Azure portal to get your key and endpoint. Enter a unique name for your resource, select your subscription, and select a resource group, supported region (see Region availability), and supported pricing tier. Then select Create.
- The resource takes a few minutes to deploy. After it finishes, Select go to resource. In the left pane, under Resource Management, select Subscription Key and Endpoint. The endpoint and either of the keys are used to call APIs.
Also create a blob storage container if you want to upload your images there. You can alternatively encode your images as Base64 strings and use them directly in the API calls.
One of the following installed:
- cURL for REST API calls.
- Python 3.x installed

Test the text custom categories (rapid) API

Use the sample code in this section to create a text incident, add samples to the incident, deploy the incident, and then detect text incidents.

In the commands below, replace <your_api_key>, <your_endpoint>, and other necessary parameters with your own values.

The following command creates an incident with a name and definition.

Bash

curl --location --request PATCH 'https://<endpoint>/contentsafety/text/incidents/<text-incident-name>?api-version=2024-02-15-preview' \
--header 'Ocp-Apim-Subscription-Key: <your-content-safety-key>' \
--header 'Content-Type: application/json' \
--data '{ \"incidentName\": \"<test-incident>\", \"incidentDefinition\": \"<string>\"}'

First, you need to install the required Python library:

Bash

pip install requests

Then, define the necessary variables with your own Azure resource details:

Python

import requests

API_KEY = '<your_api_key>'
ENDPOINT = '<your_endpoint>'

headers = {
    'Ocp-Apim-Subscription-Key': API_KEY,
    'Content-Type': 'application/json'
}

The following command creates an incident with a name and definition.

Python

import requests
import json

url = "https://<endpoint>/contentsafety/text/incidents/<text-incident-name>?api-version=2024-02-15-preview"

payload = json.dumps({
  "incidentName": "<text-incident-name>",
  "incidentDefinition": "string"
})
headers = {
  'Ocp-Apim-Subscription-Key': '<your-content-safety-key>',
  'Content-Type': 'application/json'
}

response = requests.request("PATCH", url, headers=headers, data=payload)

print(response.text)

Add samples to the incident

Use the following command to add text examples to the incident.

cURL
Python

Bash

curl --location 'https://<endpoint>/contentsafety/text/incidents/<text-incident-name>:addIncidentSamples?api-version=2024-02-15-preview' \
--header 'Ocp-Apim-Subscription-Key: <your-content-safety-key>' \
--header 'Content-Type: application/json' \
--data-raw '{
  \"IncidentSamples\": [
    { \"text\":    \"<text-example-1>\"},
    { \"text\":    \"<text-example-2>\"},
    ...
  ]
}'

Python

import requests
import json

url = "https://<endpoint>/contentsafety/text/incidents/<text-incident-name>:addIncidentSamples?api-version=2024-02-15-preview"

payload = json.dumps({
  "IncidentSamples": [
    {
      "text": "<text-example-1>"
    },
    {
      "text": "<text-example-1>"
    },
    ...
  ]
})
headers = {
  'Ocp-Apim-Subscription-Key': '<your-content-safety-key>',
  'Content-Type': 'application/json'
}

response = requests.request("POST", url, headers=headers, data=payload)

print(response.text)

Deploy the incident

Use the following command to deploy the incident, making it available for the analysis of new content.

cURL
Python

Bash

curl --location 'https://<endpoint>/contentsafety/text/incidents/<text-incident-name>:deploy?api-version=2024-02-15-preview' \
--header 'Ocp-Apim-Subscription-Key: <your-content-safety-key>' \
--header 'Content-Type: application/json'

Python

import requests
import json

url = "https://<endpoint>/contentsafety/text/incidents/<text-incident-name>:deploy?api-version=2024-02-15-preview"

payload = {}
headers = {
  'Ocp-Apim-Subscription-Key': '<your-content-safety-key>',
  'Content-Type': 'application/json'
}

response = requests.request("POST", url, headers=headers, data=payload)

print(response.text)

Detect text incidents

Run the following command to analyze sample text content for the incident you just deployed.

cURL
Python

Bash

curl --location 'https://<endpoint>/contentsafety/text:detectIncidents?api-version=2024-02-15-preview' \
--header 'Ocp-Apim-Subscription-Key: <your-content-safety-key>' \
--header 'Content-Type: application/json' \
--data '{
  \"text\":  \"<test-text>\",
  \"incidentNames\": [
    \"<text-incident-name>\"
  ]
}'

Python

import requests
import json

url = "https://<endpoint>/contentsafety/text:detectIncidents?api-version=2024-02-15-preview"

payload = json.dumps({
  "text": "<test-text>",
  "incidentNames": [
    "<text-incident-name>"
  ]
})
headers = {
  'Ocp-Apim-Subscription-Key': '<your-content-safety-key>',
  'Content-Type': 'application/json'
}

response = requests.request("POST", url, headers=headers, data=payload)

print(response.text)

Test the image custom categories (rapid) API

Use the sample code in this section to create an image incident, add samples to the incident, deploy the incident, and then detect image incidents.

Create an incident

cURL
Python

In the commands below, replace <your_api_key>, <your_endpoint>, and other necessary parameters with your own values.

The following command creates an image incident:

Bash

curl --location --request PATCH 'https://<endpoint>/contentsafety/image/incidents/<image-incident-name>?api-version=2024-02-15-preview' \
--header 'Ocp-Apim-Subscription-Key: <your-content-safety-key>' \
--header 'Content-Type: application/json' \
--data '{
  \"incidentName\": \"<image-incident-name>\"
}'

Make sure you've installed required Python libraries:

Bash

pip install requests

Define the necessary variables with your own Azure resource details:

Python

import requests

API_KEY = '<your_api_key>'
ENDPOINT = '<your_endpoint>'

headers = {
    'Ocp-Apim-Subscription-Key': API_KEY,
    'Content-Type': 'application/json'
}

The following command creates an image incident:

Python

import requests
import json

url = "https://<endpoint>/contentsafety/image/incidents/<image-incident-name>?api-version=2024-02-15-preview"

payload = json.dumps({
  "incidentName": "<image-incident-name>"
})
headers = {
  'Ocp-Apim-Subscription-Key': '<your-content-safety-key>',
  'Content-Type': 'application/json'
}

response = requests.request("PATCH", url, headers=headers, data=payload)

print(response.text)

Add samples to the incident

Use the following command to add examples images to your incident. The image samples can be URLs pointing to images in an Azure blob storage container, or they can be Base64 strings.

cURL
Python

Bash

curl --location 'https://<endpoint>/contentsafety/image/incidents/<image-incident-name>:addIncidentSamples?api-version=2024-02-15-preview' \
--header 'Ocp-Apim-Subscription-Key: <your-content-safety-key>' \
--header 'Content-Type: application/json' \
--data '{
  \"IncidentSamples\": [
    {
      \"image\": {
        \"content\": \"<base64-data>\",
        \"bloburl\": \"<your-blob-storage-url>.png\"
      }
    }
  ]
}'

Python

import requests
import json

url = "https://<endpoint>/contentsafety/image/incidents/<image-incident-name>:addIncidentSamples?api-version=2024-02-15-preview"

payload = json.dumps({
  "IncidentSamples": [
    {
      "image": {
        "content": "<base64-data>",
        "bloburl": "<your-blob-storage-url>/image.png"
      }
    }
  ]
})
headers = {
  'Ocp-Apim-Subscription-Key': '<your-content-safety-key>',
  'Content-Type': 'application/json'
}

response = requests.request("POST", url, headers=headers, data=payload)

print(response.text)

Deploy the incident

Use the following command to deploy the incident, making it available for the analysis of new content.

cURL
Python

Bash

curl --location 'https://<endpoint>/contentsafety/image/incidents/<image-incident-name>:deploy?api-version=2024-02-15-preview' \
--header 'Ocp-Apim-Subscription-Key: <your-content-safety-key>' \
--header 'Content-Type: application/json'

Python

import requests
import json

url = "https://<endpoint>/contentsafety/image/incidents/<image-incident-name>:deploy?api-version=2024-02-15-preview"

payload = {}
headers = {
  'Ocp-Apim-Subscription-Key': '<your-content-safety-key>',
  'Content-Type': 'application/json'
}

response = requests.request("POST", url, headers=headers, data=payload)

print(response.text)

Detect image incidents

Use the following command to upload a sample image and test it against the incident you deployed. You can either use a URL pointing to the image in an Azure blob storage container, or you can add the image data as a Base64 string.

cURL
Python

Bash

curl --location 'https://<endpoint>/contentsafety/image:detectIncidents?api-version=2024-02-15-preview' \
--header 'Ocp-Apim-Subscription-Key: <your-content-safety-key>' \
--header 'Content-Type: application/json' \
--data '{
    \"image\": {
      \"url\": \"<your-blob-storage-url>/image.png\",
      "content": "<base64-data>"
    },
    \"incidentNames\": [
      \"<image-incident-name>\"
    ]
  }
}'

Python

import requests
import json

url = "https://<endpoint>/contentsafety/image:detectIncidents?api-version=2024-02-15-preview"

payload = json.dumps({
  "image": {
    "url": "<your-blob-storage-url>/image.png",
    "content": "<base64-data>"
  },
  "incidentNames": [
    "<image-incident-name>"
  ]
})
headers = {
  'Ocp-Apim-Subscription-Key': '<your-content-safety-key>',
  'Content-Type': 'application/json'
}

response = requests.request("POST", url, headers=headers, data=payload)

print(response.text)

Other incident operations

The following operations are useful for managing incidents and incident samples.

Text incidents API

List all incidents

cURL
Python

Bash

curl --location GET 'https://<endpoint>/contentsafety/text/incidents?api-version=2024-02-15-preview' \
--header 'Ocp-Apim-Subscription-Key: <your-content-safety-key>'

Python

import requests

url = "https://<endpoint>/contentsafety/text/incidents?api-version=2024-02-15-preview"

payload = {}
headers = {
  'Ocp-Apim-Subscription-Key': '<your-content-safety-key>'
}

response = requests.request("GET", url, headers=headers, data=payload)

print(response.text)

Get the incident details

cURL
Python

Bash

curl --location GET 'https://<endpoint>/contentsafety/text/incidents/<text-incident-name>?api-version=2024-02-15-preview' \
--header 'Ocp-Apim-Subscription-Key: <your-content-safety-key>'

Python

import requests

url = "https://<endpoint>/contentsafety/text/incidents/<text-incident-name>?api-version=2024-02-15-preview"

payload = {}
headers = {
  'Ocp-Apim-Subscription-Key': '<your-content-safety-key>'
}

response = requests.request("GET", url, headers=headers, data=payload)

print(response.text)

Delete the incident

cURL
Python

Bash

curl --location --request DELETE 'https://<endpoint>/contentsafety/text/incidents/<text-incident-name>?api-version=2024-02-15-preview' \
--header 'Ocp-Apim-Subscription-Key: <your-content-safety-key>'

Python

import requests

url = "https://<endpoint>/contentsafety/text/incidents/<text-incident-name>?api-version=2024-02-15-preview"

payload = {}
headers = {
  'Ocp-Apim-Subscription-Key': '<your-content-safety-key>'
}

response = requests.request("DELETE", url, headers=headers, data=payload)

print(response.text)

List all samples under an incident

This command retrieves the unique IDs of all the samples associated with a given incident object.

cURL
Python

Bash

curl --location GET 'https://<endpoint>/contentsafety/text/incidents/<text-incident-name>/incidentsamples?api-version=2024-02-15-preview' \
--header 'Ocp-Apim-Subscription-Key: <your-content-safety-key>'

Python

import requests

url = "https://<endpoint>/contentsafety/text/incidents/<text-incident-name>/incidentsamples?api-version=2024-02-15-preview"

payload = {}
headers = {
  'Ocp-Apim-Subscription-Key': '<your-content-safety-key>'
}

response = requests.request("GET", url, headers=headers, data=payload)

print(response.text)

Get an incident sample's details

Use an incident sample ID to look up details about the sample.

cURL
Python

Bash

curl --location GET 'https://<endpoint>/contentsafety/text/incidents/<text-incident-name>/incidentsamples/<your-incident-sample-id>?api-version=2024-02-15-preview' \
--header 'Ocp-Apim-Subscription-Key: <your-content-safety-key>'

Python

import requests

url = "https://<endpoint>/contentsafety/text/incidents/<text-incident-name>/incidentsamples/<your-incident-sample-id>?api-version=2024-02-15-preview"

payload = {}
headers = {
  'Ocp-Apim-Subscription-Key': '<your-content-safety-key>'
}

response = requests.request("GET", url, headers=headers, data=payload)

print(response.text)

Delete an incident sample

Use an incident sample ID to retrieve and delete that sample.

cURL
Python

Bash

curl --location 'https://<endpoint>/contentsafety/text/incidents/<text-incident-name>:removeIncidentSamples?api-version=2024-02-15-preview' \
--header 'Ocp-Apim-Subscription-Key: <your-content-safety-key>' \
--header 'Content-Type: application/json' \
--data '{
  \"IncidentSampleIds\": [
    \"<your-incident-sample-id>\"
  ]
}'

Python

import requests
import json

url = "https://<endpoint>/contentsafety/text/incidents/<text-incident-name>:removeIncidentSamples?api-version=2024-02-15-preview"

payload = json.dumps({
  "IncidentSampleIds": [
    "<your-incident-sample-id>"
  ]
})
headers = {
  'Ocp-Apim-Subscription-Key': '<your-content-safety-key>',
  'Content-Type': 'application/json'
}

response = requests.request("POST", url, headers=headers, data=payload)

print(response.text)

Image incidents API

Get the incidents list

cURL
Python

Bash

curl --location GET 'https://<endpoint>/contentsafety/image/incidents?api-version=2024-02-15-preview' \
--header 'Ocp-Apim-Subscription-Key: <your-content-safety-key>'

Python

import requests

url = "https://<endpoint>/contentsafety/image/incidents?api-version=2024-02-15-preview"

payload = {}
headers = {
  'Ocp-Apim-Subscription-Key': '<your-content-safety-key>'
}

response = requests.request("GET", url, headers=headers, data=payload)

print(response.text)

Get the incident details

cURL
Python

Bash

curl --location GET 'https://<endpoint>/contentsafety/image/incidents/<image-incident-name>?api-version=2024-02-15-preview' \
--header 'Ocp-Apim-Subscription-Key: <your-content-safety-key>'

Python

import requests

url = "https://<endpoint>/contentsafety/image/incidents/<image-incident-name>?api-version=2024-02-15-preview"

payload = {}
headers = {
  'Ocp-Apim-Subscription-Key': '<your-content-safety-key>'
}

response = requests.request("GET", url, headers=headers, data=payload)

print(response.text)

Delete the incident

cURL
Python

Bash

curl --location --request DELETE 'https://<endpoint>/contentsafety/image/incidents/<image-incident-name>?api-version=2024-02-15-preview' \
--header 'Ocp-Apim-Subscription-Key: <your-content-safety-key>'

Python

import requests

url = "https://<endpoint>/contentsafety/image/incidents/<image-incident-name>?api-version=2024-02-15-preview"

payload = {}
headers = {
  'Ocp-Apim-Subscription-Key': '<your-content-safety-key>'
}

response = requests.request("DELETE", url, headers=headers, data=payload)

print(response.text)

List all samples under an incident

This command retrieves the unique IDs of all the samples associated with a given incident object.

cURL
Python

Bash

curl --location GET 'https://<endpoint>/contentsafety/image/incidents/<image-incident-name>/incidentsamples?api-version=2024-02-15-preview' \
--header 'Ocp-Apim-Subscription-Key: <your-content-safety-key>'

Python

import requests

url = "https://<endpoint>/contentsafety/image/incidents/<image-incident-name>/incidentsamples?api-version=2024-02-15-preview"

payload = {}
headers = {
  'Ocp-Apim-Subscription-Key': '<your-content-safety-key>'
}

response = requests.request("GET", url, headers=headers, data=payload)

print(response.text)

Get the incident sample details

Use an incident sample ID to look up details about the sample.

cURL
Python

Bash

curl --location GET 'https://<endpoint>/contentsafety/image/incidents/<image-incident-name>/incidentsamples/<your-incident-sample-id>?api-version=2024-02-15-preview' \
--header 'Ocp-Apim-Subscription-Key: <your-content-safety-key>'

Python

import requests

url = "https://<endpoint>/contentsafety/image/incidents/<image-incident-name>/incidentsamples/<your-incident-sample-id>?api-version=2024-02-15-preview"

payload = {}
headers = {
  'Ocp-Apim-Subscription-Key': '<your-content-safety-key>'
}

response = requests.request("GET", url, headers=headers, data=payload)

print(response.text)

Delete the incident sample

Use an incident sample ID to retrieve and delete that sample.

cURL
Python

Bash

curl --location 'https://<endpoint>/contentsafety/image/incidents/<image-incident-name>:removeIncidentSamples?api-version=2024-02-15-preview' \
--header 'Ocp-Apim-Subscription-Key: <your-content-safety-key>' \
--header 'Content-Type: application/json' \
--data '{
  \"IncidentSampleIds\": [
    \"<your-incident-sample-id>\"
  ]
}'

Python

import requests
import json

url = "https://<endpoint>/contentsafety/image/incidents/<image-incident-name>:removeIncidentSamples?api-version=2024-02-15-preview"

payload = json.dumps({
  "IncidentSampleIds": [
    "<your-incident-sample-id>"
  ]
})
headers = {
  'Ocp-Apim-Subscription-Key': '<your-content-safety-key>',
  'Content-Type': 'application/json'
}

response = requests.request("POST", url, headers=headers, data=payload)

print(response.text)

Additional resources

Documentation

Quickstart: Custom categories (preview) - Azure AI services

Use the custom categories API to create your own content categories and train the Content Safety model for your use case.
Embedded content safety - Azure AI Content Safety - Azure AI services

Embedded content safety is designed for on-device scenarios where cloud connectivity is intermittent or unavailable.
Harm categories in Azure AI Content Safety - Azure AI services

Learn about the different content moderation flags and severity levels that the Azure AI Content Safety service returns.
Custom categories in Azure AI Content Safety (preview) - Azure AI services

Learn about custom content categories and the different ways you can use Azure AI Content Safety to handle them on your platform.
What's new in Azure AI Content Safety? - Azure AI services

Stay up to date on recent releases and updates to Azure AI Content Safety.
Use blocklists for text moderation - Azure AI services

Learn how to customize text moderation in Azure AI Content Safety by using your own list of blocklistItems.
Quickstart: Analyze text content - Azure AI services

Get started using Azure AI Content Safety to analyze text content for objectionable material.

Training

Module

Use AI responsibly with Azure AI Content Safety - Training

As the amount of user-generated online content increases, so does the need to ensure harmful material is moderated effectively. Azure AI Content Safety resource includes features to help organizations moderate and manage both user-generated and AI-generated content.

Certification

Microsoft Certified: Security Operations Analyst Associate - Certifications

Investigate, search for, and mitigate threats using Microsoft Sentinel, Microsoft Defender for Cloud, and Microsoft 365 Defender.

Events

Build AI Apps and Agents

17 Mar, 21 - 21 Mar, 10

Join the meetup series to build scalable AI solutions based on real-world use cases with fellow developers and experts.

Share via

Use the custom categories (rapid) API (preview)

Prerequisites

Test the text custom categories (rapid) API

Create an incident object

Add samples to the incident

Deploy the incident

Detect text incidents

Test the image custom categories (rapid) API

Create an incident

Add samples to the incident

Deploy the incident

Detect image incidents

Other incident operations

Text incidents API

List all incidents

Get the incident details

Delete the incident

List all samples under an incident

Get an incident sample's details

Delete an incident sample

Image incidents API

Get the incidents list

Get the incident details

Delete the incident

List all samples under an incident

Get the incident sample details

Delete the incident sample

Feedback

Additional resources

Share via

Use the custom categories (rapid) API (preview)

Prerequisites

Test the text custom categories (rapid) API

Create an incident object

Add samples to the incident

Deploy the incident

Detect text incidents

Test the image custom categories (rapid) API

Create an incident

Add samples to the incident

Deploy the incident

Detect image incidents

Other incident operations

Text incidents API

List all incidents

Get the incident details

Delete the incident

List all samples under an incident

Get an incident sample's details

Delete an incident sample

Image incidents API

Get the incidents list

Get the incident details

Delete the incident

List all samples under an incident

Get the incident sample details

Delete the incident sample

Related content

Feedback

Additional resources