Use the custom categories (rapid) API (preview)

The custom categories (rapid) API lets you quickly respond to emerging harmful content incidents. You can define an incident with a few examples in a specific topic, and the service will start detecting similar content.

Follow these steps to define an incident with a few examples of text content and then analyze new text content to see if it matches the incident.

Important

This new feature is only available in select Azure regions. See Region availability.

Caution

The sample data in this guide might contain offensive content. User discretion is advised.

Prerequisites

• An Azure subscription - Create one for free
• Once you have your Azure subscription, create a Content Safety resource in the Azure portal to get your key and endpoint. Enter a unique name for your resource, select your subscription, and select a resource group, supported region (see Region availability), and supported pricing tier. Then select Create.
  • The resource takes a few minutes to deploy. After it finishes, Select go to resource. In the left pane, under Resource Management, select Subscription Key and Endpoint. The endpoint and either of the keys are used to call APIs.
• Also create a blob storage container if you want to upload your images there. You can alternatively encode your images as Base64 strings and use them directly in the API calls.
• One of the following installed:
  • cURL for REST API calls.
  • Python 3.x installed

Test the text custom categories (rapid) API

Use the sample code in this section to create a text incident, add samples to the incident, deploy the incident, and then detect text incidents.

Create an incident object

cURL

In the commands below, replace <your_api_key>, <your_endpoint>, and other necessary parameters with your own values.

The following command creates an incident with a name and definition.

curl --location --request PATCH 'https://<endpoint>/contentsafety/text/incidents/<text-incident-name>?api-version=2024-02-15-preview' \
--header 'Ocp-Apim-Subscription-Key: <your-content-safety-key>' \
--header 'Content-Type: application/json' \
--data '{ \"incidentName\": \"<test-incident>\", \"incidentDefinition\": \"<string>\"}'

Python

First, you need to install the required Python library:

pip install requests

Then, define the necessary variables with your own Azure resource details:

import requests

API_KEY = '<your_api_key>'
ENDPOINT = '<your_endpoint>'

headers = {
    'Ocp-Apim-Subscription-Key': API_KEY,
    'Content-Type': 'application/json'
}

The following command creates an incident with a name and definition.

import requests
import json

url = "https://<endpoint>/contentsafety/text/incidents/<text-incident-name>?api-version=2024-02-15-preview"

payload = json.dumps({
  "incidentName": "<text-incident-name>",
  "incidentDefinition": "string"
})
headers = {
  'Ocp-Apim-Subscription-Key': '<your-content-safety-key>',
  'Content-Type': 'application/json'
}

response = requests.request("PATCH", url, headers=headers, data=payload)

print(response.text)

Add samples to the incident

Use the following command to add text examples to the incident.

cURL

curl --location 'https://<endpoint>/contentsafety/text/incidents/<text-incident-name>:addIncidentSamples?api-version=2024-02-15-preview' \
--header 'Ocp-Apim-Subscription-Key: <your-content-safety-key>' \
--header 'Content-Type: application/json' \
--data-raw '{
  \"IncidentSamples\": [
    { \"text\":    \"<text-example-1>\"},
    { \"text\":    \"<text-example-2>\"},
    ...
  ]
}'

Python

import requests
import json

url = "https://<endpoint>/contentsafety/text/incidents/<text-incident-name>:addIncidentSamples?api-version=2024-02-15-preview"

payload = json.dumps({
  "IncidentSamples": [
    {
      "text": "<text-example-1>"
    },
    {
      "text": "<text-example-1>"
    },
    ...
  ]
})
headers = {
  'Ocp-Apim-Subscription-Key': '<your-content-safety-key>',
  'Content-Type': 'application/json'
}

response = requests.request("POST", url, headers=headers, data=payload)

print(response.text)

Deploy the incident

Use the following command to deploy the incident, making it available for the analysis of new content.

cURL

curl --location 'https://<endpoint>/contentsafety/text/incidents/<text-incident-name>:deploy?api-version=2024-02-15-preview' \
--header 'Ocp-Apim-Subscription-Key: <your-content-safety-key>' \
--header 'Content-Type: application/json' 

Python

import requests
import json

url = "https://<endpoint>/contentsafety/text/incidents/<text-incident-name>:deploy?api-version=2024-02-15-preview"

payload = {}
headers = {
  'Ocp-Apim-Subscription-Key': '<your-content-safety-key>',
  'Content-Type': 'application/json'
}

response = requests.request("POST", url, headers=headers, data=payload)

print(response.text)

Detect text incidents

Run the following command to analyze sample text content for the incident you just deployed.

cURL

curl --location 'https://<endpoint>/contentsafety/text:detectIncidents?api-version=2024-02-15-preview' \
--header 'Ocp-Apim-Subscription-Key: <your-content-safety-key>' \
--header 'Content-Type: application/json' \
--data '{
  \"text\":  \"<test-text>\",
  \"incidentNames\": [
    \"<text-incident-name>\"
  ]
}'

Python

import requests
import json

url = "https://<endpoint>/contentsafety/text:detectIncidents?api-version=2024-02-15-preview"

payload = json.dumps({
  "text": "<test-text>",
  "incidentNames": [
    "<text-incident-name>"
  ]
})
headers = {
  'Ocp-Apim-Subscription-Key': '<your-content-safety-key>',
  'Content-Type': 'application/json'
}

response = requests.request("POST", url, headers=headers, data=payload)

print(response.text)

Test the image custom categories (rapid) API

Use the sample code in this section to create an image incident, add samples to the incident, deploy the incident, and then detect image incidents.

Create an incident

cURL

In the commands below, replace <your_api_key>, <your_endpoint>, and other necessary parameters with your own values.

The following command creates an image incident:

curl --location --request PATCH 'https://<endpoint>/contentsafety/image/incidents/<image-incident-name>?api-version=2024-02-15-preview' \
--header 'Ocp-Apim-Subscription-Key: <your-content-safety-key>' \
--header 'Content-Type: application/json' \
--data '{
  \"incidentName\": \"<image-incident-name>\"
}'

Python

Make sure you've installed required Python libraries:

pip install requests

Define the necessary variables with your own Azure resource details:

import requests

API_KEY = '<your_api_key>'
ENDPOINT = '<your_endpoint>'

headers = {
    'Ocp-Apim-Subscription-Key': API_KEY,
    'Content-Type': 'application/json'
}

The following command creates an image incident:

import requests
import json

url = "https://<endpoint>/contentsafety/image/incidents/<image-incident-name>?api-version=2024-02-15-preview"

payload = json.dumps({
  "incidentName": "<image-incident-name>"
})
headers = {
  'Ocp-Apim-Subscription-Key': '<your-content-safety-key>',
  'Content-Type': 'application/json'
}

response = requests.request("PATCH", url, headers=headers, data=payload)

print(response.text)

Add samples to the incident

Use the following command to add examples images to your incident. The image samples can be URLs pointing to images in an Azure blob storage container, or they can be Base64 strings.

cURL

curl --location 'https://<endpoint>/contentsafety/image/incidents/<image-incident-name>:addIncidentSamples?api-version=2024-02-15-preview' \
--header 'Ocp-Apim-Subscription-Key: <your-content-safety-key>' \
--header 'Content-Type: application/json' \
--data '{
  \"IncidentSamples\": [
    {
      \"image\": {
        \"content\": \"<base64-data>\",
        \"bloburl\": \"<your-blob-storage-url>.png\"
      }
    }
  ]
}'

Python

import requests
import json

url = "https://<endpoint>/contentsafety/image/incidents/<image-incident-name>:addIncidentSamples?api-version=2024-02-15-preview"

payload = json.dumps({
  "IncidentSamples": [
    {
      "image": {
        "content": "<base64-data>",
        "bloburl": "<your-blob-storage-url>/image.png"
      }
    }
  ]
})
headers = {
  'Ocp-Apim-Subscription-Key': '<your-content-safety-key>',
  'Content-Type': 'application/json'
}

response = requests.request("POST", url, headers=headers, data=payload)

print(response.text)

Deploy the incident

Use the following command to deploy the incident, making it available for the analysis of new content.

cURL

curl --location 'https://<endpoint>/contentsafety/image/incidents/<image-incident-name>:deploy?api-version=2024-02-15-preview' \
--header 'Ocp-Apim-Subscription-Key: <your-content-safety-key>' \
--header 'Content-Type: application/json' 

Python

import requests
import json

url = "https://<endpoint>/contentsafety/image/incidents/<image-incident-name>:deploy?api-version=2024-02-15-preview"

payload = {}
headers = {
  'Ocp-Apim-Subscription-Key': '<your-content-safety-key>',
  'Content-Type': 'application/json'
}

response = requests.request("POST", url, headers=headers, data=payload)

print(response.text)

Detect image incidents

Use the following command to upload a sample image and test it against the incident you deployed. You can either use a URL pointing to the image in an Azure blob storage container, or you can add the image data as a Base64 string.

cURL

curl --location 'https://<endpoint>/contentsafety/image:detectIncidents?api-version=2024-02-15-preview' \
--header 'Ocp-Apim-Subscription-Key: <your-content-safety-key>' \
--header 'Content-Type: application/json' \
--data '{
    \"image\": {
      \"url\": \"<your-blob-storage-url>/image.png\",
      "content": "<base64-data>"
    },
    \"incidentNames\": [
      \"<image-incident-name>\"
    ]
  }
}'

Python

import requests
import json

url = "https://<endpoint>/contentsafety/image:detectIncidents?api-version=2024-02-15-preview"

payload = json.dumps({
  "image": {
    "url": "<your-blob-storage-url>/image.png",
    "content": "<base64-data>"
  },
  "incidentNames": [
    "<image-incident-name>"
  ]
})
headers = {
  'Ocp-Apim-Subscription-Key': '<your-content-safety-key>',
  'Content-Type': 'application/json'
}

response = requests.request("POST", url, headers=headers, data=payload)

print(response.text)

Other incident operations

The following operations are useful for managing incidents and incident samples.

Text incidents API

List all incidents

cURL

curl --location GET 'https://<endpoint>/contentsafety/text/incidents?api-version=2024-02-15-preview' \
--header 'Ocp-Apim-Subscription-Key: <your-content-safety-key>'

Python

import requests

url = "https://<endpoint>/contentsafety/text/incidents?api-version=2024-02-15-preview"

payload = {}
headers = {
  'Ocp-Apim-Subscription-Key': '<your-content-safety-key>'
}

response = requests.request("GET", url, headers=headers, data=payload)

print(response.text)

Get the incident details

cURL

curl --location GET 'https://<endpoint>/contentsafety/text/incidents/<text-incident-name>?api-version=2024-02-15-preview' \
--header 'Ocp-Apim-Subscription-Key: <your-content-safety-key>'

Python

import requests

url = "https://<endpoint>/contentsafety/text/incidents/<text-incident-name>?api-version=2024-02-15-preview"

payload = {}
headers = {
  'Ocp-Apim-Subscription-Key': '<your-content-safety-key>'
}

response = requests.request("GET", url, headers=headers, data=payload)

print(response.text)

Delete the incident

cURL

curl --location --request DELETE 'https://<endpoint>/contentsafety/text/incidents/<text-incident-name>?api-version=2024-02-15-preview' \
--header 'Ocp-Apim-Subscription-Key: <your-content-safety-key>'

Python

import requests

url = "https://<endpoint>/contentsafety/text/incidents/<text-incident-name>?api-version=2024-02-15-preview"

payload = {}
headers = {
  'Ocp-Apim-Subscription-Key': '<your-content-safety-key>'
}

response = requests.request("DELETE", url, headers=headers, data=payload)

print(response.text)

List all samples under an incident

This command retrieves the unique IDs of all the samples associated with a given incident object.

cURL

curl --location GET 'https://<endpoint>/contentsafety/text/incidents/<text-incident-name>/incidentsamples?api-version=2024-02-15-preview' \
--header 'Ocp-Apim-Subscription-Key: <your-content-safety-key>'

Python

import requests

url = "https://<endpoint>/contentsafety/text/incidents/<text-incident-name>/incidentsamples?api-version=2024-02-15-preview"

payload = {}
headers = {
  'Ocp-Apim-Subscription-Key': '<your-content-safety-key>'
}

response = requests.request("GET", url, headers=headers, data=payload)

print(response.text)

Get an incident sample's details

Use an incident sample ID to look up details about the sample.

cURL

curl --location GET 'https://<endpoint>/contentsafety/text/incidents/<text-incident-name>/incidentsamples/<your-incident-sample-id>?api-version=2024-02-15-preview' \
--header 'Ocp-Apim-Subscription-Key: <your-content-safety-key>'

Python

import requests

url = "https://<endpoint>/contentsafety/text/incidents/<text-incident-name>/incidentsamples/<your-incident-sample-id>?api-version=2024-02-15-preview"

payload = {}
headers = {
  'Ocp-Apim-Subscription-Key': '<your-content-safety-key>'
}

response = requests.request("GET", url, headers=headers, data=payload)

print(response.text)

Delete an incident sample

Use an incident sample ID to retrieve and delete that sample.

cURL

curl --location 'https://<endpoint>/contentsafety/text/incidents/<text-incident-name>:removeIncidentSamples?api-version=2024-02-15-preview' \
--header 'Ocp-Apim-Subscription-Key: <your-content-safety-key>' \
--header 'Content-Type: application/json' \
--data '{
  \"IncidentSampleIds\": [
    \"<your-incident-sample-id>\"
  ]
}'

Python

import requests
import json

url = "https://<endpoint>/contentsafety/text/incidents/<text-incident-name>:removeIncidentSamples?api-version=2024-02-15-preview"

payload = json.dumps({
  "IncidentSampleIds": [
    "<your-incident-sample-id>"
  ]
})
headers = {
  'Ocp-Apim-Subscription-Key': '<your-content-safety-key>',
  'Content-Type': 'application/json'
}

response = requests.request("POST", url, headers=headers, data=payload)

print(response.text)

Image incidents API

Get the incidents list

cURL

curl --location GET 'https://<endpoint>/contentsafety/image/incidents?api-version=2024-02-15-preview' \
--header 'Ocp-Apim-Subscription-Key: <your-content-safety-key>'

Python

import requests

url = "https://<endpoint>/contentsafety/image/incidents?api-version=2024-02-15-preview"

payload = {}
headers = {
  'Ocp-Apim-Subscription-Key': '<your-content-safety-key>'
}

response = requests.request("GET", url, headers=headers, data=payload)

print(response.text)

Get the incident details

cURL

curl --location GET 'https://<endpoint>/contentsafety/image/incidents/<image-incident-name>?api-version=2024-02-15-preview' \
--header 'Ocp-Apim-Subscription-Key: <your-content-safety-key>'

Python

import requests

url = "https://<endpoint>/contentsafety/image/incidents/<image-incident-name>?api-version=2024-02-15-preview"

payload = {}
headers = {
  'Ocp-Apim-Subscription-Key': '<your-content-safety-key>'
}

response = requests.request("GET", url, headers=headers, data=payload)

print(response.text)

Delete the incident

cURL

curl --location --request DELETE 'https://<endpoint>/contentsafety/image/incidents/<image-incident-name>?api-version=2024-02-15-preview' \
--header 'Ocp-Apim-Subscription-Key: <your-content-safety-key>'

Python

import requests

url = "https://<endpoint>/contentsafety/image/incidents/<image-incident-name>?api-version=2024-02-15-preview"

payload = {}
headers = {
  'Ocp-Apim-Subscription-Key': '<your-content-safety-key>'
}

response = requests.request("DELETE", url, headers=headers, data=payload)

print(response.text)

List all samples under an incident

This command retrieves the unique IDs of all the samples associated with a given incident object.

cURL

curl --location GET 'https://<endpoint>/contentsafety/image/incidents/<image-incident-name>/incidentsamples?api-version=2024-02-15-preview' \
--header 'Ocp-Apim-Subscription-Key: <your-content-safety-key>'

Python

import requests

url = "https://<endpoint>/contentsafety/image/incidents/<image-incident-name>/incidentsamples?api-version=2024-02-15-preview"

payload = {}
headers = {
  'Ocp-Apim-Subscription-Key': '<your-content-safety-key>'
}

response = requests.request("GET", url, headers=headers, data=payload)

print(response.text)

Get the incident sample details

Use an incident sample ID to look up details about the sample.

cURL

curl --location GET 'https://<endpoint>/contentsafety/image/incidents/<image-incident-name>/incidentsamples/<your-incident-sample-id>?api-version=2024-02-15-preview' \
--header 'Ocp-Apim-Subscription-Key: <your-content-safety-key>'

Python

import requests

url = "https://<endpoint>/contentsafety/image/incidents/<image-incident-name>/incidentsamples/<your-incident-sample-id>?api-version=2024-02-15-preview"

payload = {}
headers = {
  'Ocp-Apim-Subscription-Key': '<your-content-safety-key>'
}

response = requests.request("GET", url, headers=headers, data=payload)

print(response.text)

Delete the incident sample

Use an incident sample ID to retrieve and delete that sample.

cURL

curl --location 'https://<endpoint>/contentsafety/image/incidents/<image-incident-name>:removeIncidentSamples?api-version=2024-02-15-preview' \
--header 'Ocp-Apim-Subscription-Key: <your-content-safety-key>' \
--header 'Content-Type: application/json' \
--data '{
  \"IncidentSampleIds\": [
    \"<your-incident-sample-id>\"
  ]
}'

Python

import requests
import json

url = "https://<endpoint>/contentsafety/image/incidents/<image-incident-name>:removeIncidentSamples?api-version=2024-02-15-preview"

payload = json.dumps({
  "IncidentSampleIds": [
    "<your-incident-sample-id>"
  ]
})
headers = {
  'Ocp-Apim-Subscription-Key': '<your-content-safety-key>',
  'Content-Type': 'application/json'
}

response = requests.request("POST", url, headers=headers, data=payload)

print(response.text)

Related content

• Custom categories concepts
• What is Azure AI Content Safety?