Use the custom categories (rapid) API (preview)

The custom categories (rapid) API lets you quickly respond to emerging harmful content incidents. You can define an incident with a few examples in a specific topic, and the service will start detecting similar content.

Follow these steps to define an incident with a few examples of text content and then analyze new text content to see if it matches the incident.

Important

This new feature is only available in select Azure regions. See Region availability.

Caution

The sample data in this guide might contain offensive content. User discretion is advised.

Prerequisites

  • An Azure subscription - Create one for free
  • Once you have your Azure subscription, create a Content Safety resource in the Azure portal to get your key and endpoint. Enter a unique name for your resource, select your subscription, and select a resource group, supported region (see Region availability), and supported pricing tier. Then select Create.
    • The resource takes a few minutes to deploy. After it finishes, Select go to resource. In the left pane, under Resource Management, select Subscription Key and Endpoint. The endpoint and either of the keys are used to call APIs.
  • Also create a blob storage container if you want to upload your images there. You can alternatively encode your images as Base64 strings and use them directly in the API calls.
  • One of the following installed:

Test the text custom categories (rapid) API

Use the sample code in this section to create a text incident, add samples to the incident, deploy the incident, and then detect text incidents.

Create an incident object

In the commands below, replace <your_api_key>, <your_endpoint>, and other necessary parameters with your own values.

The following command creates an incident with a name and definition.

curl --location --request PATCH 'https://<endpoint>/contentsafety/text/incidents/<text-incident-name>?api-version=2024-02-15-preview' \
--header 'Ocp-Apim-Subscription-Key: <your-content-safety-key>' \
--header 'Content-Type: application/json' \
--data '{ \"incidentName\": \"<test-incident>\", \"incidentDefinition\": \"<string>\"}'

Add samples to the incident

Use the following command to add text examples to the incident.

curl --location 'https://<endpoint>/contentsafety/text/incidents/<text-incident-name>:addIncidentSamples?api-version=2024-02-15-preview' \
--header 'Ocp-Apim-Subscription-Key: <your-content-safety-key>' \
--header 'Content-Type: application/json' \
--data-raw '{
  \"IncidentSamples\": [
    { \"text\":    \"<text-example-1>\"},
    { \"text\":    \"<text-example-2>\"},
    ...
  ]
}'

Deploy the incident

Use the following command to deploy the incident, making it available for the analysis of new content.

curl --location 'https://<endpoint>/contentsafety/text/incidents/<text-incident-name>:deploy?api-version=2024-02-15-preview' \
--header 'Ocp-Apim-Subscription-Key: <your-content-safety-key>' \
--header 'Content-Type: application/json' 

Detect text incidents

Run the following command to analyze sample text content for the incident you just deployed.

curl --location 'https://<endpoint>/contentsafety/text:detectIncidents?api-version=2024-02-15-preview' \
--header 'Ocp-Apim-Subscription-Key: <your-content-safety-key>' \
--header 'Content-Type: application/json' \
--data '{
  \"text\":  \"<test-text>\",
  \"incidentNames\": [
    \"<text-incident-name>\"
  ]
}'

Test the image custom categories (rapid) API

Use the sample code in this section to create an image incident, add samples to the incident, deploy the incident, and then detect image incidents.

Create an incident

In the commands below, replace <your_api_key>, <your_endpoint>, and other necessary parameters with your own values.

The following command creates an image incident:

curl --location --request PATCH 'https://<endpoint>/contentsafety/image/incidents/<image-incident-name>?api-version=2024-02-15-preview' \
--header 'Ocp-Apim-Subscription-Key: <your-content-safety-key>' \
--header 'Content-Type: application/json' \
--data '{
  \"incidentName\": \"<image-incident-name>\"
}'

Add samples to the incident

Use the following command to add examples images to your incident. The image samples can be URLs pointing to images in an Azure blob storage container, or they can be Base64 strings.

curl --location 'https://<endpoint>/contentsafety/image/incidents/<image-incident-name>:addIncidentSamples?api-version=2024-02-15-preview' \
--header 'Ocp-Apim-Subscription-Key: <your-content-safety-key>' \
--header 'Content-Type: application/json' \
--data '{
  \"IncidentSamples\": [
    {
      \"image\": {
        \"content\": \"<base64-data>\",
        \"bloburl\": \"<your-blob-storage-url>.png\"
      }
    }
  ]
}'

Deploy the incident

Use the following command to deploy the incident, making it available for the analysis of new content.

curl --location 'https://<endpoint>/contentsafety/image/incidents/<image-incident-name>:deploy?api-version=2024-02-15-preview' \
--header 'Ocp-Apim-Subscription-Key: <your-content-safety-key>' \
--header 'Content-Type: application/json' 

Detect image incidents

Use the following command to upload a sample image and test it against the incident you deployed. You can either use a URL pointing to the image in an Azure blob storage container, or you can add the image data as a Base64 string.

curl --location 'https://<endpoint>/contentsafety/image:detectIncidents?api-version=2024-02-15-preview' \
--header 'Ocp-Apim-Subscription-Key: <your-content-safety-key>' \
--header 'Content-Type: application/json' \
--data '{
    \"image\": {
      \"url\": \"<your-blob-storage-url>/image.png\",
      "content": "<base64-data>"
    },
    \"incidentNames\": [
      \"<image-incident-name>\"
    ]
  }
}'

Other incident operations

The following operations are useful for managing incidents and incident samples.

Text incidents API

List all incidents

curl --location GET 'https://<endpoint>/contentsafety/text/incidents?api-version=2024-02-15-preview' \
--header 'Ocp-Apim-Subscription-Key: <your-content-safety-key>'

Get the incident details

curl --location GET 'https://<endpoint>/contentsafety/text/incidents/<text-incident-name>?api-version=2024-02-15-preview' \
--header 'Ocp-Apim-Subscription-Key: <your-content-safety-key>'

Delete the incident

curl --location --request DELETE 'https://<endpoint>/contentsafety/text/incidents/<text-incident-name>?api-version=2024-02-15-preview' \
--header 'Ocp-Apim-Subscription-Key: <your-content-safety-key>'

List all samples under an incident

This command retrieves the unique IDs of all the samples associated with a given incident object.

curl --location GET 'https://<endpoint>/contentsafety/text/incidents/<text-incident-name>/incidentsamples?api-version=2024-02-15-preview' \
--header 'Ocp-Apim-Subscription-Key: <your-content-safety-key>'

Get an incident sample's details

Use an incident sample ID to look up details about the sample.

curl --location GET 'https://<endpoint>/contentsafety/text/incidents/<text-incident-name>/incidentsamples/<your-incident-sample-id>?api-version=2024-02-15-preview' \
--header 'Ocp-Apim-Subscription-Key: <your-content-safety-key>'

Delete an incident sample

Use an incident sample ID to retrieve and delete that sample.

curl --location 'https://<endpoint>/contentsafety/text/incidents/<text-incident-name>:removeIncidentSamples?api-version=2024-02-15-preview' \
--header 'Ocp-Apim-Subscription-Key: <your-content-safety-key>' \
--header 'Content-Type: application/json' \
--data '{
  \"IncidentSampleIds\": [
    \"<your-incident-sample-id>\"
  ]
}'

Image incidents API

Get the incidents list

curl --location GET 'https://<endpoint>/contentsafety/image/incidents?api-version=2024-02-15-preview' \
--header 'Ocp-Apim-Subscription-Key: <your-content-safety-key>'

Get the incident details

curl --location GET 'https://<endpoint>/contentsafety/image/incidents/<image-incident-name>?api-version=2024-02-15-preview' \
--header 'Ocp-Apim-Subscription-Key: <your-content-safety-key>'

Delete the incident

curl --location --request DELETE 'https://<endpoint>/contentsafety/image/incidents/<image-incident-name>?api-version=2024-02-15-preview' \
--header 'Ocp-Apim-Subscription-Key: <your-content-safety-key>'

List all samples under an incident

This command retrieves the unique IDs of all the samples associated with a given incident object.

curl --location GET 'https://<endpoint>/contentsafety/image/incidents/<image-incident-name>/incidentsamples?api-version=2024-02-15-preview' \
--header 'Ocp-Apim-Subscription-Key: <your-content-safety-key>'

Get the incident sample details

Use an incident sample ID to look up details about the sample.

curl --location GET 'https://<endpoint>/contentsafety/image/incidents/<image-incident-name>/incidentsamples/<your-incident-sample-id>?api-version=2024-02-15-preview' \
--header 'Ocp-Apim-Subscription-Key: <your-content-safety-key>'

Delete the incident sample

Use an incident sample ID to retrieve and delete that sample.

curl --location 'https://<endpoint>/contentsafety/image/incidents/<image-incident-name>:removeIncidentSamples?api-version=2024-02-15-preview' \
--header 'Ocp-Apim-Subscription-Key: <your-content-safety-key>' \
--header 'Content-Type: application/json' \
--data '{
  \"IncidentSampleIds\": [
    \"<your-incident-sample-id>\"
  ]
}'