What is Azure AI Content Safety?
Azure AI Content Safety detects harmful user-generated and AI-generated content in applications and services. Azure AI Content Safety includes text and image APIs that allow you to detect material that is harmful. We also have an interactive Content Safety Studio that allows you to view, explore and try out sample code for detecting harmful content across different modalities.
Content filtering software can help your app comply with regulations or maintain the intended environment for your users.
This documentation contains the following article types:
- Concepts provide in-depth explanations of the service functionality and features.
- Quickstarts are getting-started instructions to guide you through making requests to the service.
- How-to guides contain instructions for using the service in more specific or customized ways.
Where it's used
The following are a few scenarios in which a software developer or team would require a content moderation service:
- User prompts submitted to a generative AI service.
- Content produced by generative AI models.
- Online marketplaces that moderate product catalogs and other user-generated content.
- Gaming companies that moderate user-generated game artifacts and chat rooms.
- Social messaging platforms that moderate images and text added by their users.
- Enterprise media companies that implement centralized moderation for their content.
- K-12 education solution providers filtering out content that is inappropriate for students and educators.
Important
You cannot use Azure AI Content Safety to detect illegal child exploitation images.
Product features
There are different types of analysis available from this service. The following table describes the currently available APIs.
Feature | Functionality | Concepts guide | Get started |
---|---|---|---|
Prompt Shields (preview) | Scans text for the risk of a User input attack on a Large Language Model. | Prompt Shields concepts | Quickstart |
Groundedness detection (preview) | Detects whether the text responses of large language models (LLMs) are grounded in the source materials provided by the users. | Groundedness detection concepts | Quickstart |
Protected material text detection (preview) | Scans AI-generated text for known text content (for example, song lyrics, articles, recipes, selected web content). | Protected material concepts | Quickstart |
Custom categories API (preview) | Lets you create and train your own custom content categories and scan text for matches. | Custom categories concepts | Quickstart |
Custom categories (rapid) API (preview) | Lets you define emerging harmful content patterns and scan text and images for matches. | Custom categories concepts | How-to guide |
Analyze text API | Scans text for sexual content, violence, hate, and self harm with multi-severity levels. | Harm categories | Quickstart |
Analyze image API | Scans images for sexual content, violence, hate, and self harm with multi-severity levels. | Harm categories | Quickstart |
Content Safety Studio
Azure AI Content Safety Studio is an online tool designed to handle potentially offensive, risky, or undesirable content using cutting-edge content moderation ML models. It provides templates and customized workflows, enabling users to choose and build their own content moderation system. Users can upload their own content or try it out with provided sample content.
Content Safety Studio not only contains out-of-the-box AI models but also includes Microsoft's built-in terms blocklists to flag profanities and stay up to date with new trends. You can also upload your own blocklists to enhance the coverage of harmful content that's specific to your use case.
Studio also lets you set up a moderation workflow, where you can continuously monitor and improve content moderation performance. It can help you meet content requirements from all kinds of industries like gaming, media, education, E-commerce, and more. Businesses can easily connect their services to the Studio and have their content moderated in real-time, whether user-generated or AI-generated.
All of these capabilities are handled by the Studio and its backend; customers don’t need to worry about model development. You can onboard your data for quick validation and monitor your KPIs accordingly, like technical metrics (latency, accuracy, recall), or business metrics (block rate, block volume, category proportions, language proportions, and more). With simple operations and configurations, customers can test different solutions quickly and find the best fit, instead of spending time experimenting with custom models or doing moderation manually.
Content Safety Studio features
In Content Safety Studio, the following Azure AI Content Safety service features are available:
Moderate Text Content: With the text moderation tool, you can easily run tests on text content. Whether you want to test a single sentence or an entire dataset, our tool offers a user-friendly interface that lets you assess the test results directly in the portal. You can experiment with different sensitivity levels to configure your content filters and blocklist management, ensuring that your content is always moderated to your exact specifications. Plus, with the ability to export the code, you can implement the tool directly in your application, streamlining your workflow and saving time.
Moderate Image Content: With the image moderation tool, you can easily run tests on images to ensure that they meet your content standards. Our user-friendly interface allows you to evaluate the test results directly in the portal, and you can experiment with different sensitivity levels to configure your content filters. Once you've customized your settings, you can easily export the code to implement the tool in your application.
Monitor Online Activity: The powerful monitoring page allows you to easily track your moderation API usage and trends across different modalities. With this feature, you can access detailed response information, including category and severity distribution, latency, error, and blocklist detection. This information provides you with a complete overview of your content moderation performance, enabling you to optimize your workflow and ensure that your content is always moderated to your exact specifications. With our user-friendly interface, you can quickly and easily navigate the monitoring page to access the information you need to make informed decisions about your content moderation strategy. You have the tools you need to stay on top of your content moderation performance and achieve your content goals.
Security
Use Microsoft Entra ID or Managed Identity to manage access
For enhanced security, you can use Microsoft Entra ID or Managed Identity (MI) to manage access to your resources.
- Managed Identity is automatically enabled when you create a Content Safety resource.
- Microsoft Entra ID is supported in both API and SDK scenarios. Refer to the general AI services guideline of Authenticating with Microsoft Entra ID. You can also grant access to other users within your organization by assigning them the roles of Cognitive Services Users and Reader. To learn more about granting user access to Azure resources using the Azure portal, refer to the Role-based access control guide.
Encryption of data at rest
Learn how Azure AI Content Safety handles the encryption and decryption of your data. Customer-managed keys (CMK), also known as Bring Your Own Key (BYOK), offer greater flexibility to create, rotate, disable, and revoke access controls. You can also audit the encryption keys used to protect your data.
Pricing
Currently, Azure AI Content Safety has an F0 and S0 pricing tier. See the Azure pricing page for more information.
Service limits
Input requirements
See the following list for the input requirements for each feature.
- Analyze text API:
- Default maximum length: 10K characters (split longer texts as needed).
- Analyze image API:
- Maximum image file size: 4 MB
- Dimensions between 50 x 50 and 2048 x 2048 pixels.
- Images can be in JPEG, PNG, GIF, BMP, TIFF, or WEBP formats.
- Prompt Shields (preview):
- Maximum prompt length: 10K characters.
- Up to five documents with a total of 10K characters.
- Groundedness detection (preview):
- Maximum length for grounding sources: 55,000 characters (per API call).
- Maximum text and query length: 7,500 characters.
- Protected material detection (preview):
- Default maximum length: 1K characters.
- Default minimum length: 110 characters (for scanning LLM completions, not user prompts).
- Custom categories (standard):
- Maximum inference input length: 1K characters.
Language support
Content Safety models have been specifically trained and tested in the following languages: English, German, Spanish, Japanese, French, Italian, Portuguese, and Chinese. However, the service can work in many other languages, but the quality might vary. In all cases, you should do your own testing to ensure that it works for your application.
Custom Categories currently only works well in English. You can try to use other languages with your own dataset, but the quality might vary across languages.
For more information, see Language support.
Region availability
To use the Content Safety APIs, you must create your Azure AI Content Safety resource in the supported regions. Currently, the Content Safety features are available in the following Azure regions:
Region | Moderation APIs (text and image) |
Prompt Shields (preview) |
Protected material detection (preview) |
Groundedness detection (preview) |
Custom categories (rapid) (preview) |
Custom categories (standard) |
Blocklists |
---|---|---|---|---|---|---|---|
East US | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
East US 2 | ✅ | ✅ | ✅ | ✅ | |||
West US | ✅ | ||||||
West US 2 | ✅ | ✅ | ✅ | ||||
Central US | ✅ | ✅ | |||||
North Central US | ✅ | ✅ | ✅ | ||||
South Central US | ✅ | ✅ | ✅ | ||||
Canada East | ✅ | ✅ | ✅ | ||||
Switzerland North | ✅ | ✅ | ✅ | ✅ | |||
Sweden Central | ✅ | ✅ | ✅ | ✅ | |||
UK South | ✅ | ✅ | ✅ | ||||
France Central | ✅ | ✅ | ✅ | ||||
West Europe | ✅ | ✅ | ✅ | ✅ | ✅ | ||
Japan East | ✅ | ✅ | ✅ | ||||
Australia East | ✅ | ✅ | ✅ | ✅ | ✅ | ||
USGov Arizona | ✅ | ||||||
USGov Virginia | ✅ |
Feel free to contact us if you need other regions for your business.
Query rates
Content Safety features have query rate limits in requests-per-second (RPS) or requests-per-10-seconds (RP10S) . See the following table for the rate limits for each feature.
Pricing tier | Moderation APIs (text and image) |
Prompt Shields (preview) |
Protected material detection (preview) |
Groundedness detection (preview) |
Custom categories (rapid) (preview) |
Custom categories (standard) (preview) |
---|---|---|---|---|---|---|
F0 | 1000 RP10S | 1000 RP10S | 1000 RP10S | 50 RP10S | 1000 RP10S | 5 RPS |
S0 | 1000 RP10S | 1000 RP10S | 1000 RP10S | 50 RP10S | 1000 RP10S | 5 RPS |
If you need a faster rate, please contact us to request.
Contact us
If you get stuck, email us or use the feedback widget at the bottom of any Microsoft Learn page.
Next steps
Follow a quickstart to get started using Azure AI Content Safety in your application.
Povratne informacije
https://aka.ms/ContentUserFeedback.
Kmalu na voljo: V letu 2024 bomo ukinili storitev Težave v storitvi GitHub kot mehanizem za povratne informacije za vsebino in jo zamenjali z novim sistemom za povratne informacije. Za več informacij si oglejte:Pošlji in prikaži povratne informacije za