Edit

Share via

Run AI Red Teaming Agent in the cloud (preview)

Note

This document refers to the Microsoft Foundry (classic) portal.

🔄 Switch to the Microsoft Foundry (new) documentation if you're using the new portal.

Note

This document refers to the Microsoft Foundry (new) portal.

Important

Items marked (preview) in this article are currently in public preview. This preview is provided without a service-level agreement, and we don't recommend it for production workloads. Certain features might not be supported or might have constrained capabilities. For more information, see Supplemental Terms of Use for Microsoft Azure Previews.

Though the AI Red Teaming Agent (preview) can be run locally during prototyping and development to help identify safety risks, running them in the cloud allows for pre-deployment AI red teaming runs on larger combinations of attack strategies and risk categories for a fuller analysis.

Though the AI Red Teaming Agent can be run locally during prototyping and development to help identify safety risks, running them in the cloud allows for the following scenarios:

  • Pre-deployment AI red teaming runs on larger combinations of attack strategies and risk categories for a fuller analysis,
  • Post-deployment continuous AI red teaming runs that can be scheduled to run at set time intervals
  • Agentic-specific risk scenarios to support a minimally sandboxed environment for the AI red teaming run

Prerequisites

Note

You must use a Foundry project for this feature. A hub-based project isn't supported. See How do I know which type of project I have? and Create a Foundry project. To migrate your hub-based project to a Foundry project, see Migrate from hub-based to Foundry projects.

If this is your first time running evaluations and logging it to your Microsoft Foundry project, you might need to do a few additional steps:

  1. Create and connect your storage account to your Foundry project at the resource level. There are two ways you can do this. You can use a Bicep template, which provisions and connects a storage account to your Foundry project with key authentication.
    You can also manually create and provision access to your storage account in the Azure portal.
  2. Make sure the connected storage account has access to all projects.
  3. If you connected your storage account with Microsoft Entra ID, make sure to give managed identity Storage Blob Data Owner permissions to both your account and the Foundry project resource in the Azure portal.

Getting started

First, install Microsoft Foundry SDK's project client, which runs the AI Red Teaming Agent in the cloud.

uv install azure-ai-projects==1.1.0b3 azure-identity

uv install azure-ai-projects>=2.0.0b1 azure-identity

Note

For more detailed information, see the REST API Reference Documentation.

Then, set your environment variables for your Microsoft Foundry resources

import os

endpoint = os.environ["PROJECT_ENDPOINT"] # Sample : https://<account_name>.services.ai.azure.com/api/projects/<project_name>


import os

endpoint = os.environ["PROJECT_ENDPOINT"] # Sample : https://<account_name>.services.ai.azure.com/api/projects/<project_name>
agent_name = os.environ["AZURE_AI_AGENT_NAME"] # Required. The name of the Agent to perform red teaming evaluation on.

Supported targets

Running the AI Red Teaming Agent in the cloud currently only supports Azure OpenAI model deployments in your Foundry project as a target.

Running the AI Red Teaming Agent in the cloud currently only supports the following:

  • Foundry project deployments
  • Azure OpenAI model deployments
  • Foundry Agents (prompt and container agents) in your Microsoft Foundry project as a target.

Configure your target model

You can configure your target model deployment in two ways:

Option 1: Foundry project deployments

If you're using model deployments that are part of your Foundry project, set up the following environment variables:

import os

model_endpoint = os.environ["MODEL_ENDPOINT"] # Sample : https://<account_name>.openai.azure.com
model_api_key = os.environ["MODEL_API_KEY"]
model_deployment_name = os.environ["MODEL_DEPLOYMENT_NAME"] # Sample : gpt-4o-mini

Option 2: Azure OpenAI/Foundry Tools deployments

If you want to use deployments from your Azure OpenAI or Foundry Tools accounts, you first need to connect these resources to your Foundry project through connections.

  1. Create a connection: Follow the instructions in Configure project connections to connect your Azure OpenAI or AI Services resource to your Foundry project.

  2. Get the connection name: After connecting the account, you'll see the connection created with a generated name in your Foundry project.

  3. Configure the target: Use the format "connectionName/deploymentName" for your model deployment configuration:

# Format: "connectionName/deploymentName"
model_deployment_name = "my-openai-connection/gpt-4o-mini"

Create an AI red teaming run

from azure.identity import DefaultAzureCredential
from azure.ai.projects import AIProjectClient
from azure.ai.projects.models import (
    RedTeam,
    AzureOpenAIModelConfiguration,
    AttackStrategy,
    RiskCategory,
)

with AIProjectClient(
  endpoint=endpoint,
  credential=DefaultAzureCredential(exclude_interactive_browser_credential=False),
) as project_client:

  # Create target configuration for testing an Azure OpenAI model
  target_config = AzureOpenAIModelConfiguration(model_deployment_name=model_deployment_name)

  # Instantiate the AI Red Teaming Agent
  red_team_agent = RedTeam(
      attack_strategies=[AttackStrategy.BASE64],
      risk_categories=[RiskCategory.VIOLENCE],
      display_name="red-team-cloud-run", 
      target=target_config,
  )

  # Create and run the red teaming scan
  # If you configured target using Option 1, use:
  # headers = {"model-endpoint": model_endpoint, "api-key": model_api_key}
  # If you configured target using Option 2, use:
  # headers = {}

  # Choose one of the following based on your configuration option:
  headers = {"model-endpoint": model_endpoint, "api-key": model_api_key}  # For Option 1
  # headers = {}  # For Option 2

  red_team_response = project_client.red_teams.create(red_team=red_team_agent, headers=headers)

Get an AI red teaming run

# Use the name returned by the create operation for the get call
get_red_team_response = project_client.red_teams.get(name=red_team_response.name)
print(f"Red Team scan status: {get_red_team_response.status}")

List all AI red teaming runs

for scan in project_client.red_teams.list():
  print(f"Found scan: {scan.name}, Status: {scan.status}")

Create an AI red team

Create a red team to hold one or more runs that share a data source and risk categories.

def main() -> None:
    load_dotenv()

    endpoint = os.environ.get("AZURE_AI_PROJECT_ENDPOINT", "")
    agent_name = os.environ.get("AZURE_AI_AGENT_NAME", "")
    model_deployment = os.environ.get("AZURE_AI_MODEL_DEPLOYMENT_NAME", "")
    data_folder = os.environ.get("DATA_FOLDER", "./redteam_outputs")
    os.makedirs(data_folder, exist_ok=True)

    with (
        DefaultAzureCredential() as credential,
        AIProjectClient(endpoint=endpoint, credential=credential) as project_client,
        project_client.get_openai_client() as client,
    ):
        # (Optional) Create a new agent version for this run
        agent_version = project_client.agents.create_version(
            agent_name=agent_name,
            definition=PromptAgentDefinition(
                model=model_deployment,
                instructions="You are a helpful assistant that answers general questions."
            ),
        )
        print(f"[Agent] Created: id={agent_version.id}, name={agent_version.name}, version={agent_version.version}")

        # Create an Red Team
        red_team_name = f"Red Team Agentic Safety Evaluation - {int(time.time())}"
        data_source_config = {"type": "azure_ai_source", "scenario": "red_team"}
        testing_criteria = _get_agent_safety_evaluation_criteria()

        print("[Group] Creating red team...")
        red_team = client.evals.create(
            name=red_team_name,
            data_source_config=data_source_config,
            testing_criteria=testing_criteria, 
        )
        print(f"[Group] Created: id={red_team.id}, name={red_team.name}")

What it does:

  • Creates a red team to hold all red teaming runs
  • Configures the red team with three built‑in evaluators (Prohibited Actions, Task Adherence, Sensitive Data Leakage).

You’ll receive:

  • A JSON body with the group’s metadata, including ID (save it as {{red_team_id}} for later).

Get a red team

Use this to verify the red team exists and review configuration (criteria, data source, timestamps).

print(f"[Group] Retrieving group by id={red_team.id} ...")
red_team_fetched = client.evals.retrieve(red_team.id)
print("[Group] Response:")
print(red_team_fetched)

Create (or update) an evaluation taxonomy

To red team for the agentic risk category of prohibited actions, you need to be able to confirm, edit, or update the evaluation taxonomy of prohibited actions generated by the prohibited action red teaming workflow. The next example will generate a JSON file with a generated taxonomy of prohibited actions to be used in dynamically generating the attack prompts to test agentic behavior based on user-approved policy. Once you've reviewed and confirmed the taxonomy, it will then be used to create a red teaming run as well as assess the Attack Success Rate (ASR) of the agent outputs.

print("[Taxonomy] Creating...")
target = AzureAIAgentTarget(
    name=agent_name,
    version=agent_version.version,
    tool_descriptions=_get_tool_descriptions(agent_version),
)
taxonomy_input = AgentTaxonomyInput(
    risk_categories=[RiskCategory.PROHIBITED_ACTIONS],  # add more risks if desired
    target=target
)  # type: ignore

eval_taxonomy = EvaluationTaxonomy(
    description="Taxonomy for red teaming run",
    taxonomy_input=taxonomy_input,
)

taxonomy = project_client.evaluation_taxonomies.create(
    name=agent_name,  # you can choose another unique taxonomy name
    body=eval_taxonomy
)
taxonomy_file_id = taxonomy.id  # used as the 'file_id' source for runs

# Save taxonomy metadata for reference
taxonomy_path = os.path.join(data_folder, f"taxonomy_{agent_name}.json")
with open(taxonomy_path, "w") as f:
    f.write(json.dumps(_to_json_primitive(taxonomy), indent=2))
print(f"[Taxonomy] Created. Saved to {taxonomy_path}")

What it does:

  • Creates/updates a taxonomy resource named {{name}} that:
    • Defines an agent target and tool descriptions
    • Specifies the risk categories of ProhibitedActions

You’ll reference it

  • via a file_id URI in the Create Run request.

Create a run in a red team

A run generates items from a source (for example, taxonomy) and red teams the target agent with chosen attack strategies.

eval_run_name = f"Red Team Agent Safety Eval Run for {agent_name} - {int(time.time())}"

print("[Run] Creating eval run...")
eval_run = client.evals.runs.create(
    eval_id=red_team.id,
    name=eval_run_name,
    data_source={  
        "type": "azure_ai_red_team",
        "item_generation_params": {
            "type": "red_team_taxonomy",
            "attack_strategies": ["Flip", "Base64", "IndirectJailbreak"],
            "num_turns": 5,
            "source": {
                "type": "file_id",
                "id": taxonomy_file_id,
            },
        },
        "target": target.as_dict(),
    },
)
print(f"[Run] Created: id={eval_run.id}, name={eval_run.name}, status={eval_run.status}")

Key fields to configure your run:

  • attack_strategies: For example, "Flip", "Base64", "IndirectJailbreak" (choose the ones you want to test)
  • num_turns: multi‑turn depth for generated red‑team items
  • source.id: points to your taxonomy by file‑ID URI
  • target: the agent under test (name, version, tools)

You’ll receive

  • A run object including id (save as {{eval_run_id}})

Get a red teaming run (by ID)

Use this to check status of your red teaming run (for example, queued, running, succeeded, failed).

print("[Run] Polling for completion...")
while True:
    run = client.evals.runs.retrieve(run_id=eval_run.id, eval_id=red_team.id)
    print(f"    status={run.status}")
    if run.status in ("completed", "failed", "canceled"):
        break
    time.sleep(5)

Note

The API is synchronous per request, but runs themselves are processed server‑side; poll this endpoint until completion before fetching output items.

List red teaming run output items and results

Use this to inspect summary metrics after completion of the red teaming run.

print("[Run] Fetching output items...")
items = list(client.evals.runs.output_items.list(run_id=run.id, eval_id=red_team.id))
output_path = os.path.join(data_folder, f"redteam_eval_output_items_{agent_name}.json")
with open(output_path, "w") as f:
    f.write(json.dumps(_to_json_primitive(items), indent=2))
print(f"[Run] Done. Status={run.status}. Output items saved to {output_path}")

Viewing AI red teaming results in Microsoft Foundry project (preview)

After your automated scan finishes, the results also get logged to your Foundry project, which you specified in the creation of your AI red teaming agent.

View report of each scan

In your Foundry project or hub-based project, navigate to the Evaluation page. Select AI red teaming to view the report with detailed drill-down results of each scan.

Screenshot of AI Red Teaming tab in Foundry project page.

When you select into the scan, you can view the report by risk categories, which shows the overall number of successful attacks and a breakdown of successful attacks per risk categories:

Screenshot of AI Red Teaming report view by risk category in Foundry.

Or by attack complexity classification:

Screenshot of AI Red Teaming report view by attack complexity category in Foundry.

Drilling down further into the data tab provides a row-level view of each attack-response pair. This information offers deeper insights into system issues and behaviors. For each attack-response pair, you can see more information, such as whether or not the attack was successful, what attack strategy was used, and its attack complexity. A human in the loop reviewer can provide human feedback by selecting the thumbs up or thumbs down icon.

Screenshot of AI Red Teaming data page in Foundry.

To view each conversation, select View more to see the full conversation for more detailed analysis of the AI system's response.

Screenshot of AI Red Teaming data page with a conversation history opened in Foundry.

Related content

Try out an example workflow for agent red teaming in the cloud in our GitHub samples.

Try out an example workflow for agent red teaming in the cloud in our GitHub samples.

  • Last updated on