แชร์ผ่าน

Identify risk for autonomous agentic AI systems

Pillar name: Monitor and detect threats
Pattern name: Reduce autonomous agentic AI system risk

Context and Problem

Autonomous agentic AI systems can plan, execute, and adapt actions toward goals rather than responding to a single prompt. Because they might invoke tools, call APIs, access data, and coordinate across services, they can produce real‑world effects with limited human intervention. This autonomy increases both the impact of errors and the attractiveness of the system to adversaries. Every agent‑to‑tool, agent‑to‑service, and agent‑to‑agent interaction expands the attack surface and can introduce risks such as indirect prompt injection attacks, unintended actions, or data exfiltration.

The following risks (while not exhaustive) commonly emerge in autonomous agentic AI systems.

Design Risks

  • Task adherence: The agent takes actions that do not align with the user’s intended task, plan, or objective.
  • Human oversight and control: The system lacks meaningful points for user review, approval, correction, or interruption of autonomous behavior.
  • System intelligibility: Users lack visibility into what the agent is doing, plans to do, or has already done.
  • Transparency & disclosure: Users or downstream recipients are not aware they are interacting with an AI system or encountering AI‑generated actions/outputs.

Security Risks

  • Agent hijacking: Malicious or untrusted inputs hijack tool calls due to blurred boundaries between data and instructions.
  • Sensitive data leakage: Confidential, proprietary, or personal data is exposed through outputs, logs, memory, or downstream actions.
  • Supply chain compromise: Vulnerabilities are introduced through models, tools, plugins, grounding data, or other agent dependencies.
  • Agent sprawl: Unmanaged or over-permissioned agents proliferate, increasing security risk and reducing IT oversight.

Addressing these risks requires both foundational design principles and risk-specific mitigations, applied consistently across the agent lifecycle.

Solution

Reduce risk in autonomous agentic AI systems by combining foundational design pillars (how the agent behaves and how users stay in control) with targeted security and governance mitigations (how the system resists attack and scales safely). The following pillars form the foundation for responsible agentic system design to address these threats. They apply across all agentic use cases and help mitigate multiple risks simultaneously.

Foundational design pillars

Task adherence

Inadequate task adherence occurs when an agent takes actions that don't fully allign with the user's intended task, plan, or objective. An agent may misinterpret intent, skip required steps, or pursue an inferred objective the user did not authorize.

To manage this risk:

  • Define clear system purpose and boundaries so the agent reliably interprets intent and executes only intended actions.
  • Use deterministic controls to block prohibited actions regardless of model output.
  • Apply least privilege and least action. Allow only the minimum tools, data, and operations required. Deny everything else by default.
  • Communicate about tasks that involve elevated risk and about how the system handles that risk, to prevent overreliance.

Human oversight and control

Human oversight means giving users meaningful control to guide, correct, and interrupt autonomous behavior—especially when input is ambiguous, actions are high‑impact, or adversarial manipulation is possible.

To manage this risk:

  • Let users set boundaries for what agents can access, do, and remember.
  • Require approval for high-risk or irreversible actions.
  • Provide reliable, system-level mechanisms to pause or stop agents safely and immediately.
  • Enforce organizational policies and user preferences consistently across executions.

AI system intelligibility

Intelligibility means the system shows what it plans to do, provides feedback during execution, and summarizes what happened, including which tools and data were used. Without visibility, users can’t undo mistakes, respond to incidents, or improve outcomes.

To design for system intelligibility:

  • Show planned actions before execution, especially for high-risk or irreversible steps.
  • Provide real-time status and progress so users can track behavior as it unfolds.
  • Summarize outcomes: what happened, key decisions, and what the agent used to get there.
  • Maintain accessible post-execution logs that record actions, tools, and outcomes for audit and incident response.

Transparency and disclosure

Autonomous agentic systems might act behind the scenes and affect people who did not initiate the interaction. Clear disclosure sets expectations, reduces confusion, and supports safer use.

To make interactions transparent and understandable:

  • Clearly indicate when users are interacting with an AI system, especially in high‑risk domains or downstream contexts.
  • Explain the system’s purpose, boundaries, and what it can and cannot do.
  • Surface limitations and uncertainty so users can calibrate trust appropriately.
  • Ensure downstream recipients can recognize AI‑generated outputs or actions and understand their provenance.

Systemic security and governance risks

Agent hijacking

Agent hijacking occurs when malicious or untrusted inputs manipulate agent reasoning or tool execution. In agentic systems, ambiguous separation between data and instructions can allow cross‑prompt injection attacks to redirect tool calls or workflows.

To manage the risk of agent hijacking:

  • Treat all external inputs (including retrieved content and tool outputs) as untrusted by default.
  • Enforce strict separation between instructions, data, memory, and tool parameters.
  • Filter inputs to detect and block malicious patterns before they reach agent reasoning or tool execution paths.
  • Implement allowlist tools and validate parameters deterministically before execution.
  • Minimize implicit instruction-following by grounding agent behavior in explicit, system-defined rules rather than inferred intent.

Sensitive data leakage

Sensitive data leakage occurs when confidential, proprietary, or personal information is exposed through outputs, logs, memory, or downstream actions. Risk increases when agents aggregate across multiple sources or retain long‑lived context.

To manage the risk of sensitive data leakage:

  • Apply least privilege to agent identities and data sources, grant access only for the current task.
  • Classify and govern sensitive data, and enforce deterministic rules for use, retention, and output.
  • Limit long-lived memory and persist only what is necessary and explicitly governed.
  • Monitor and filter outputs and logs to detect and prevent unauthorized disclosure.

Supply chain compromise

Supply chain compromise occurs when vulnerabilities are introduced through models, tools, plugins, grounding data, or other dependencies. Weakness in any component can propagate into autonomous decision‑making and execution.

To mitigate supply chain risk:

  • Inventory all models, tools, plugins, and data sources used by agents and review them as part of the security boundary.
  • Apply versioning and change control so updates are deliberate and reviewable.
  • Isolate components to reduce blast radius and prevent cascading failures.
  • Monitor for anomalies that could indicate dependency compromise or data poisoning.
  • Assume that individual components may fail and design compensating controls accordingly.

Agent sprawl

Agent sprawl is the uncontrolled proliferation of unmanaged or over‑permissioned agents. Sprawl expands attack surface, weakens least-privilege, and reduces accountability and IT oversight.

To mitigate agent sprawl:

  • Inventory all models, tools, plugins, and data sources used by agents and review them as part of the security boundary.
  • Establish clear ownership and accountability for every agent, including a responsible team or individual.
  • Enforce agent lifecycle governance, including registration, approval, expiration, and decommissioning.
  • Apply least privilege by default, granting each agent only the minimum permissions, tools, and data access required for its role.
  • Assign unique, auditable identities to agents to enable authorization, policy enforcement, and traceability.

Guidance

Organizations seeking to adopt this pattern can apply the following actionable practices.

Practice category Recommended actions Resource
Shared responsibility Human oversight enables organizations to remain accountable for agent behvior Artificial intelligence (AI) shared responsibility model
Model choices Model selection is a baseline control and a key supply-chain decision in agentic systems. Intentional model choices unlock safer, smarter agents Microsoft Foundry Model Catalog
Content safety & task adherence Detect and block malicious or manipulative inputs, include indirect prompt injection attacks Microsoft Foundry Risk & Safety Evaluators
Abuse monitoring Monitor for misuse patterns, repeated bypass attempts, or anomalous agent behavior Microsoft Foundry Azure OpenAI Abuse Monitoring
Agent identity Enforce least privilege, isolation, lifecycle management, and auditability to prevent agent sprawl Microsoft Entra Agent ID
Dependency governance Inventory, validate, version, and monitor models, tools, plugins, and data sources used by agents Microsoft Foundry Model Catalog
Human‑centered design Enable user understanding of the agent's capabilities and limitations, human oversight, and reduced misuse and overreliance Secure by Design UX Toolkit

Outcomes

Benefits

  • Agents execute only within defined intent, permissions, and boundaries.
  • Users can review, approve, and interrupt high-risk actions.
  • System behavior is observable and auditable through clear plans, feedback, and logs.
  • Sensitive data exposure is reduced through least privilege, governance, and monitoring.
  • Organizations maintain visibility and control as agent usage scales across teams and tools.
  • Users build and maintain trust in the system’s behavior.

Trade‑offs

  • Additional design and engineering effort is required to build deterministic safeguards, oversight, and logging.
  • Multi agent systems increase complexity and multiply opportunities for unexpected interactions and outcomes.
  • Clear disclosure and intelligibility require intentional UX planning and may add friction to workflows.

Key success factors

  • Task adherence: The agent executes actions as intended.
  • Human involvement: Humans remain accountable for high impact or ambiguous agent actions.
  • Deterministic safeguards: Prohibited actions are blocked reliably regardless of model behavior.
  • Transparency and disclosure: Users and downstream recipients understand when agents act and what they used.
  • Agent Hijacking: Agents have layered defenses to mitigate against Indirect Prompt Injection, are monitored for incidents, and are set-up for safe shutdown.
  • Least privilege and governance: Agent identities, permissions, and lifecycles are managed to prevent sprawl.
  • Supply chain awareness: Models, tools, and data sources are treated as security dependencies.

Summary

Autonomous agentic AI systems expand what AI-enabled software can do, but their autonomy amplifies risk. Foundational design pillars – task adherence, human oversight, system intelligibility, and disclosure – help keep agents aligned with intent and users in control. Systemic risks such as agent hijacking, sensitive data leakage, supply chain compromise, and agent sprawl require targeted mitigations grounded in least privilege, deterministic guardrails, governance, and monitoring. With layered defenses and clear accountability, organizations can scale agentic systems that are autonomous, observable, and resilient by design.

  • Last updated on