Transparency note: Real-time agents

What is a Transparency Note?

An AI system includes not only the technology, but also the people who will use it, the people who will be affected by it, and the environment in which it is deployed. Creating a system that is fit for its intended purpose requires an understanding of how the technology works, what its capabilities and limitations are, and how to achieve the best performance. Microsoft’s Transparency Notes are intended to help you understand how our AI technology works, the choices system owners can make that influence system performance and behavior, and the importance of thinking about the whole system, including the technology, the people, and the environment. You can use Transparency Notes when developing or deploying your own system, or share them with the people who will use or be affected by your system.

Microsoft’s Transparency Notes are part of a broader effort at Microsoft to put our AI Principles into practice. Learn more in Microsoft AI principles.

The basics of real-time agents

Introduction

Real-time agents in Dynamics 365 Contact Center enable organizations to create conversational, low-latency voice experiences for digital messaging and voice calling experiences. You author these agents in Microsoft Copilot Studio and deploy them through Dynamics 365 Contact Center. Callers and end customers can chat or speak naturally and receive spoken responses in real time.

Unlike traditional interactive voice response (IVR) or virtual agent (VA) systems that rely on separate steps for speech recognition, language understanding, and text to speech, real-time agents use a real-time speech-to-speech architecture. This approach streams audio input and output through a real-time model, reducing latency and enabling more fluid, conversational interactions. The system takes caller audio as input and produces spoken responses as output, while leveraging Copilot Studio for agent instructions, tools, and knowledge configuration.

Key terms

Microsoft Copilot Studio: The authoring environment where makers define agent behavior, instructions, tools, and knowledge sources.

Dynamics 365 Contact Center: Provides telephony integration and orchestration for voice calls and connects callers to real-time voice agents.

Real-time agent: An AI‑powered agent that conducts speech‑to‑speech conversations with customers in real time.

Real-time multimodal model: The model that processes streaming audio and text input and generates streaming responses. The model used for this release is the Azure Foundry GPT real-time Model or GPT Realtime Mini.

Tool/tool call: An action invoked by the agent (such as a workflow or connector) to retrieve or update information during a call.

Topics: Deterministic dialog flows that can be used for specific system or guided interaction scenarios.

Capabilities

System behavior

Real-time agents support natural, conversational interactions. During a call, the telephony layer streams caller audio to the real-time model, which interprets intent and generates spoken responses. During a chat, the agent receives the text, interprets it, and responds in a similar fashion. Copilot Studio provides the agent’s configuration, including instructions that guide behavior and tools that allow the agent to take action or retrieve information.

Depending on configuration, the system can support IVR‑style capabilities such as call escalation, handling of key pad input, and silence detection. These capabilities allow organizations to combine conversational AI with structured contact center workflows where appropriate.

Use cases

Intended uses

Use real-time agents in Dynamics 365 Contact Center across a range of customer service scenarios. The system supports routine, repeatable interactions, while enabling escalation to human agents when requests become complex, sensitive, or high‑risk.

• Self‑service and knowledge‑based assistance: The agent answers common questions and provides guidance using configured knowledge sources, enabling callers to resolve routine requests quickly without human involvement (for example, order status and returns in retail, policy FAQs in insurance, or hours and eligibility questions in public services).

• Automated task completion through tools and workflows: The agent performs structured actions such as creating or updating records, triggering workflows, or collecting required information by invoking configured tools (for example, resetting credentials, scheduling an appointment, creating a service ticket, or sending a confirmation message).

• Transfer and escalation to human agents: The agent transfers calls or chats to human agents when requests exceed its capabilities, require additional review, or need personalized assistance. This transfer ensures continuity of the customer experience. Examples include complex billing issues, account disputes, or case‑specific inquiries.

• Multilingual interactions: The agent understands and responds in multiple languages, subject to supported language availability. This capability enables interactions across diverse caller populations. Examples include supporting callers in different regions or offering multilingual access to customer support lines.

Considerations when choosing other use cases

Leverage real-time agents in Dynamics 365 Contact Center in innovative solutions and applications. When selecting a use case, consider the following factors:

• Scope and structure of interactions: The system is optimized for interactions where guidance can be anchored in configured knowledge, tools, workflows, or topics, while still allowing flexibility for more complex or specialized conversations when properly authored.

• User experience expectations: Organizations should consider how conversational interactions are combined with structured flows, including how prompts, confirmations, and handoffs are presented to callers to ensure a clear and predictable experience.

• Latency and responsiveness trade‑offs: Additional safeguards, validation steps, or processing might introduce latency. This latency can affect responsiveness in real‑time scenarios. Evaluate this factor during design and testing.

• Language, regional availability, and deployment constraints: Multilingual experiences, regional availability, and data residency requirements may influence which use cases are appropriate and how solutions are deployed.

• Legal and regulatory considerations. Organizations need to evaluate potential specific legal and regulatory obligations when using any AI services and solutions, which may not be appropriate for use in every industry or scenario. Restrictions may vary based on regional or local regulatory requirements. Additionally, AI services or solutions are not designed for and may not be used in ways prohibited in applicable terms of service and relevant codes of conduct.

Limitations

This section describes the technical, behavioral, operational, and safety‑related factors that influence how Realtime agents behave in real‑world deployments. These limitations help system owners understand where the agent might produce unpredictable results, what risks require human oversight, and where additional safeguards or configuration choices might be needed. Some of these limitations reflect fundamental constraints of current AI systems and aren't fully mitigable through configuration alone.

Technical limitations, operational factors and ranges

Technology‑related limitations

Voice type configuration constraints

Real-time agents have a one‑time configuration setting. Switching back to Classic requires creating a new agent. Scenarios that require frequent voice‑type changes might experience reduced flexibility.

Static system messaging requirements

Certain system messages and instruction patterns must remain static to preserve performance, reliability, and privacy guarantees. Workflows dependent on frequently changing or fully dynamic prompts may not behave as intended.

Sensitive data handling in real‑time audio

Comprehensive protection of sensitive data in streamed speech‑to‑speech interactions may be limited depending on release stage and configuration. Scenarios requiring strict, deterministic filtering or masking of sensitive content may face constraints.

Tool invocation reliability

Tool calls may fail, time out, or return incomplete results. Real‑time workflows should include fallback behaviors, retries, or escalation pathways if tools do not respond as expected.

Model behavior variance

Real-time language model outputs can vary across turns based on timing, barge‑in events, and context shifts. Highly deterministic flows should use topics, structured prompts, or tool‑driven logic instead of relying solely on generative responses.

Turn‑taking and interruption sensitivity

Latency, network variability, and device performance can affect interruption timing and barge‑in behavior. Scenarios requiring strict turn ordering or tightly bounded responses may require tuning.

Multilingual consistency limitations

Languages that haven't yet undergone full evaluation may exhibit reduced accuracy or inconsistent behavior compared to fully validated languages. Model capability does not imply general availability readiness.

Context retention boundaries

Very long or highly stateful conversations may exceed optimal memory retention windows, requiring explicit resets or topic boundaries to maintain predictable behavior.

Concurrency and GPU capacity constraints

Real-time performance is constrained by GPU availability within each region. Peak concurrency may be limited in geographies without dedicated model hosting capacity.

Safety‑relevant behavioral limitations

Risk of unauthorized commitments or confident hallucinations

Generative models may produce confident but incorrect confirmations (for example, approving donations, inferring inventory, or fabricating “offline approvals”). To mitigate this risk, customers should implement human review and approval for any commitments, compliance‑sensitive, or financial actions, and enforce these workflows through deterministic controls and escalation paths.

Risk of information over‑disclosure

When provided access to unscoped knowledge content, the agent might disclose information beyond what was requested, like providing all items in a file instead of limiting responses to the queried record. Guardrails depend heavily on the structure and clarity of customer‑authored instructions and knowledge boundaries. To reduce the risk of unintended or potentially harmful outputs, limit the use of general knowledge for real-time agents and explicitly define clear instructions, scoped knowledge boundaries, and allowed behaviors.

Inconsistent age appropriate responses

The agent might not reliably recognize contextual indicators of age or enforce age restricted safety guidelines. Attackers can override initial age disclosures by simply stating a different age later. This reflects model level limitations that customer should mitigate by employing external verification techniques and enforcing appropriate policies.

System instructions may not meaningfully constrain behavior

Changes to system prompts alone may not be sufficient to prevent undesired behavior in all scenarios. To reduce risk, customers should limit general knowledge, explicitly define allowed and disallowed behaviors in their instructions, and use deterministic controls for safety critical actions rather than relying solely on free form model responses.

Unresponsiveness during safety filter activation

When content filters are triggered (including self-harm scenarios), the agent may stall or loop canned messages. Customers should mitigate this by (1) defining explicit escalation triggers for unresponsiveness and safety critical intents, and (2) ensuring a reliable service representative handoff path is available to continue support without delay.

Operational Factors and Performance Considerations

Latency and responsiveness tradeoffs

Safety validation steps, network conditions, or multiple tool calls may introduce latency. Real time use cases requiring immediate responses should be tested under production like conditions.

Regional availability and infrastructure differences

Model hosting, telephony infrastructure, and tool execution vary across regions. Regional GPU capacity and deployment availability may affect responsiveness, multilingual support, or quality.

Identity and context persistence constraints

Where identity persistence is restricted for privacy reasons, scenarios that assume stable caller identity or long-lived context may not behave predictably.

Impact across user populations

Differences in accents, dialects, speech patterns, background noise, or language proficiency can affect recognition accuracy and interaction quality. Assess accessibility and equity considerations during deployment.

Human and organizational limitations

Over reliance on AI responses

Users may overtrust the system’s authoritative tone even when it is uncertain or hallucinating. Generative outputs should be reviewed for accuracy in workflows involving compliance, commitments, or operational risk.

Variability in authoring quality

Reliance on makers to write precise instructions, topics, and tool logic means incorrect or incomplete configuration may surface as agent errors, unsafe behaviors, or unintended disclosures.

System performance

For real-time agents, performance refers to how accurately, reliably, and responsively the system supports interactions across common use cases such as self service, knowledge retrieval, tool based task completion, and contact escalation. Performance is influenced by speech recognition quality, model response accuracy, tool execution reliability, and end to end latency.

Because the system operates in real time, performance should be evaluated holistically across the full interaction rather than by any single component. Performance metrics and evaluation signals

Common metrics used to assess system performance include:

• Speech recognition accuracy: How accurately caller speech is transcribed across accents, speaking styles, background noise levels, and supported languages.
• Intent or topic matching accuracy: How often the system correctly identifies when a request can be handled through knowledge responses, tools, or escalation.
• Task completion success rate: Whether configured tools or workflows execute successfully and produce the intended outcome.
• Latency and responsiveness: The time between end user input and system response, including delays introduced by safeguards or tool calls.
• Escalation accuracy: How reliably the system transfers interactions to service representative when required.

Evaluation might include scripted scenario testing, simulated calls, and analysis of production telemetry. Results might vary by language, region, and configuration. Like all AI systems, real-time agents can produce errors. These errors might surface as misunderstandings of caller intent, incomplete answers, incorrect tool invocation, or missed escalation opportunities. Designing for errors - by including confirmations, fallbacks, and service representative handoff - is an essential part of deploying the system responsibly.

The following table provides illustrative examples using common scenarios.

Outcome type	Definition	Example
True positive	The system correctly handles a request it was designed to support	A customer asks for order status, and the agent retrieves the correct information from a configured system
False positive	The system attempts to handle a request it should have escalated	A customer describes a complex billing dispute, and the agent responds with a generic answer instead of transferring
True negative	The system correctly escalates a request beyond its scope	A customer asks for account changes requiring manual review, and the agent transfers to a service representative
False negative	The system escalates a request it could have handled	A ccustomer asks a common FAQ question, but the agent transfers instead of answering from knowledge sources

Best practices for improving system performance

The following practices can help improve reliability, manage error conditions, and support a consistent experience across different use cases:

• Design for failure and fallbacks: Include user friendly recovery behaviors when a tool call fails or times out, such as asking the caller to repeat information, offering alternative paths, or escalating to a service representative.
• Keep latency sensitive messages simple: Where greeting or first turn experiences are cached or optimized, prefer static content and avoid dynamic variable references that may introduce latency, privacy, or caching risks.
• Be deliberate about deterministic flows: When using deterministic topic flows, test how interruptions, transitions, and retries affect responsiveness and the overall user experience, especially in real-time interactions.
• Validate across real operating environments: Test the system under expected conditions, including telephony infrastructure, end customer applications and digital messaging channels, accents, background noise, call quality, and end-to-end business workflows, to reduce unexpected behavior at launch.
• Tune escalation behavior intentionally: Different use cases may benefit from earlier or later escalation to human agents; adjusting this balance can affect false positives and false negatives and should be evaluated carefully.
• Monitor and iterate post deployment: Continuously review performance signals such as latency, tool success rates, and escalation frequency to identify areas for refinement and improvement.

Evaluation of real-time agents

Evaluation methods

Evaluation Methodology

The evaluation framework leverages an LLM-as-a-judge approach, where an independent AI model (e.g., GPT-4.1-mini and GPT-audio) evaluates the agent’s responses against predefined quality metrics (Interruption, Missed-window, Latency, Audio language model tone, Intent determination, Intent resolution, Acknowledgement).

To ensure broad and representative coverage, we define a diverse set of evaluation scenarios spanning both positive and edge cases, closely mirroring real-world customer interactions. Synthetic conversations are generated using the customer simulator to emulate these scenarios at scale.

Run test prompts against the real-time agent, and the system captures audio recordings and recording transcripts.

Evaluate each interaction across the following quality dimensions: Interruption, Missed-window, Latency, Audio LLM tone, Intent determination, Intent resolution, Acknowledgement. Perform Responsible AI evaluations as well. Conduct evaluations in:

• Single-turn mode – assessing isolated prompt–response pairs
• Multi-turn mode – evaluating full conversational context

Scoring is aggregated using configurable thresholds to determine pass or fail outcomes. Results are exported in CSV format and published to the analytics dashboard for centralized reporting and analysis.

Dataset and environment characteristics

The evaluation dataset comprises curated test prompts organized by scenario, designed to closely mirror real-world customer interactions. Each scenario represents a realistic customer journey, including both standard (positive) flows and edge-case conditions.

For evaluation, the customer simulator receives each scenario and conducts a dynamic conversation with the real-time agent under evaluation. This setup enables end-to-end testing of conversational behavior in a controlled yet realistic environment.

The dataset is intentionally constructed to ensure comprehensive coverage, including:

• Expected or ideal interaction paths
• Error handling and recovery flows
• Edge cases and boundary conditions
• Variations in intent clarity and conversational complexity

This approach ensures that the evaluation environment reflects production-like conditions while maintaining reproducibility and consistency across test runs.

Evaluation results

Evaluate real-time agents by using a combination of automated evaluations and human review to assess conversational quality, latency, and expected behavior across common contact center scenarios. These evaluations validate fitness for purpose rather than guarantee outcomes in all production environments.

Automated evaluations are conducted using curated test scenarios and predefined quality criteria, focusing on areas such as response relevance, ambiguity handling, turn taking behavior, and tool invocation. These tests are primarily run in pre production environments with controlled inputs. As a result, they may not fully capture real world variability such as diverse accents, background noise, call center load, custom configurations, or integration differences across customer environments.

Use human evaluations and structured hands-on testing sessions to complement automated results. They particularly assess conversational naturalness, latency perception, and interruption behavior during live interactions. While this combined approach helps surface common issues and informs defaults and best practice guidance, it doesn't represent exhaustive testing of all possible customer use cases or operating conditions. The responsible AI assessment is also completed by testing the scenarios across harmful content categories for the contact center use cases. The system demonstrated harmful content mitigation through safety policies and guardrails.

Evaluation datasets are limited in scope and language coverage. They might emphasize representative enterprise scenarios rather than edge cases or highly specialized workflows. Additionally, some aspects of live interactions - such as subjective user perception, regional telephony behavior, and environmental conditions - can't be fully simulated in automated testing. For these reasons, customers should perform their own validation and testing in their target environments, especially for business critical scenarios, regulatory requirements, or region specific deployments.

Evaluating and integrating real-time agents for your use

For real-time agents, train and evaluate the system by using representative, real-world call and chat data. Iteratively tune prompts, confidence thresholds, and latency parameters. Validate performance across expected call and volumes, dialects, and accents. Vary system parameters such as response latency, interruption sensitivity, and confidence thresholds to improve naturalness and responsiveness. However, these changes might introduce tradeoffs between speed, accuracy, and conversational stability. For example, lower latency can feel more human but might increase the likelihood of incomplete or less accurate responses. Test these tradeoffs in your specific use cases to determine appropriate settings. Evaluation tools such as call and chat transcripts, confidence scores, and post call analytics can help identify failure patterns and inform tuning decisions. Appropriate human oversight is critical. Ensure operators understand the system’s intended use, how to interpret its responses, and when to intervene. For example, in sensitive customer service scenarios, use confidence scores or predefined escalation conditions to route calls to human agents when uncertainty is high. This approach helps mitigate automation bias and ensure reliable outcomes.

Learn more about responsible AI

Microsoft AI principles
Microsoft responsible AI resources
Microsoft Azure Learning courses on responsible AI

Learn more about real-time agents

Configure real-time agents