Communicate agent value with telemetry
Monitoring data becomes more useful when it shows stakeholders what an agent achieves for the business. In this unit, you learn how to build a value story that links an agent's usage, quality, and cost signals to business outcomes.
Separate activity from value
Session and user counts show that people use an agent. Usage alone isn't the same as value. To connect usage to value, trace every key performance indicator (KPI) back to a value driver the business cares about.
Business value from agents typically falls into four value drivers. Each driver names one kind of value an agent can deliver:
| Value driver | What it measures |
|---|---|
| Efficiency | Productive hours a team gets back and reinvests in higher-value work |
| Quality | Error reduction, consistency, and compliance |
| Revenue | Top-line lift from retained, expanded, or new business |
| Strategic | Decision velocity, employee confidence, optionality, and resilience |
Start a value story by naming the one or two outcomes the agent moves. Then assign each outcome to the value driver it serves. For example, an agent that answers employee policy questions mainly serves efficiency. If its answers are also more consistent than the manual process they replace, the agent serves quality as well.
Base the story on measured evidence. A claim built on theoretical time savings alone undermines the story's credibility.
Learn more in Measure the impact of your agents.
Build a chain of evidence to business outcomes
A defensible value story connects three links, from adoption through operational KPIs to business outcomes:
- Adoption: People use the agent for the work it supports.
- Operational KPIs: The agent completes that work reliably and to users' satisfaction.
- Business outcomes: The work changes a measure the business already tracks, such as time per task, error rate, or cost per interaction.
Adoption and operational KPIs show that people use the agent and that it performs well. The business outcome link is the one that shows the agent's effect on the business.
Business outcomes need a starting point. Before the agent goes live, document the current state of the process it supports. Three numbers captured before launch carry the measurement story: time per task, error rate, and cost per interaction. These numbers describe the business process itself, so they come from your organization's own data, not from the agent.
To track progress against that baseline, include at least one indicator of each kind for every value driver you report:
- Leading indicators: Show early whether you're on track, while there's still time to adjust. A falling thumbs-down rate is a leading indicator for quality.
- Lagging indicators: Confirm outcomes after they happen. A lower error rate compared with the baseline is a lagging indicator for quality.
Map the story to your agent's signals
Each part of the value story draws on a specific source. The following table maps each part to the signals available for your agent.
| Part of the story | Signals | Where to find them |
|---|---|---|
| Adoption | Total sessions, total users, and autonomous runs | Overview on the Monitor tab |
| Operational KPIs | Success rate, fail rate, and average duration | Overview on the Monitor tab |
| Quality and user voice | Thumbs-up and thumbs-down reactions, and the comments users add | See details on the Reactions card |
| Cost | Total estimated credits used, and the breakdown by experience when available | The Total estimated credits used card on the Monitor tab |
| Process-level evidence | Your own measures, such as tool failure rate or tool call duration | Environment-level telemetry (preview) in Application Insights |
| Business outcomes | Time per task, error rate, cost per interaction, and revenue measures | Your organization's own business data |
Credits form the cost side of the story. The credit total on the Monitor tab is the final, source-of-truth number used for billing. Evaluations (preview) and testing on the Preview tab also consume credits, so report the production share from the breakdown when one is available. Report that consumption next to the sessions and autonomous runs in the same period.
Environment-level telemetry adds process-level evidence when your administrator sets it up for the environment. Queries over its spans can measure the steps the agent performs. For example, a tool failure rate by tool name shows whether a step that the agent automates runs reliably.
Qualitative signals show how people experience the agent. Reactions and comments show user confidence in the agent's output, especially where that output feeds a decision. For efficiency, stories from managers about reclaimed capacity add evidence of how their teams reinvest the time. Structured interviews with a few frontline managers each quarter are one way to collect these stories.
Present the value story to stakeholders
A scorecard turns metrics into one story on one page, in the language your stakeholders use. An executive scorecard for a quarterly value review uses these sections:
| Section | What it contains |
|---|---|
| Headline | One sentence that summarizes the quarter's outcome and the value your agents delivered |
| Portfolio snapshot | Live, piloting, and retired agents, and the routine adoption rate across them |
| Value delivered | Assisted hours and value by value driver, with the change from the previous quarter and from the baseline |
| Agent scorecard | The top five agents by value, and the bottom two with a keep, fix, or retire decision |
| Voice of the user | Two or three verbatim quotes, and the satisfaction and negative-sentiment trends |
| Risk and governance | Compliance cases, Responsible AI reviews, incidents, and open audit findings |
| Next quarter | Two or three candidate use cases, and the one workflow redesign you plan to ship |
The agent you monitor contributes evidence to several of these sections. Its business outcome movement against the baseline feeds Value delivered. Its reactions and comments supply the quotes and the negative-reaction trend for Voice of the user. Its operational KPIs support the keep, fix, or retire decision in Agent scorecard.
Set expectations about when each kind of value appears. In the first 90 days, value comes mainly from adoption and early efficiency gains. Quality and revenue value emerge between day 91 and the end of the first year. Strategic value, such as redesigned workflows, builds from the second year onward.
Make measurement a recurring practice. Schedule the 90-day review before the agent goes live. At 30 and 90 days, compare movement against the baseline, and name the value driver in every review. When a review shows a weak operational KPI, the sessions and transcripts behind it are where your diagnosis starts.
Learn more in Communicate the value story to stakeholders.
Reflect: For an agent you manage or plan to publish, which business outcome would show its value? Which signal from the agent would you pair with that outcome as a leading indicator?