Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
Note
This article describes features used in agents or agent flows powered by the standard harness.
[This article is prerelease documentation and is subject to change.]
Use custom metrics to measure outcomes that are specific to your agent and business scenario. Describe what you want to measure in natural language, and Copilot Studio generates the metric configuration and evaluates it against your agent's conversation sessions.
You can create up to three custom metrics for an agent. Custom metrics support conversational agents that use classic or generative orchestration. They don't support autonomous agents or agent flows.
Important
This article contains Microsoft Copilot Studio preview documentation and is subject to change.
Preview features aren't meant for production use and may have restricted functionality. These features are available before an official release so that you can get early access and provide feedback.
If you're building a production-ready agent, see Microsoft Copilot Studio Overview.
Prerequisites
Before you create a custom metric:
- Configure and publish your agent.
- Ensure your agent has conversation sessions available in Monitor. A meaningful preview might not be available when the agent has insufficient session data.
- Define a metric that can be evaluated from the messages exchanged during a session. Custom metrics don't verify outcomes that occur outside the conversation, such as a transaction completed later in an external system.
You need the Bot Transcript Viewer security role to open the session messages used in preview and calculated results. Learn more in Share an agent's analytics.
Custom metric types
Copilot Studio determines the metric type from your description. Each custom metric produces one result for each applicable session.
| Metric type | Description |
|---|---|
| Binary classification | Groups sessions into a defined outcome and the default Other category. |
| Multiclass classification | Groups sessions into generated categories. A metric can have up to 10 categories. Sessions that don't match a generated category are grouped under Other. |
| Numeric score | Assigns a numeric score to each applicable session and calculates the average score. A null result means that the session has no data for the metric; it doesn't represent a score of zero. |
Start from an example
Copilot Studio provides examples that you can use as a starting point. When you select an example, change its description to match the signals and outcomes in your agent's conversations.
| Example | Metric type | Measurement |
|---|---|---|
| Follow-up Needed | Binary classification | Determines whether the conversation creates a later action, such as a callback, appointment, investigation, or escalation. |
| Business Outcome Type | Multiclass classification | Groups the final meaningful outcome as Completed Action, Follow-up Required, Information Provided, or No Business Outcome. |
| Lead Quality Score | Numeric score | Assigns a score from 1 through 3 based on the customer's need, fit, urgency, and intent. Sessions without relevant business or commercial context return no data. |
Create a custom metric
On your agent's Monitor page, select Custom metrics above the Summary card.
In Create a custom metric, start with a provided example or create a metric from scratch.
Learn more in Start from an example.
Describe what you want to measure across your agent's conversations.
Define a specific, observable result. For example, describe the signals that indicate whether a conversation requires a follow-up instead of entering a broad goal such as "measure success."
Select Next.
Copilot Studio uses your description, agent information, and sample sessions to:
- Validate whether the metric can be calculated from available session data.
- Generate a metric title.
- Determine whether the metric is a binary classification, multiclass classification, or numeric score.
- Generate categories and their criteria for classification metrics.
- Generate the scoring criteria for a numeric metric.
- Generate a read-only calculation prompt.
- Select a compatible visualization.
If the metric isn't supported, Copilot Studio explains why and provides guidance for refining its description. Your original description remains available for you to edit.
Preview and review the generated metric. Learn more in Preview a custom metric.
Select Save metric.
The custom metric appears in the Custom metrics section of the Monitor page. Copilot Studio calculates the metric for applicable sessions from the preceding seven days. After the initial calculation, Copilot Studio calculates the metric daily for new sessions.
Preview a custom metric
Preview uses a sample of your agent's sessions to help you evaluate the metric before you save it. The sample can include sessions from multiple days so you can review aggregate results and trends. Preview results don't represent calculations for all sessions.
In Create a custom metric, select Next to validate and generate the metric.
Review the aggregated result and generated visualization in Preview.
Select Results to review representative sessions, the result assigned to each session, and the AI-generated reasoning for that result.
Select a session to review its messages.
Note
You need the Bot Transcript Viewer security role to review session messages.
Select Prompt to review how Copilot Studio interprets your metric description and calculates its results.
The generated calculation prompt is read-only.
If the results don't match your intent, return to the metric description, refine it, and preview the metric again.
When the preview produces the expected results, select Save metric.
Important
AI-generated metric configuration, classifications, scores, and reasoning can be inaccurate. Review the preview results and reasoning before you save a metric. Treat a custom metric as an analysis of conversation data, not as an authoritative record of an outcome that occurs outside the conversation.
Understand custom metric calculations
Preview and saved metrics use session data differently:
| Calculation | Session data used |
|---|---|
| Preview | A sample of historical sessions used to validate and test the metric. |
| Initial saved calculation | All applicable sessions from the preceding seven days. |
| Daily calculation | All applicable new sessions available for the daily calculation. |
Custom metrics evaluate the conversation messages in a session, including content sent in Adaptive Card messages. They don't evaluate tool invocation details or information stored in external systems.
The visualization depends on the type of metric. Classification metrics show the distribution of sessions across categories. Numeric metrics show the average score for applicable sessions. Calculated results also show the number of sessions used and the metric's trend.
Edit a custom metric
Important
Saving changes recreates the custom metric and deletes its previous calculation history. Copilot Studio then recalculates the metric for applicable sessions from the preceding seven days.
On your agent's Monitor page, select the three dots (…) on the custom metric that you want to edit, and then select Edit.
Update the metric description.
Select Next to regenerate and preview the metric.
Review the generated configuration, results, reasoning, and calculation prompt.
Select Save metric.
Edit a metric created in the previous experience
When you edit a metric that you created in the previous custom metrics experience, Copilot Studio consolidates the original description, category names, and category criteria into the metric description field. Review the consolidated description to ensure that it preserves the metric's intent.
Saving the edited metric recreates it by using the new custom metrics experience and deletes the previous calculation history.
Delete a custom metric
On your agent's Monitor page, select the three dots (…) on the custom metric that you want to delete, and then select Delete.
In the confirmation dialog, select Delete.
Limitations
- You can create up to three custom metrics for an agent.
- Custom metrics support conversational agents that use classic or generative orchestration. Autonomous agents and agent flows aren't supported.
- A custom metric produces one result for each applicable session.
- Numeric metrics support scoring with average aggregation. They don't extract and aggregate transaction amounts or other arbitrary numeric values.
- Custom metrics use conversation messages, including Adaptive Card message content. Tool invocation details and data from external systems aren't included.
- Preview uses sample sessions. Saved calculations use all applicable sessions available for the calculation period.
- A metric might not produce meaningful results when its description is vague or when the available sessions don't contain the information needed to calculate it.
- Individual classifications, scores, and reasoning can be inaccurate. Review results in aggregate and refine the metric description when the preview doesn't match your intended measurement.
- If you make substantive changes to your agent's instructions or configuration, review its custom metrics to ensure that they still measure the intended outcome.