Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
Important
Items marked preview in this article are currently in preview. This preview is provided without a service-level agreement, and Microsoft doesn't recommend it for production workloads. Certain features might not be supported or might have constrained capabilities. For more information, see Supplemental Terms of Use for Microsoft Azure Previews.
Annotate production traces to record what your agent did well, what it got wrong, and what it should have done instead. These annotations become the expected outputs and task-level criteria in Human in the loop evaluation datasets.
Prerequisites
- A Microsoft Foundry project with an agent that emits traces.
- Tracing configured with a connected Application Insights resource. See Set up tracing for your agent.
- Permissions to view and annotate traces. See Set up permissions for evaluation workflows.
Traces and annotations can contain sensitive information. Review your privacy, retention, and access requirements before you start.
Find traces to review
- In the Foundry portal, open your project and agent, and then open Traces.
- Choose a view:
- Trace view for single responses.
- Conversation view for multi-turn tasks, where the whole conversation is the test case.
- Use filters to narrow the list to your evaluation goal, such as a specific task, a recurring failure, or an edge case your current dataset doesn't cover.
Include successful interactions as well as failures, so your dataset shows both expected behavior and areas to improve.
Add annotations
Open a trace or conversation, and add any of the following annotations:
| Annotation | Purpose | Example |
|---|---|---|
| Rating (thumbs up or down) | Record an overall judgment. | Thumbs down for an incomplete answer. |
| Tags | Categorize the interaction for curation and analysis. | Failure category: missing-policy-detail |
| Comments | Explain the judgment and what should improve. | The response explained how to request a refund but omitted the 30-day deadline. |
| Expected output | Describe the correct response or behavior. | Explain the refund deadline and how to submit a request. |
For tags, choose a standard category such as Task outcome or Failure category, or create your own category. You can also add free-form tags.
For thumbs-up and thumbs-down basics, end-user feedback, and querying annotations, see Annotate traces with human feedback.
Write useful feedback
A rating alone doesn't explain what to fix. For the most useful test cases:
- Describe the missing or incorrect behavior.
- Explain why it matters.
- Provide an expected output when you can.
The expected output becomes the reference answer for the test case. Comments and other feedback help Foundry generate task-level criteria.