Share via


Tune your knowledge sources (preview)

Adding knowledge sources enables you to provide business specific data which augments the responses of your agents. A key to improving actions is understanding how and when knowledge sources were used. Additionally, providing context or scoping to the knowledge sources can enhance the reliability of answers provided by your agent.

Screenshot of the Knowledge page and the three tuning options for knowledge source improvements.

Use knowledge source suggestions

To access a list of suggested knowledge sources, select View suggestions on the Knowledge page.

Screenshot of the Suggested knowledge dialog.

From here, you can discover, search, and add sources:

  • Used in one of your previous agents
  • Used in agents shared with you
  • Used previously while working with Office products

The list contains the 100 most recently used knowledge sources.

Knowledge analytics

On the Knowledge page, you can view analytic summaries about how the knowledge sources are used in your agent, including the following key metrics:

  • Total sources used: The percentage of engaged sessions that used at least one knowledge source.
  • Error rate: The percentage of engaged sessions that produced a knowledge source-related error.
  • Total answer rate: The percentage of engaged sessions that your agent provided an answer to a user.

In addition, you can get advanced metrics by navigating to the Analytics page and viewing the Knowledge source use section.

Screenshot of the knowledge analytics graph.

Note

It can take up to one hour before questions from users are reflected in the metrics.

Improve answer rates

On the Knowledge page, you can access a fine-tuning tool, which allows you to define, in natural language, how your Dataverse table should be used.

Screenshot of the Fine-tune your knowledge sources dialog.

On this page, you can review recent unanswered questions to better understand what needs to be updated. This process includes defining the precise attributes involved or adding instructions to shape responses when answering a specific question.

Note

The agent uses your custom knowledge source name to identify when to use that knowledge source. A topic is paired with this custom knowledge source, with the paired topic accessible on the Overview page or the Topics page. The information on this page allows you to fine-tune the trigger phrases of the paired topic to influence how it's targeted to certain types of questions. By default, we generate trigger phrases automatically based on the name of your custom Dataverse knowledge source and tables used by it.

Example

Your agent uses Dataverse tables as knowledge sources. These tables contain the following:

  • reservations for students' accommodation
  • the students' individual information
  • the facilities booked
  • the price category for each

The answers to questions about reservations must always consider the date, period, student name, facility name, city, and the price calculated based on the facility size.

You can enter the following instructions to define this business logic:

Assess availabilities looking into [Reservation.Period], [Reservation.Date], [Reservation.Accommodation (Facility).Name], [Reservation.Facility (Facility).City], and [Reservation.Student (Individual).Name].

Retrieve the [Category.Price] corresponding to the [Reservation.Facility (Facility).Size] in range of [Category.Min Size] and [Category.Max Size].

Official sources

When adding knowledge sources to your agent, you might not always control how the information evolves over time, or you might not fully trust this information. It's important to let your users know that they should consider answers with caution, and they should verify them when appropriate.

However, when you know that information from a specific a knowledge source goes through a strict verification process and is highly trusted, you can mark this knowledge source as an official source that can be used directly, without verification.

To mark a knowledge source as official, on the Knowledge page, select the menu icon (⋮) for the knowledge source, point to Official source and select Yes.

Note

This feature is not yet compatible with generative orchestration. If you want your agent to use official knowledge sources and mark them as such, turn off generative orchestration.

When an agent uses authoritative knowledge sources, the response starts with a distinctive indication.