How Copilot for Microsoft 365 decides which plugin to use
Microsoft Copilot for Microsoft 365 is your personal assistant for work. It helps with various general tasks, such as writing, summarizing, researching, and more. Copilot has different skills that correspond to these different types of tasks. For example, Copilot can summarize action items from a meeting, suggest edits to a file, or track down resources and experts on a given topic within your organization. Each skill has its own parameters and outputs that are tailored to the specific task.
Like any copilot, Copilot for Microsoft 365 is trained with data at a point in time. To retrieve and process new and real-time information, especially data that's specific to your organization and workflows, Copilot requires plugins. Plugins extend Copilot for Microsoft 365's skills and utility for end users, enabling it to choose the right skill from thousands.
But how does Copilot know which skill to use when you ask for help? How does it interpret your request and match it to the best skill available? That's the job of the Copilot for Microsoft 365 orchestrator.
This article will explain the logic behind Copilot's skill selection process and how you can ensure Copilot employs your plugin at every opportunity it can benefit your users.
Important
- Plugins for Microsoft Copilot for Microsoft 365 are in public preview and only work in Microsoft 365 Chat in Microsoft Teams.
- The capability is enabled by default in all Microsoft 365 Copilot licensed tenants. Admins can disable this functionality on a user and group basis and control how individual plugins are approved for use, and which plugins are enabled. For details see: Manage Plugins for Copilot in Integrated Apps.
The Copilot orchestrator
Between the end-user's natural language input to Copilot's natural language output, the Microsoft Copilot orchestrator works behind the scenes to select and execute the right skill(s) from the right plugin(s) for the end-user's given task.
The orchestration layer represents the interface between foundation LLMs and the many ways you can extend, enrich, and customize Copilot for the way your customers work.
The following chart illustrates how the Copilot for Microsoft 365 orchestrator selects the right plugin, with the right skill, at the right time, even when there are thousands of options to choose from.
Natural language input
The user types a prompt to Copilot through Microsoft 365 chat.
Search for relevant tools
The Copilot orchestrator searches its tool catalog of installed and enabled plugins for an initial list of relevant skills.
Reasoning
Copilot draws upon the combined knowledge of its LLM and your organizational data from Microsoft Graph to analyze the request and determine the context of the user. It then breaks down the user's prompt into intents, or goals. Each goal is then broken into tasks.
Mapping user's intent to slots
The orchestrator maps the user's tasks to plugin skills and/or built-in Copilot skill functions, mapping implicit constraints from the user's prompt to the slots (parameter values) required to execute each of the identified skills.
Execute tool
Copilot and its plugins are called to execute their selected skills.
Generate summary
Copilot merges, filters, or ranks the responses from different assistants, and generates a single response for the user.
Natural language output
Finally, Copilot delivers the response to the user and updates the conversation state. Copilot is ready for its next prompt.
If you imagine a user's prompt to Copilot like a construction project, then the Copilot orchestrator is the general contractor, who coordinates and organizes the work of the specialist subcontractors, your plugins. Similar to a general contractor, the orchestrator is responsible for ensuring the project is "completed" according to specifications implied by the user's input (in other words, that Copilot's response satisfies the user's intent in their request).
However, its the responsibility of each plugin to provide Copilot with an accurate description of its skills and to execute its skills effectively. This will instill a sense of trust in your users and ensure Copilot will call your plugin each time its skills are needed. The next section provides more details on how to optimize your plugin for the orchestrator to find and use.
Plugin search optimization
Copilot for Microsoft 365 can uniquely choose the right skill from thousands. But how can you make sure Copilot will choose your plugin to provide the right skill?
The answer lies in how you describe your plugin, its skills, and the parameters for skill execution. Specifying concise and accurate descriptions in your plugin manifest is critical to ensuring Copilot knows when and how to invoke your plugin.
The following sections provide guidance and examples for plugins, skills, and parameter descriptions.
Plugin descriptions
Plugin descriptions should include the user actions, the types of input, and the types of output, for which your plugin is intended.
Long and short app descriptions in the app manifest must clearly define the scope of your app. To render an app as a plugin in Copilot, app description must be modified to suit the following plugin requirements:
- App long description must clearly explain how users can use a message extension plugin in Copilot and what functionality they can expect. For example, Use Contoso cloud in Copilot to search and summarize your tasks.
- Short description must briefly describe the app's functionality in a natural language and can include name of the app.
The following table lists the short description examples for various plugin scenarios:
Description: Create, search, view tickets, bugs, and projects.
App description example:
{
"$schema": "https://developer.microsoft.com/en-us/json-schemas/teams/v1.16/MicrosoftTeams.schema.json",
"version": "1.0.0",
"manifestVersion": "1.16",
"id": "2bxxxxc5-5xxx-4xxx-aXXX-94xxxx8919e5",
"name": {
"short": "Tasks",
"full": "Contoso Tasks"
},
"description": {
"short": "Create, search, view tickets, bugs, and projects",
"full": "Contoso Tasks makes it easy to stay organized. Create, assign, and track tasks individually or collaboratively with your team, and see everything come together in one place."
},
Skill descriptions
Skill descriptions are used to interpret the user's prompt into an intent and then map that to the most relevant skill available to Copilot. Include which action the skill handles, and also the expected format of input and description of the output.
Currently Teams message extension plugin support is limited to search commands, with support for action commands coming soon.
Search skill descriptions should:
- Focus on what and how the command searches (detailed list) in natural language.
- Include verbs and synonyms, if applicable.
- Focus on keywords that are likely to be used in the search function of your native apps.
The following table lists search command examples for various plugin scenarios:
Description: Search for high priority tasks related to Northwind that are due tomorrow.
Command description example:
"commands": [
{
"id": "Search",
"type": "query",
"title": "Tasks",
"description": "Search for high priority tasks related to Northwind that are due tomorrow.",
"initialRun": true,
"fetchTask": false,
"context": [
"commandBox",
"compose",
"message"
],
Parameter descriptions
A good parameter description explains what the parameter is, not what the parameter does, and the expected format.
When used directly in Microsoft Teams chat and Outlook mail compose, you can use a message extension to query one parameter at a time. When used as a plugin, message extension search commands support up to five parameters (one parameter must be visible in the message extension search bar). A parameter must have a good description, which should describe the expected input, including format or type.
The following are few examples on basic and advances search requests for various plugin scenarios:
Basic search: Search for tasks related to Northwind. Advanced search: Search for high priority tasks related to Northwind that are due tomorrow.
Parameter description example:
"parameters": [
{
"name": "Name",
"title": "Project or Task Name",
"description": "Project name or task name as keyword",
"inputType": "text"
},
{
"name": "Time",
"title": "Time",
"description": "Date or number of days for which to find tasks. Output: Number",
"inputType": "text"
},
{
"name": "Priority",
"title": "Priority",
"description": "Priority of tasks. Acceptable values: high, medium, low, NA ",
"inputType": "text"
}]
Debugging plugin selection
You can use developer mode while testing your plugin to verify if and how the orchestrator selected your plugin for use in response to a given prompt. From M365 Chat, you can enable developer mode by typing -developer on (or off to disable).
While developer mode is enabled, a card with debug information will be returned whenever the orchestrator searches specifically within your enterprise knowledge (data) and/or skills (plugins) to respond to a prompt. The debug info card includes the following fields:
- Enabled plugins: A list of plugins enabled by the user (from Plugins control below the chat input box)
- Matched functions: A list of plugins and functions matched in the runtime app index lookup
- Selected functions for execution: A list of plugin functions selected for invocation based on orchestrator reasoning
- Function execution details: Request and response function execution status
Troubleshooting execution failures
Here are some common failures you might encounter when debugging plugin execution, and possible causes.
No debug card
If the orchestrator doesn't require your Microsoft 365 data or skills to respond to a prompt, no debug info card will be returned.
Debug cards are also not returned in cases of capacity throttling, where you will typically see an error message to try again later.
Empty debug card
If no plugins were enabled, the debug info card will return empty.
Card with empty Matched functions
If relevant plugins are enabled, yet no matched functions were returned for the given prompt, this likely indicates the prompt did not explicitly mention the plugin name.
Card with empty Selected functions for execution
If no enabled plugin matched the search intent of the prompt, the debug info card will report No functions selected for execution. This is likely because the command description in the manifest is not semantically related to the search intent of the given prompt.
If Copilot was previously matching and executing your plugin functions successfully, this could be an indication of throttling.
Card with empty or failed Function execution details
For non-message extension plugins, if the function execution details or request status is empty or failed, it indicates a failure during Copilot's attempt to assign parameters to the selected function of your plugin. If the failure is consistent, it is most likely due to unclear plugin or parameter descriptions, invalid host urls, or other problems with your Open API definition.
For message extension plugins, best practice is to optimize for responses under nine seconds. For more more info, review the technical requirements for message extension plugins.
Card with function execution response status of 0
If the Function execution details is reporting a Response Status of 0, but Request status of Success, this might be an indication of timeout. Currently the timeout limit for Copilot execution of a plugin API is set at 10 seconds.
Next step
Learn best practices for optimizing plugin discoverability and usefulness in Copilot for Microsoft 365.
Feedback
Coming soon: Throughout 2024 we will be phasing out GitHub Issues as the feedback mechanism for content and replacing it with a new feedback system. For more information see: https://aka.ms/ContentUserFeedback.
Submit and view feedback for