Unstructured data as a knowledge source

By using Copilot Studio, you can enhance your agents with domain-specific knowledge powered by the same trusted, familiar data sources you build through Power Platform connectors.

When you upload external content from your device, OneDrive, or SharePoint, you can enrich your agents with contextual knowledge tailored to your business. Microsoft Dataverse securely stores these files and automatically processes them into semantic indexes and vector embeddings. This configuration enables your agents to generate more accurate, grounded responses based on the information you provide.

Files uploaded in Copilot Studio use Microsoft Dataverse to ingest raw files and create indexes and vector embeddings. These indexes and embeddings help provide quality responses for your agents. You can upload these files from your computer or by connecting to OneDrive or SharePoint.

When you upload files as knowledge sources, you help enrich your agents with extra data, augment the language model's knowledge, and ground the agent in specific information you provide. You can upload various files, which are semantically indexed as vector embeddings and then used as knowledge for agents. You can share this knowledge used in agents with authenticated and unauthenticated users of the agent.

To improve an agent’s responses, the system chunks uploaded files into pieces for faster processing and vector-indexes them to provide semantic matches with the user's query. The system stores the files securely in Dataverse. When a user queries through an agent, Copilot Studio finds the most relevant chunks that match the intent of the user query and returns the results to the user. 

Similarly, Dataverse ingests OneDrive files, SharePoint files (by using the options under file upload), and unstructured content like knowledge base articles from other enterprise systems such as Salesforce, ServiceNow, Confluence, and ZenDesk to provide better semantic results for the agent.

Note

For more information on using structured data, see Use code interpreter to analyze structured data.

Power Platform connectors for unstructured data

The following Power Platform connectors are configured to work with unstructured data sources:

OneDrive

OneDrive allows makers to use a file selector interface to choose the files and folders they want to include. Once selected, the system retrieves the items into Dataverse and indexes them for use. Folders you add include all of the supported files and subfolders within that folder up to the total file limit.

SharePoint

SharePoint allows makers to use a file selector interface to choose the files and folders they want to include. Once selected, the items are retrieved into Dataverse and indexed for use. Folders you add include all of the supported files and subfolders within that folder up to the total file limit. Currently, there's no support for Pages.

Salesforce

The Salesforce connector for unstructured data supports the ability to retrieve Knowledge Bases containing knowledge articles. Makers select a Knowledge Base and all articles within that Knowledge Base are indexed for use. You can't select individual articles or topics. When querying data, you can't specify a specific article or knowledge base. The Knowledge list shows a single object for all knowledge objects you select when you create the source.

ServiceNow

The ServiceNow connector for unstructured data supports the ability to retrieve Knowledge Bases containing knowledge articles. Knowledge Bases contain articles. Makers select a Knowledge Base and all articles within that Knowledge Base are indexed for use. You can't select individual articles. When querying data, you can't specify a knowledge base, folder, or individual article. The Knowledge list shows a single object for all knowledge objects you select when you create the source.

Confluence

The Confluence connector for unstructured data supports the ability to retrieve the spaces containing pages. The connector also supports subfolders. You can't select individual pages. When querying data, you can't specify a page. The Knowledge list shows a single object for all pages within the space.

Zendesk

The Zendesk connector for unstructured data supports the ability to retrieve the knowledge base containing knowledge articles. You can't select individual articles, categories, or sections. When querying data, you can't specify an article, category, or section. The Knowledge list shows a single object for all articles within the knowledge base.

Security

When a user queries an agent that's using a Power Platform Connector source, the system performs a few authorization checks.

Connector Access

When a maker first uses a connector-based source, the system prompts them to either select an existing Power Platform connector or to add one. This process ensures that data is only shared with makers who have the appropriate permissions, and provides access to the data source itself.

Content access

When a user makes a query, the system uses their connection information to check the data source and verify they have permission to see the content. Even though the system stores chunks and indexes locally in Dataverse, it performs a live check on the queries to make sure the current user has access to the data before providing a summary or response.

Note

• If a user doesn't have permission for a specific set of files or knowledge base articles, the system doesn't return a result to them. Instead, they receive a standard message that says "no results could be found." If users feel there should be results for that source, they need to work with their administrators to ensure they have permissions to the data they're trying to reach.
• The system doesn't store content permission information locally. It performs all permission checks live with the source to ensure they're the most up-to-date.

Synchronization and file refresh frequency

A scheduled synchronization job keeps connected files from OneDrive and SharePoint, and unstructured knowledge articles fresh. This job runs automatically in the background, refreshing the contents of the files and reindexing the changes to provide accurate results for queries. Refreshes manage not only changes to content, but also ensure any content deleted from the source no longer appears as part of any query responses. Currently, there's no way to manually trigger a refresh.

For more information about refresh frequency timing, see Copilot Studio unstructured data knowledge source limits.

Licensing

All requests that involve knowledge are charged at the Microsoft Copilot generative answers messaging rates. For more information, see Billing rates and management.

If knowledge sources require data ingestion, the storage of the data and the corresponding indexes to retrieve that data are subject to the storage entitlements the customer has. For more information on Dataverse natural language search, see Enhance AI-powered experiences with Dataverse search.

Limits and limitations

When you first enable unstructured data support, there might be a delay between 5 and 30 minutes for Dataverse configuration and indexing before the added files are processed. The length of time depends on the size of the current Dataverse environment.

Each agent can have a maximum of 500 knowledge objects. These objects can be files, folders, knowledge articles, websites, or other sources.

At this time, an agent can use only five different sources at a time. For example, SharePoint, Dataverse, OneDrive, or other sources.

For more information about specific limits and limitations for the supported unstructured data sources, see Copilot Studio unstructured data knowledge source limits.

Note

Copilot Studio agents require Dataverse search to use this knowledge source. If you can't add a Dataverse-enabled file to an agent, ask your administrator to turn on Dataverse search in your environment. For more information about Dataverse search and how to manage it, see What is Dataverse search and Configure Dataverse search for your environment.

To access OneDrive and SharePoint content stored in Dataverse, users must have at least a Basic User license for Power Apps or Dynamics 365. In addition, the Basic User permissions must also include read permissions for the following tables and entities:

• Plug-in Assembly
• Plug-in Type
• Sdk Message
• Sdk Message Processing Step
• Sdk Message Processing Step Image

You can configure these permissions in the Power Platform Admin Center or the Dynamics Admin Center.

FAQ

In the Add knowledge dialog, what is the difference between the two SharePoint options?

In the Add knowledge dialog, there are two SharePoint options.

The SharePoint option in the file upload section (1) is used to upload individual SharePoint files or folders. This option syncs files from SharePoint, and queries data at runtime without copying content to Dataverse.

This SharePoint option is used when you want real-time synchronization, but don't want to copy files to Dataverse. It also provides automatic updates when source files are changed.

The other SharePoint option (2) provides the full SharePoint integration in Copilot Studio using the SharePoint connector. This option is used when you need full SharePoint connector capabilities, custom authentication configurations, or advanced query options.

The SharePoint icon isn't displayed in the Upload files section of the Add knowledge dialog?

There's a slight delay between installing a solution and it being displayed in all existing organizations. To initiate a manual update, follow these steps:

1. Sign in to the Power Platform admin center by using administrator credentials.

2. On the side navigation, select Manage.

3. From the list of products, select Dynamics 365 Apps. A pane opens.

4. Search for "PowerAIExtensions".

5. Select the three dots (…) for Microsoft Dynamics 365 - PowerAIExtensions and select Install.

6. From the dropdown menu, select your environment and then select Install.

7. After the installation completes, open Power Apps in a new window.

8. In the left pane, select Solutions.

9. Select See History.

10. Search for "PowerAIExtensions_Anchor" and ensure it's set to 1.01.688 or higher.

What happens when I add more than 500 knowledge objects to my agent?

You can't add more objects unless you first delete previous objects.

Does each agent have their own index of the knowledge source?

Dataverse stores knowledge sources for use in the environment where you create them. If multiple agents use the same SharePoint folder, all the agents use a single instance of that folder.

What happens if I select a folder that has more than the maximum number of files, folders, and subfolders when adding a SharePoint or OneDrive source?

Copilot Studio retrieves and indexes up to the maximum number of files, folders, and subfolders, and index those. The remaining aren't processed. Currently, there isn't any messaging to indicate what was or wasn't processed.

One of the files I added (or that was part of a folder I added) is displayed as part of the knowledge source, but I can't get answers from it. Why?

This issue could be related to one of the following reasons:

• The file or folder is set to "Ready" on the Knowledge page.
• Ensure that the file name doesn't include an unsupported character (specifically for SharePoint files).
• Ensure that the file doesn't have a sensitivity setting of Confidential or Highly Confidential, or have password protection.
• Ensure that it's a supported file type.
• If the file or folder is from a different user's OneDrive or SharePoint site, verify that it's shared with the maker.
• If the file is knowledge base file, ensure that your account has permissions to view the content in the source system.