Microsoft Discovery Bookshelf

Microsoft Discovery includes the Bookshelf, a capability that enables customers to convert their data into curated graphs known as Knowledge Bases (KBs). The key components of the Bookshelf are the Bookshelf resource and the Knowledge Base within each Bookshelf. A Knowledge Base contains a vector database and knowledge graph of your indexed artifacts. Discovery agents can use KBs as grounding skills and query them for various use cases, including answering questions, summarization, and reasoning.

When to use the Bookshelf

The Bookshelf is best for reasoning over your curated, proprietary data. Knowledge Bases are especially effective when their scoped contents are thematically related and directly applicable to your Discovery workflow. For example, an Application-Specific Integrated Circuit (ASIC) design team could create a Knowledge Base with their project's hardware specifications, simulation result reports, and the latest relevant literature from the field. Querying this Knowledge Base during design workflows ensures Discovery's reasoning is grounded with previous engineering content and scientific literature.

Additionally, the Bookshelf is currently best suited for global-level queries, or queries that require synthesis, summarization, and otherwise reasoning over the entire corpus to answer. For example, "what are the themes," "what are the implications," "describe the relationships," and so on. For local-level queries, or lookup questions that one or a few specific chunks of text can answer, use Azure AI Search or other vector-based search tools.

For using data in a tool call or otherwise directly using data in Discovery, creating a Knowledge Base is often not necessary. Similarly, to search over vast repositories of data or to find resources that might be relevant to your workflow, we suggest using Azure AI Search, SharePoint Search, or similar general purpose search tools. Once you have identified the data that is most relevant to your workflow, a Knowledge Base including this curated data can help ground your Discovery workflows and derive new insights in context.

Features

At a high level, the Bookshelf works by converting diverse file formats to text, then generating a graphical representation of that text, which can be queried using natural language.

The Bookshelf uses an advanced technique developed by Microsoft Research called Graph Retrieval-Augmented Generation (GraphRAG) to transform customer data into graph-based representations and generate responses to queries. Unlike traditional RAG methods, GraphRAG-based algorithms not only create an indexed vector database of the source content but also construct a knowledge graph that captures entity relationships within the data. Research from Microsoft demonstrates that GraphRAG delivers more accurate and comprehensive grounding information than standard RAG or vector-based techniques, leading to higher-quality responses.

Both Microsoft Discovery and the Microsoft Discovery app use the same GraphRAG-based indexing technology under the hood.

Indexing

Currently, the Bookshelf supports indexing unstructured (text-based) file formats stored in Azure Blob Storage or stored locally if using the app. Supported file formats are documented in the Create a Bookshelf and index a Knowledge Base how-to guide.

The Bookshelf in Microsoft Discovery uses Azure AI Search Enrichment to process supported file formats. Images embedded in supported file formats are processed using Azure AI Search's built-in Vision skill, which automatically generates alt-text for embedded images. The knowledge graph and vector database that results from indexing, collectively known as a Knowledge Base (KB), are stored in an Azure SQL DB in your subscription.

The Bookshelf in the Microsoft Discovery app uses different libraries to process files locally, but the end result is still a queryable knowledge graph.

Query

The Bookshelf provides the query function that any agent running on the Microsoft Discovery platform can invoke, including your own agent. Through the query and/or agent definition, you can control how query outputs are formatted. For example, you can specify your preferred verbosity, the intended style of the report, if certain information should be ranked or organized in a table, and so on. All Bookshelf query outputs display citations as clickable links to source content.

Known limitations

Unsupported file types

Encrypted, password-protected, or sensitivity-labeled files aren't supported for indexing. Any unsupported file types are skipped during indexing.

One knowledge base per Bookshelf

Each Bookshelf can only contain one Knowledge Base. However, Projects can contain many Bookshelves.

Note

The ability to create multiple Knowledge Bases within the same Bookshelf is a planned feature for future releases.

Incremental indexing

Incremental indexing isn't currently supported. To update Knowledge Bases, you must delete them and re-index.

Note

Incremental indexing is a planned feature for future releases.

Local scope queries

Currently, the Bookshelf is best suited for global scope queries, not local scope queries. Your retrieval quality for local scope queries might vary.

Note

Equal retrieval quality for global and local scope queries is a planned feature for future releases.

Scale

The Bookshelf in Microsoft Discovery currently supports Small (<200 MB of text), Medium (<500 MB of text, default size), and Large (<1 GB of text)-sized deployments. For more information on supported index sizes and the resources required to support each size, see the Bookshelf creation How To guide. Bookshelf scale in the Microsoft Discovery app is limited by your machine's compute power.

Best practices

The Bookshelf is an evolving feature. Over the course of future releases, Microsoft will improve the costs and time associated with creating Bookshelf deployments and indexing and searching over KBs. The product team also plans to support incremental indexing and take advantage of newer LLMs for search. Currently, for the best performance and to minimize costs of re-deployment, re-indexing, re-enrichment, or search, follow these best practices:

• Limit each Knowledge Base to Small or Medium (default)-sized deployments
• Ensure each KB's content is thematically coherent and directly applicable to your Discovery workflow.
• Use generic search tools like Azure AI Search, Copilot, M365/SharePoint search, and others to broadly search your documents and help curate the subset for Bookshelf deep reasoning.
• Use a Knowledge Base to identify what is missing from that Knowledge Base. For example, try querying "what context is missing to be able to determine <x>?" or "what additional information is required to assess <y>?" Answers to these kinds of queries can help you further curate and refine your KBs, especially as your Discovery workflows progress.

Best practices at a glance

Bookshelf scenario?	Data set size	Data type	Curation level	Primary use case	Recommendation	Requirements	Example
☑️ Yes	<200 MB	Unstructured, text-based	Highly curated	Global-level reasoning	Bookshelf in the Discovery app, or Small Bookshelf enterprise deployment	Supported filetypes in Azure Blob storage or stored locally	Focused literature packet, design docs, internal whitepapers, or project-specific docs
☑️ Yes	~200MB to ~500 MB	Unstructured, text-based	Curated, coherent	Global-level reasoning	Medium Bookshelf enterprise deployment	Supported filetypes in Azure Blob storage; compute, quota, and indexing time planning	Curated literature review, domain-specific corpus, or project-wide knowledge
☑️ Yes	~500 MB to ~1 GB	Unstructured, text-based	Curated, coherent	Global-level reasoning	Large Bookshelf enterprise deployment	Supported filetypes in Azure Blob storage; higher compute, quota, and indexing time planning	Large curated technical library, project-specific research corpus
🚫 No	Any	Structured, relational data, SQL, warehouses, etc.	Schema-governed	Precise facts, aggregations, joins, metrics, analytics, calculations	Write a Discovery Agent to leverage SQL, KQL, Fabric semantic model, or Data Agent	Tables, views, and functions scoped with descriptions, joins, business terms, and example queries	"Show yield by batch," "compare experiment outcomes," "calculate inventory trend"
🚫 No	Any	Structured, CSVs, datasets, images, simulation outputs, code, model inputs/outputs	Not curated for reasoning	Input for/output from models and tools, running computations, reading/writing files	Add data to Discovery Storage Containers / Storage Assets or directly access via the model/tool	Storage Container/Storage Asset, permissions, tool bindings	Simulation inputs, generated outputs, HPC jobs, notebooks, model artifacts