Analyze files in volumes with a Genie Agent

A Genie Agent can analyze the unstructured files stored in Unity Catalog volumes, such as PDFs, Word documents, slide decks, and images. An agent author attaches volumes to the agent, and business users ask questions about their content alongside the agent's structured data in a single conversation.

Important

This feature is in Public Preview. Workspace admins can control access to Analyze Files in Volumes with Genie Agents from the Previews page. See Manage Azure Databricks previews.

Note

Genie Agents were formerly known as Genie Spaces.

What you can ask a Genie Agent about your files

By attaching volumes to an agent, business users can combine data from documents such as support tickets, sales call logs, and policy files with data from the agent's structured tables. Users can then answer questions such as:

  • Which support issues are taking the longest to resolve, and what are customers saying in escalation requests?
  • Why are late-stage sales opportunities stalling, and what objections appear most often in sales call logs?
  • Which regions are underperforming, and what internal escalation policies apply?

When to use file analysis in a Genie Agent

Attach volumes to a Genie Agent when you want a specialized agent for a specific domain. Choose a different approach when your needs are broader:

  • Use Genie One as a unified entry point for all questions about your structured data. Genie One can call a Genie Agent to answer questions about files.
  • Build a custom agent in code with the Agent Bricks CLI only when you need fine-grained control over agent behavior, such as model selection, chunking strategy, or orchestrating multiple sub-agents.
  • Use file upload to blend a local CSV or Excel file into a single conversation instead of analyzing files stored in volumes.

How a Genie Agent analyzes files

When a user asks a question that requires context from a volume, the Genie Agent retrieves the file content that is most relevant to the question and adds it to the agent's context. The agent then reasons over that content together with any structured data to generate an answer with citations. These citations can point to the specific page of the source file.

File analysis runs only in Agent mode, which uses multi-step reasoning to retrieve and analyze the relevant files. See Agent mode.

The Genie Agent always uses the permissions of the user who asks the question. If a user doesn't have at least READ VOLUME permission on some of the attached volumes, the agent answers using only the volumes that the user can access. See Manage data objects. Users can ask questions across a specific volume, multiple volumes, and structured data in the same conversation.

Requirements

The following requirements apply to the agent and its users:

Attach volumes to a Genie Agent

Before you begin, review the Requirements.

To attach volumes to a Genie Agent:

  1. Open the Genie Agent that you want to add files to.
  2. Open the Sources tab, then click Add.
  3. Select one or more volumes. The Genie Agent processes the entire volume. You cannot attach a subset of the folders or files in a volume.
  4. Add a clear description in the Description field. See Optimize your agent for file questions.

You can attach a volume as a source even when its catalog isn't accessible from this workspace by default, such as a catalog that is bound to specific workspaces.

When you attach a volume, Azure Databricks automatically enables content search.

Content search parses and indexes your files so the Genie Agent can reason over a larger set of files, draw on specific passages across many files, and answer questions with lower latency. For file and volume limits, see Limitations.

Content search stays in sync with your volume automatically. When you add, change, or remove files, Azure Databricks updates the index to reflect your changes, so the Genie Agent reasons over current content without any manual step.

Optimize your agent for file questions

After you attach your volumes, follow these recommendations to help the Genie Agent use their content:

  • Add a clear description: Describe what content the volume contains, how it is organized, and how the agent should use it. The agent uses this description to retrieve relevant files.
  • Avoid duplicate content: Attaching multiple volumes that contain overlapping information makes it harder for the agent to retrieve relevant documents. The same applies to individual files within a volume.
  • Avoid irrelevant files: Include only files that are relevant to the agent's domain. Irrelevant files can confuse the agent.
  • Use clear file names: Use descriptive file names so the agent can distinguish between files.

Ask questions about files

To ask questions about files in a Genie Agent:

  1. Open a Genie Agent that has volumes attached.
  2. Turn on Agent mode in the chat box. You must use Agent mode to answer questions about files.
  3. Enter a question and send your prompt. Agent mode retrieves context from the relevant volume, and then generates an answer after the analysis is complete.

Content search supported regions

Content search is available in the regions listed in this section.

In some regions, content search runs only if an account admin enables cross-Geo processing, which allows Azure Databricks to process your files outside your workspace Geo. See Enable cross-Geo processing.

Regions that don't require cross-Geo processing

Content search runs in your workspace region for the following regions:

  • Australia East
  • Central US
  • East US
  • East US 2
  • France Central
  • Germany West Central
  • Japan East
  • North Central US
  • North Europe
  • South Central US
  • West Europe
  • West US
  • West US 2

Regions that require cross-Geo processing

For the following regions, an account admin must enable cross-Geo processing before content search runs:

  • Brazil South
  • Canada Central
  • Central India
  • East Asia
  • Southeast Asia
  • UK South

Limitations

The following limitations apply:

  • Only Unity Catalog managed volumes support content search. External volumes and My Files volumes aren't supported. See Store files in My Files.
  • Each workspace can have up to 250 volumes attached to Genie Agents.
  • Each indexed volume can contain up to 10,000 files. To use a larger volume, enable file events. See Use volumes with more than 10,000 files.
  • Content search skips files larger than 100 MB and files with more than 500 pages.
  • Content search supports PDF, JPG, JPEG, PNG, TIFF, TIF, DOC, DOCX, PPT, PPTX, MD, or TXT files. It skips all other file types.

Use volumes with more than 10,000 files

Without file events, content search supports Unity Catalog managed volumes with up to 10,000 files. To use a managed volume with more than 10,000 files, enable file events on the external location that contains the volume. File events let content search detect new, changed, and deleted files through cloud storage notifications instead of repeatedly scanning your storage. To learn what file events are and how they work, see File events FAQ.

The setup has the following steps:

  1. Find the external location that contains your volume.
  2. Turn on file events for that external location and test the connection.
  3. If the test fails, configure the cloud permissions that file events need. See Set up file events for an external location.

You can skip this setup if file events are already enabled and working on the external location that contains your volume. To find the external location and check its status, see Step 1: Find the external location that contains your volume.

Step 1: Find the external location that contains your volume

Permissions required:

  • USE CATALOG on the catalog that contains the volume.
  • USE SCHEMA on the schema that contains the volume.
  • READ METADATA on the volume. READ VOLUME, MANAGE, or ownership of the volume also lets you view its metadata.
  • READ METADATA on each external location that you need to inspect. MANAGE, ownership of the external location, or metastore admin access also qualifies.

Catalog Explorer

  1. In the sidebar, click Data icon. Catalog.

  2. Open the volume that your Genie Agent uses. On the Details tab, find Storage location and copy the cloud path. You use it to find the matching external location. For example:

    abfss://container-1@account1.dfs.core.windows.net/6c55cf12…/volumes/de8e8205…

  3. Click Connect > External Locations.

  4. Find the external location whose URL is the beginning of the volume's storage path.

    For example, an external location at abfss://container-1@account1.dfs.core.windows.net/6c55cf12… covers a managed volume stored at abfss://container-1@account1.dfs.core.windows.net/6c55cf12…/volumes/de8e8205….

  5. Open the matching external location's Overview tab and check Enable file events. If it shows Enabled, click Test connection to confirm that file events are working. If the test passes, you don't need Steps 2 and 3. If it fails, skip to Step 3.

Python SDK

Run the following code in a notebook. Replace <catalog>.<schema>.<volume> with the full name of your volume. These snippets require a recent version of the Databricks SDK for Python. If you get an ImportError or AttributeError, run %pip install --upgrade databricks-sdk, and then run dbutils.library.restartPython().

from databricks.sdk import WorkspaceClient

w = WorkspaceClient()

# Storage location of the volume
print(w.volumes.read("<catalog>.<schema>.<volume>").storage_location)

# External locations that you can access
for location in w.external_locations.list():
    print(location.name, location.url)

Find the external location whose URL is the beginning of the volume's storage location.

If you found a matching external location, check whether file events are enabled:

print(w.external_locations.get("<external-location-name>").effective_enable_file_events)

If it prints True, click Test connection on the external location in Catalog Explorer to confirm that file events are working. If the test passes, you don't need Steps 2 and 3. If it fails, skip to Step 3. If it prints False or None, continue with Step 2.

If no external location matches, you might not have access to it. The list shows only external locations that you can access. Ask your metastore admin or the owner of the external location to check. If the volume uses metastore root storage without an external location, you can't enable file events until one exists. Removing metastore-level storage creates one, but after that, every new catalog must have its own storage location. See Remove metastore-level storage.

On the external location's Overview tab in Catalog Explorer, note the storage credential name in the Credential row. You need this name for the permissions in Step 2 and the cloud configuration in Step 3.

Step 2: Turn on file events and test the connection

Use the external location and storage credential that you found in Step 1.

Permissions required: Metastore admins can complete this step. Otherwise, you need both of the following:

  • MANAGE on the external location, or ownership of the external location.
  • MANAGE on the storage credential, ownership of the credential, or CREATE EXTERNAL LOCATION on both the credential and the metastore that contains the external location.

Automatic setup isn't available for Premium storage accounts. For these accounts, follow the provided queue setup guidance instead of the following automatic setup.

Catalog Explorer

  1. Go to the external location in Catalog Explorer. Click the Kebab menu icon. kebab menu next to Test connection and select Edit.
  2. Expand Advanced Options. Turn on Enable file events and set File event type to Automatic.
  3. Click Update. If validation fails and you're asked to force the update, cancel and continue with Step 3. A forced update leaves file events enabled but not working.
  4. Wait a few seconds, and then click Test connection to confirm that file events are working. If the test fails and file events were off before this step, turn off Enable file events again so that file arrival triggers on this location keep working. Then continue with Step 3.

Python SDK

Run the following code in a notebook. Replace <external-location-name> with the external location from Step 1.

For an external location on ADLS:

from databricks.sdk import WorkspaceClient
from databricks.sdk.service.catalog import AzureQueueStorage, FileEventQueue

w = WorkspaceClient()

w.external_locations.update(
    name="<external-location-name>",
    enable_file_events=True,
    file_event_queue=FileEventQueue(managed_aqs=AzureQueueStorage()),
)

Then click Test connection in Catalog Explorer to confirm that file events are working. If update raises a validation error, continue with Step 3. If the test fails and file events were off before this step, turn them off again so that file arrival triggers on this location keep working, and then continue with Step 3:

w.external_locations.update(name="<external-location-name>", enable_file_events=False)

Step 3: Configure cloud permissions

Complete this step if Test connection failed in Step 1 or Step 2, or if file events validation failed when you clicked Update in Step 2.

Follow Set up file events for an external location for the complete cloud configuration instructions. The following notes and examples help you identify the permissions and cloud resources needed for your external location.

Permissions required: Permission to grant access to the cloud identity that your storage credential uses, and to change the storage settings that the guide requires. If you don't have this access, ask your cloud administrator.

To find that cloud identity, open the external location's Overview tab in Catalog Explorer and click the name in the Credential row to view the storage credential's details.

If a cloud administrator makes the change, provide them with the external location's URL and its storage credential details so that they can identify the correct cloud resources and identity.

For example, for an external location on ADLS with the URL abfss://container-1@account1.dfs.core.windows.net/6c55cf12…, the storage account name is account1.

Grant the managed identity or service principal of the storage credential the following roles:

  • Storage Blob Data Contributor, Storage Queue Data Contributor, and Storage Account Contributor on the storage account account1.
  • EventGrid Data Contributor on the resource group that contains account1.

Make sure that Allow trusted Microsoft services to access this resource is enabled in your storage account networking rules.

After completing the guide's cloud configuration for automatic setup, return to Step 2 to enable file events and test the connection.

Frequently asked questions

The following questions cover common topics for analyzing files in volumes.

How is this feature billed?

Azure Databricks charges for the following:

  • Preparing files: When Azure Databricks enables content search or syncs a volume, it parses your files with the ai_parse_document AI function, which is billed under AI Functions pricing.
  • Answering questions: The agent's reasoning over your files is billed as part of standard Genie usage. See Genie pricing.

Can I analyze files in volumes through the API?

Yes. You can analyze files in volumes through the Agent mode APIs. See the Agent mode API reference.

How can I use volumes with more than 10,000 files?

To use a Unity Catalog managed volume with more than 10,000 files, enable file events on the external location that contains the volume. See Use volumes with more than 10,000 files.

Additional resources

See the following resources for related information: