Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
Important
This feature is in Beta. To use it, a workspace admin must turn on Lakeflow AI Function Task from the Previews page. See Manage Azure Databricks previews.
To parse documents in a job, use an ai_parse_document task. The task runs ai_parse_document on each document in a Unity Catalog volume or table. The task parses text, tables, figure descriptions, and layout from documents such as PDFs and images, and writes the result for each document to a Unity Catalog table. For supported file formats and limits, see ai_parse_document.
Configure the task in the Jobs UI, create it with the Jobs API, or define it in YAML with Declarative Automation Bundles. In the API and bundles, use ai_function_task with the ai_parse_document function set. For requirements and how AI Function tasks process data, see AI Function tasks for jobs.
Add an ai_parse_document task in the Jobs UI
To add an ai_parse_document task:
- Navigate to the Tasks tab in the Jobs UI.
- Click
Add task. If the job has no tasks yet, click
Add another task type.
- Under Document intelligence, select Document Parsing.
- Enter a name into the Task name field. It defaults to
parsing. - Under Input, select the documents to parse. See Input.
- Volume: Select a volume or a folder in a volume.
- Table: Select a table and a
FILEorBINARYColumn.
- Under Output, select the Location, and optionally enter a Table name. See Output.
- (Optional) For a table input, add an ID column. See ID column.
- In Describe element types, select which parsed elements get an AI-generated description: All supported types, Figures only, or None. None generates no descriptions, which reduces compute and cost for documents with many figures. See Understanding elements.
- (Optional) To configure retries, run-duration or streaming-backlog thresholds, or notifications, see Advanced task settings.
- Click Save task.
The task runs the next time the job runs. To run the job on a schedule, see Run jobs on a schedule. To edit, clone, disable, or delete this task, see Configure and edit tasks in Lakeflow Jobs.
Create the task with the Jobs API
Send a request to POST /api/2.2/jobs/create with the following JSON body. This example creates a job that parses documents in /Volumes/main/finance/invoices and writes the results to main.finance.parsed_invoices.
Replace the example locations with your own. The input volume and output schema must exist. The task creates and manages its output table. For the required privileges, see Requirements.
{
"name": "parse_invoices",
"tasks": [
{
"task_key": "parsing",
"ai_function_task": {
"volume": {
"volume_path": "/Volumes/main/finance/invoices"
},
"output": {
"catalog": "main",
"schema": "finance",
"table": "parsed_invoices"
},
"ai_parse_document": {
"version": "2.0"
}
}
}
]
}
For authentication and request details, see jobs/create. For the task settings, see ai_function_task.
Define the task in Declarative Automation Bundles
Declarative Automation Bundles let you define the job in source files, so you can version, review, and deploy changes to it like code.
Add the following job definition to the databricks.yml file of an existing bundle. It defines the same task as the API example. To create a bundle and validate, deploy, and run the job, see Develop a job with Declarative Automation Bundles.
resources:
jobs:
parse_invoices:
name: parse_invoices
tasks:
- task_key: parsing
ai_function_task:
volume:
volume_path: /Volumes/main/finance/invoices
output:
catalog: main
schema: finance
table: parsed_invoices
ai_parse_document:
version: '2.0'
Replace the example locations as described in the API example. For the complete configuration keys and more examples, including chained tasks, see AI Function task.
Output
The response column of the output table holds the full ai_parse_document result for each document as a VARIANT. The task doesn't add a column for each value. To get a value as a column, select it from response in a query. For the format, see ai_parse_document returns. For the other columns, see Output table.
Use with other AI functions
To extract fields from or classify the parsed documents, add an ai_extract or ai_classify task that runs after the ai_parse_document task. In the Jobs UI, the new task's input defaults to the ai_parse_document task's output table and response column. See Use the output in a later task.
To define an ai_parse_document task followed by an ai_extract task in Declarative Automation Bundles, see the examples in AI Function task.
Additional resources
- AI Function tasks for jobs: Learn about AI Function tasks.
- Document Parsing: Test parsing on sample documents in the Agent Bricks UI.