ai_parse_document task for jobs

Important

This feature is in Beta. To use it, a workspace admin must turn on Lakeflow AI Function Task from the Previews page. See Manage Azure Databricks previews.

To parse documents in a job, use an ai_parse_document task. The task runs ai_parse_document on each document in a Unity Catalog volume or table. The task parses text, tables, figure descriptions, and layout from documents such as PDFs and images, and writes the result for each document to a Unity Catalog table. For supported file formats and limits, see ai_parse_document.

Configure the task in the Jobs UI, create it with the Jobs API, or define it in YAML with Declarative Automation Bundles. In the API and bundles, use ai_function_task with the ai_parse_document function set. For requirements and how AI Function tasks process data, see AI Function tasks for jobs.

Add an ai_parse_document task in the Jobs UI

To add an ai_parse_document task:

  1. Navigate to the Tasks tab in the Jobs UI.
  2. Click Plus icon. Add task. If the job has no tasks yet, click Plus icon. Add another task type.
  3. Under Document intelligence, select Document Parsing.
  4. Enter a name into the Task name field. It defaults to parsing.
  5. Under Input, select the documents to parse. See Input.
    • Volume: Select a volume or a folder in a volume.
    • Table: Select a table and a FILE or BINARY Column.
  6. Under Output, select the Location, and optionally enter a Table name. See Output.
  7. (Optional) For a table input, add an ID column. See ID column.
  8. In Describe element types, select which parsed elements get an AI-generated description: All supported types, Figures only, or None. None generates no descriptions, which reduces compute and cost for documents with many figures. See Understanding elements.
  9. (Optional) To configure retries, run-duration or streaming-backlog thresholds, or notifications, see Advanced task settings.
  10. Click Save task.

The task runs the next time the job runs. To run the job on a schedule, see Run jobs on a schedule. To edit, clone, disable, or delete this task, see Configure and edit tasks in Lakeflow Jobs.

Create the task with the Jobs API

Send a request to POST /api/2.2/jobs/create with the following JSON body. This example creates a job that parses documents in /Volumes/main/finance/invoices and writes the results to main.finance.parsed_invoices.

Replace the example locations with your own. The input volume and output schema must exist. The task creates and manages its output table. For the required privileges, see Requirements.

{
  "name": "parse_invoices",
  "tasks": [
    {
      "task_key": "parsing",
      "ai_function_task": {
        "volume": {
          "volume_path": "/Volumes/main/finance/invoices"
        },
        "output": {
          "catalog": "main",
          "schema": "finance",
          "table": "parsed_invoices"
        },
        "ai_parse_document": {
          "version": "2.0"
        }
      }
    }
  ]
}

For authentication and request details, see jobs/create. For the task settings, see ai_function_task.

Define the task in Declarative Automation Bundles

Declarative Automation Bundles let you define the job in source files, so you can version, review, and deploy changes to it like code.

Add the following job definition to the databricks.yml file of an existing bundle. It defines the same task as the API example. To create a bundle and validate, deploy, and run the job, see Develop a job with Declarative Automation Bundles.

resources:
  jobs:
    parse_invoices:
      name: parse_invoices
      tasks:
        - task_key: parsing
          ai_function_task:
            volume:
              volume_path: /Volumes/main/finance/invoices
            output:
              catalog: main
              schema: finance
              table: parsed_invoices
            ai_parse_document:
              version: '2.0'

Replace the example locations as described in the API example. For the complete configuration keys and more examples, including chained tasks, see AI Function task.

Output

The response column of the output table holds the full ai_parse_document result for each document as a VARIANT. The task doesn't add a column for each value. To get a value as a column, select it from response in a query. For the format, see ai_parse_document returns. For the other columns, see Output table.

Use with other AI functions

To extract fields from or classify the parsed documents, add an ai_extract or ai_classify task that runs after the ai_parse_document task. In the Jobs UI, the new task's input defaults to the ai_parse_document task's output table and response column. See Use the output in a later task.

To define an ai_parse_document task followed by an ai_extract task in Declarative Automation Bundles, see the examples in AI Function task.

Additional resources