ai_parse_document task for jobs

Important

This feature is in Beta. To use it, a workspace admin must turn on Lakeflow AI Function Task from the Previews page. See Manage Azure Databricks previews.

To parse documents in a job, use an ai_parse_document task. The task runs ai_parse_document on each document in a Unity Catalog volume or table. The task parses text, tables, figure descriptions, and layout from documents such as PDFs and images, and writes the result for each document to a Unity Catalog table. For supported file formats and limits, see ai_parse_document.

An ai_parse_document task is a type of AI Function task. In the Jobs API and Declarative Automation Bundles, it's an ai_function_task with the ai_parse_document function set. For requirements and how AI Function tasks process data, see AI Function tasks for jobs.

Add an ai_parse_document task

To add an ai_parse_document task:

  1. Navigate to the Tasks tab in the Jobs UI.
  2. Click Plus icon. Add task. If the job has no tasks yet, click Plus icon. Add another task type.
  3. Under Document intelligence, select Document Parsing.
  4. Enter a name into the Task name field. It defaults to parsing.
  5. Under Input, select the documents to parse. See Input.
    • Volume: Select a volume or a folder in a volume.
    • Table: Select a table and a FILE or BINARY Column.
  6. Under Output, select the Location, and optionally enter a Table name. See Output.
  7. (Optional) For a table input, add an ID column. See ID column.
  8. In Describe element types, select which parsed elements get an AI-generated description: All supported types, Figures only, or None. None generates no descriptions, which reduces compute and cost for documents with many figures. See Understanding elements.
  9. (Optional) To configure retries, run-duration or streaming-backlog thresholds, or notifications, see Advanced task settings.
  10. Click Save task.

The task runs the next time the job runs. To run the job on a schedule, see Run jobs on a schedule. To edit, clone, disable, or delete this task, see Configure and edit tasks in Lakeflow Jobs.

Define the task in Declarative Automation Bundles

Declarative Automation Bundles let you define the job in source files, so you can version, review, and deploy changes to it like code. Support for AI Function tasks in Declarative Automation Bundles is in Private Preview. For the ai_parse_document keys and an example, see AI Function task. To create the job with the Jobs API instead, see jobs/create.

Output

The response column of the output table holds the full ai_parse_document result for each document as a VARIANT. The task doesn't add a column for each value. To get a value as a column, select it from response in a query. For the format, see ai_parse_document returns. For the other columns, see Output table.

Use with other AI functions

To extract fields from or classify the parsed documents, add an ai_extract or ai_classify task that runs after the ai_parse_document task. In the Jobs UI, the new task's input defaults to the ai_parse_document task's output table and response column. See Use the output in a later task.

To define an ai_parse_document task followed by an ai_extract task in Declarative Automation Bundles, see the examples in AI Function task.

Additional resources

  • Last updated on