ai_classify task for jobs

Important

This feature is in Beta. To use it, a workspace admin must turn on Lakeflow AI Function Task from the Previews page. See Manage Azure Databricks previews.

To classify text in a job, use an ai_classify task. The task runs ai_classify on each row of a Unity Catalog table. The task classifies text or parsed documents into labels that you define and writes the result for each row to a Unity Catalog table.

Configure the task in the Jobs UI, create it with the Jobs API, or define it in YAML with Declarative Automation Bundles. In the API and bundles, use ai_function_task with the ai_classify function set. For requirements and how AI Function tasks process data, see AI Function tasks for jobs.

Add an ai_classify task in the Jobs UI

To add an ai_classify task:

  1. Navigate to the Tasks tab in the Jobs UI.
  2. Click Plus icon. Add task. If the job has no tasks yet, click Plus icon. Add another task type.
  3. Under Document intelligence, select Classification.
  4. Enter a name into the Task name field. It defaults to classification.
  5. Under Input, select a table and the Column to classify. To classify documents, parse them first. See Use with other AI functions.
  6. Under Output, select the Location, and optionally enter a Table name. See Output.
  7. (Optional) Add an ID column. See ID column.
  8. Define the labels to classify into. See Define the labels.
  9. (Optional) Configure the settings under Optional settings. See Optional settings.
  10. (Optional) To configure retries, run-duration or streaming-backlog thresholds, or notifications, see Advanced task settings.
  11. Click Save task.

The task runs the next time the job runs. The first run classifies every row in the table. Later runs classify only the rows added or changed since the previous run, and remove the results of deleted rows. To run the job on a schedule, see Run jobs on a schedule. To edit, clone, disable, or delete this task, see Configure and edit tasks in Lakeflow Jobs.

Define the labels

Under Labels, add each label that an input can be classified as, with a name and an optional description. Label names must be unique. A description helps the function tell similar labels apart.

The function assigns each input the best-matching label, even when no label fits well. To catch inputs that don't fit, add a catch-all label, such as other, and describe what belongs in it. To review uncertain results, turn on Confidence scores.

An ai_classify task takes 2 to 10 labels. The ai_classify function itself takes up to 500 labels, so to use more than 10, call ai_classify in a SQL, notebook, or pipeline task.

Optional settings

Under Optional settings, you can configure the following:

  • Confidence scores: Include a confidence score for each label in the response.
  • Rationales: Include a rationale that explains each classification in the response.
  • Multiple labels: Assign more than one label to each input.
  • Instructions: Guidance applied to every row, such as which label to choose when two labels fit.
  • Version: The ai_classify version that the task runs, shown for reference. The version is pinned, so scheduled runs don't change when the function is updated.

Create the task with the Jobs API

Send a request to POST /api/2.2/jobs/create with the following JSON body. This example creates a job that classifies the body column of main.support.tickets as urgent or normal and writes the results to main.support.ticket_labels. The ticket_id column identifies each input row across runs.

Replace the example table and column names with your own. The input table and output schema must exist, and the input table must have change data feed enabled. The task creates and manages its output table. For the required privileges and table settings, see Requirements.

{
  "name": "classify_support_tickets",
  "tasks": [
    {
      "task_key": "classification",
      "ai_function_task": {
        "delta_table": {
          "table_name": "main.support.tickets",
          "input_column": "body",
          "merge_key_columns": ["ticket_id"]
        },
        "output": {
          "catalog": "main",
          "schema": "support",
          "table": "ticket_labels"
        },
        "ai_classify": {
          "version": "2.1",
          "inline_labels": {
            "labels": [
              {
                "name": "urgent",
                "description": "The customer is blocked or reports an outage."
              },
              {
                "name": "normal",
                "description": "A question or a non-blocking issue."
              }
            ]
          }
        }
      }
    }
  ]
}

For authentication and request details, see jobs/create. For the task settings, see ai_function_task. To classify documents in a volume, parse them first.

Define the task in Declarative Automation Bundles

Declarative Automation Bundles let you define the job in source files, so you can version, review, and deploy changes to it like code.

Add the following job definition to the databricks.yml file of an existing bundle. It defines the same task as the API example. To create a bundle and validate, deploy, and run the job, see Develop a job with Declarative Automation Bundles.

resources:
  jobs:
    classify_support_tickets:
      name: classify_support_tickets
      tasks:
        - task_key: classification
          ai_function_task:
            delta_table:
              table_name: main.support.tickets
              input_column: body
              merge_key_columns: [ticket_id]
            output:
              catalog: main
              schema: support
              table: ticket_labels
            ai_classify:
              version: '2.1'
              inline_labels:
                labels:
                  - name: urgent
                    description: The customer is blocked or reports an outage.
                  - name: normal
                    description: A question or a non-blocking issue.

Replace the example table and column names as described in the API example. For the complete configuration keys and more examples, see AI Function task.

To classify documents in a volume, parse them first. See Use with other AI functions.

Output

The response column of the output table holds the full ai_classify result for each row as a VARIANT, including confidence scores and rationales when you turn them on. The task doesn't add a column for each value. To get a value as a column, select it from response in a query. For the format, see ai_classify returns. For the other columns, see Output table.

Additional resources