Orchestrate workloads with Lakeflow Jobs

Orchestrate your workloads with Lakeflow Jobs, the workflow automation for Azure Databricks, so that you can run production workloads unattended and see what happened in each run. A job coordinates the tasks in a workflow, such as running a pipeline, a notebook, or a SQL query, in the order that you define. It runs them on a schedule or when an event occurs, such as new data arriving in cloud storage. Jobs can retry failed tasks, send notifications, and record every run.

The following video introduces orchestration with Lakeflow Jobs:

Jobs, tasks, and triggers

These three terms describe how a job organizes and starts its work:

  • Job: The resource that coordinates, schedules, and runs your tasks. A job can include conditional logic, such as if-else branches and for each loops. A graph shows the tasks and their dependencies.
  • Task: A unit of work in a job, such as a notebook, a pipeline, a Python script, or a SQL query. A task can depend on other tasks, so that it runs only after they finish. See Types of tasks.
  • Trigger: The condition that starts a job run. A trigger can run a job on a schedule, when a table changes, when files arrive in cloud storage, or continuously. See Automate jobs with schedules and triggers.

Requirements

You need the following before you create the job:

Create a job

The following steps create a job that runs your pipeline, and then runs a notebook after the pipeline update completes.

Add a pipeline task

Create the first task, which runs your pipeline:

  1. In your workspace, click Workflows icon. Jobs & Pipelines in the sidebar.
  2. Click Create, and then click Job.
  3. Replace the default job name with a name for your job.
  4. Configure the first task:
    • Task name: Enter a name, such as refresh-pipeline.
    • Type: Select Pipeline.
    • Pipeline: Select your pipeline.
  5. Click Save task.

Add a notebook task

Next, create a notebook task that runs after the pipeline update completes:

  1. Click Plus icon. Add task > Notebook.
  2. Configure the task:
    • Task name: Enter a name, such as report.
    • Source: Select Workspace.
    • Path: Select your notebook.
    • Depends on: Confirm that Depends on shows the pipeline task.
  3. Click Save task.

Add a schedule

Set a schedule so the job runs without a manual start:

  1. In the Job details pane, scroll down to the Schedules & Triggers section, and then click Add trigger.
  2. In Trigger type, select Scheduled, and then set the schedule.
  3. Click Save.

To run the job when new data arrives instead, select the File arrival or Table update trigger type.

Add notifications

Pick where notifications go, and which run events trigger them:

  1. In the Job details pane, in the Job notifications section, click Edit notifications.
  2. Click Add notification. In Destination, select Email address or a system destination. An administrator must configure system destinations before you can select one.
  3. Select the events to notify about, such as Failure.
  4. Click Save.

Run the job

Click Run Now Button to run the job immediately. On the Runs tab, click the run's start time to view the output and status of each task.

If your workspace is enabled for Unity Catalog and serverless compute for jobs, serverless compute is the default for all tasks that support it.

For a quickstart with sample data, see Create your first workflow with Lakeflow Jobs.

Choose the UI or Declarative Automation Bundles

Build in the UI when you're setting up your first job and want to see it run quickly, when you need a one-time or exploratory workflow, or when you want to prototype a task graph before you write it as code. Start here before you move to Declarative Automation Bundles for production.

Switch to Declarative Automation Bundles when you need to version control the job and deploy it across environments, or when you want to define the pipelines, schemas, and compute alongside the job.

Define jobs as code

For production workflows, define jobs in Declarative Automation Bundles together with the pipelines and other resources that they run, so that you version control every change and deploy the same job to development, staging, and production. See Operations and CI/CD.

Troubleshooting

Use this entry if a task can't find its pipeline or notebook.

The pipeline or notebook doesn't appear when you configure the task

Confirm that you have the CAN RUN, CAN MANAGE, or IS OWNER permission on the pipeline, and that you can open the notebook path in the workspace. See Manage identities, permissions, and privileges for pipelines and Notebook task for jobs.

Next steps

Resource Description
Answer business questions with Genie Ontology Turn your silver tables into governed metrics and a Genie Agent for business users
Lakeflow Jobs Learn about job configuration, task types, triggers, and monitoring