Production readiness for Lakeflow pipelines

Production readiness is the point where a pipeline can run unattended against real business data, and this page gives you a checklist to confirm you're there before you rely on it.

Overview

A production-ready pipeline runs on a schedule, against real business data, with failures caught automatically rather than reported by a downstream consumer. Readiness is a checklist across data quality, reliability, observability, deployment, cost, and governance.

Lakeflow pipelines give you several built-in primitives that map directly onto these dimensions. Rather than trying to intuit whether you're ready, walk through the checklist below and treat any unchecked box as a known gap. If most boxes are checked, the pipeline is ready to run in production. If several are unchecked, treat those as your prioritized to-do list. Start with data quality and notifications, since those are the cheapest to add and the most likely to catch an undetected bad run.

Checklist

Walk through the following checklist across each dimension of production readiness, and treat any unchecked box as a known gap to close before you rely on the pipeline.

Data quality

  • [ ] Every dataset that can receive bad data has at least one expectation (expect, expect_or_drop, or expect_or_fail) defined, not just a docstring comment saying "assume clean input."
  • [ ] You've deliberately chosen warn versus drop versus fail per constraint: fail for conditions that should stop the world (such as a broken primary key), drop for records you can safely discard (with a quarantine table capturing what was dropped), and warn only where you actively watch the trend.
  • [ ] You've looked at the Data quality tab in the pipeline UI, or queried the event log at least once, and know your current expectation pass rate, not just that expectations exist. See Manage data quality with pipeline expectations.

Reliability

  • [ ] You've consciously chosen between triggered and continuous pipeline mode. Triggered is the right default for the vast majority of pipelines. Reserve continuous for genuine sub-minute latency needs, since it requires an always-on cluster. See Triggered vs. continuous pipeline mode.
  • [ ] The pipeline is scheduled through the job scheduler or a Lakeflow Job rather than someone remembering to select Start. See Run pipelines in a workflow.
  • [ ] Failure notifications are configured for email, a webhook, or an event hook so you find out about a broken run before your stakeholders do.
  • [ ] For streaming sources, you've tested what happens on a checkpoint failure and know the recovery procedure rather than discovering it during an incident. See Recover a pipeline from streaming checkpoint failure.
  • [ ] The pipeline runs as a service principal, not a personal user identity, making runs more secure and stable for unattended automation.

Observability

  • [ ] You know where the event log lives and have run at least one query against it for lineage, data-quality metrics, or resource usage. The event log is the primary observability primitive for pipelines, and the pipeline UI is a view over it. See Monitor pipelines.
  • [ ] You've checked system.lakeflow.pipelines and related system tables, or built a small dashboard, so pipeline health is queryable, not just eyeballed in the UI.
  • [ ] You know your current update duration and can tell whether it's trending up or holding steady update over update.

Deployment and change management

  • [ ] The pipeline is defined and deployed through a bundle, not hand-clicked into existence in the UI, so it can be code-reviewed and version-controlled. See Create a source-controlled pipeline.
  • [ ] You have at least a dev and a prod target (ideally dev, staging, and prod) so changes get validated somewhere before they hit real data.
  • [ ] Environment-specific values (catalog names, schedules, source paths) are parameterized through pipeline configuration rather than hardcoded, so the same code promotes cleanly across targets.

Compute and cost

  • [ ] You've made an explicit choice between serverless and classic compute (serverless is the default recommendation for new pipelines) and documented why, if you chose classic. See Configure a serverless pipeline.
  • [ ] If on classic compute, enhanced autoscaling is enabled rather than a fixed cluster size, so the pipeline doesn't silently starve for workers under load.
  • [ ] You've looked at system.billing.usage at least once for this pipeline's Databricks unit (DBU) consumption and it isn't a surprise number.

Governance

  • [ ] Target tables live in Unity Catalog with an intentional catalog and schema layout (for example, separate catalogs or schemas per environment), not the legacy Hive metastore.
  • [ ] Access to source and target objects follows least privilege: the pipeline's service principal can read and write what it needs and nothing more. See Manage identities, permissions, and privileges for pipelines.

Additional resources

  • Last updated on