Organize datasets across Lakeflow pipelines

Use Lakeflow pipelines to process anywhere from one to hundreds of datasets in a single pipeline. Decide which datasets belong together based on domain, cadence, and dependencies, and split work into separate pipelines when ownership, latency, or scale differ.

Overview

A decision that never comes up in a tutorial becomes important in real deployments: which tables belong together in the same pipeline, and when should something be its own pipeline instead? Getting this wrong is one of the most common ways teams paint themselves into a corner. The classic failure mode is dumping everything into one giant pipeline and later hitting scaling, concurrency, and blast-radius problems that are painful to unwind.

There's no single right answer, but there are clear forces pulling in each direction and one hard constraint worth knowing before you design.

Guidelines

Group datasets by domain, shared cadence, and dependency, and split them at ownership, layer, and latency boundaries. Keep any single pipeline's count of independently updatable datasets comfortably under the parallel-update ceiling.

Know the concurrency limit before you design

A single triggered pipeline update runs at most 16 dataset updates in parallel. This is the most common surprise for teams who put everything in one pipeline: once you have more than roughly 16 datasets that could otherwise run concurrently, the extra ones queue behind the first 16 rather than running in parallel, so total update time stretches out even though there's compute available. If you have dozens of independent datasets and you care about wall-clock update time, that alone is a reason not to cram them all into one pipeline.

What belongs in the same pipeline

Keep datasets together when they share structure or scheduling:

  • Datasets that form one dependency chain or one logical domain, such as the bronze, silver, and gold tables for orders. Keeping a connected directed acyclic graph (DAG) together lets the pipeline schedule it, checkpoint it, and full-refresh it as a coherent unit, and keeps lineage readable. See Load and process data incrementally with Lakeflow pipeline flows.
  • Datasets that share the same freshness requirement and run cadence, such as things that should update together, on the same trigger, in the same transaction boundary.
  • Datasets small enough in aggregate that the whole graph comfortably fits under the parallel-update ceiling and refreshes in an acceptable time.

What belongs in a separate pipeline

Split datasets apart when they differ in ownership, layer, or latency:

  • Different domains or teams. Separate ownership should usually mean separate pipelines, so one team's change or failure doesn't block another's.
  • Layers you want to scale or schedule independently. A widely recommended split is to separate ingestion (bronze) from transformation (silver and gold) into distinct pipelines, so a slow or failing ingestion doesn't hold up transformation and each can size compute to its own needs.
  • Different latency profiles. A continuous, low-latency stream shouldn't share a pipeline with a once-a-day batch aggregate. See Triggered vs. continuous pipeline mode.
  • Datasets that push you past the parallel-update limit and would otherwise queue.

To run a set of datasets in isolation, consider a standalone pipeline. See Standalone pipelines vs. Lakeflow pipelines.

A practical rule of thumb

Don't default to one monolithic pipeline, and don't shatter every table into its own pipeline either. Group by domain + shared cadence + dependency, split at ownership, layer, and latency boundaries, and keep any single pipeline's independently updatable dataset count comfortably under the parallel-update ceiling. When in doubt, prefer a handful of medium pipelines aligned to domains over one giant pipeline. It's far easier to merge two small pipelines later than to carve up a monolith that's already in production.

Limitations

  • A single triggered update runs at most 16 dataset updates in parallel. Datasets beyond that ceiling queue rather than running concurrently, so a pipeline with dozens of independent datasets can take longer to update even when compute is available.
  • Splitting into multiple pipelines costs some end-to-end visibility. When you split, lean on the system tables (system.lakeflow.pipelines, system.lakeflow.job_run_timeline) and orchestrate the pieces together with a Lakeflow Job so you still get a single end-to-end view of the whole flow. See Run pipelines in a workflow.

Additional resources

  • Last updated on