Understanding the data orchestration capability

Note

Data playbook capabilities: The data playbook defines a set of capabilities that represent conceptual building blocks that are used to build data-related solutions. See Defining data capabilities to see the full set of capabilities defined in the playbook.

Data orchestration is an automated process for processing and organizing data from one or more sources and making it available for analysis. Other common references to data orchestration are - data pipeline orchestration, data workflow orchestration.

Data orchestration encompasses various aspects of data management, including data governance, quality management, and automating job combinations, in addition to handling data movement and transformation like Extract, Transform, and Load (ETL) operations. Data orchestration enables organizations to fully leverage their data by providing a unified view for analysis, reporting, and decision-making. Depending on the specific needs, it's typical to find multiple orchestration tools being used within a data estate.

Refer to Understanding data orchestration characteristics section to note the common steps associated with data orchestration.

Understanding data orchestration characteristics

Factors like data source support, data volume, latency, budget, and type of tool (visual/low-code, code-based, open source, etc.) should be considered while selecting the orchestration tool. Some of the key steps involved in data orchestration include:

  • Integration of data from multiple sources similar to ETL.
  • Reducing the need for manual intervention and increasing efficiency using automated tools and technologies to manage and coordinate data across different systems and applications.
  • Scheduling jobs/tasks, managing workflows, and handling dependencies among jobs.

Refer to choosing data orchestrator for some key features to consider.

Learn more about data orchestration in Microsoft Fabric

Here are a few ways to orchestrate data in Microsoft Fabric:

Refer to decision guide for a comparison of different orchestration options.

Examples

For more information