Fabric Data Engineering documentation

Fabric Data Engineering enables users to design, build, and maintain infrastructures and systems that enable their organizations to collect, store, process, and analyze large volumes of data.

About Data Engineering

Overview

• What is Data Engineering?
• What is a lakehouse?

Concept

• Lakehouse and Delta tables
• Workspace roles and permissions
• Choose a data store

How-To Guide

• Delta optimization and V-Order
• Load to tables

Get started with lakehouse

Get started

• Create a lakehouse
• Navigate the lakehouse explorer

How-To Guide

• Get data into lakehouse using notebook

Tutorial

• Step 1 - Create a workspace
• Step 2 - Create a lakehouse
• Step 3 - Ingest data into lakehouse
• Step 4 - Transform data into delta tables
• Step 5 - Create a semantic model and build a report
• Step 6 - Clean up resources

Get started with notebooks

Get started

• Create and use notebooks

How-To Guide

• Develop and run notebooks
• Explore lakehouse data with a notebook
• Visualize notebooks
• Notebooks source control and deployment
• Author and run T-SQL Notebooks

Query with API for GraphQL

Overview

• What is Fabric API for GraphQL?

Get started

• Create an API for GraphQL and add data

How-To Guide

• Manage relationships
• Connect to applications
• Query multiple data sources
• GraphQL source control and deployment pipelines
• Monitoring dashboard and logging (preview)

VS Code integration

Concept

• What is Data Engineering VS Code extension?

How-To Guide

• Use notebooks from VS Code
• Use container image from VS Code
• Use Spark job definitions from VS Code
• Explore lakehouse from VS Code
• Develop notebook from VS Code web experience

Apache Spark for Data Engineering

Concept

• Apache Spark job definition
• What is Apache Spark advisor?

Get started

• Create a Spark job definition
• Create a Spark job definition with a Fabric API
• Spark job definition Git integration

How-To Guide

• Schedule and run a Spark job definition
• Manage Apache Spark libraries
• How to use Fabric MSSparkUtils

Get started with environment

Get started

• Fabric environments

How-To Guide

• Manage Spark compute in environments
• Manage libraries in environments
• Environment Git integration and deployment pipeline
• Manage the environment through public APIs
• Managing libraries in environment secured by outbound access protection

Fabric Livy API

Overview

• What is the Livy API?

Get started

• Create and run Spark jobs using the Livy API
• Submit Livy API session jobs
• Submit Livy API batch jobs

Optimize with materialized lake views

Overview

• What are materialized lake views?

Get started

• Get started with materialized lake views
• Implement medallion architecture with materialized lake views
• Manage materialized lake views