Episode

Microsoft Fabric Learn Together Ep08: Ingest data with Spark and Microsoft Fabric notebooks

with Johan Ludvig Brattås, Heini Ilmarinen, Marcel Magalhães, Olivier Van Steenlandt

Learn Live

Discover how to use Apache Spark and Python for data ingestion into a Microsoft Fabric lakehouse. Fabric notebooks provide a scalable and systematic solution.

Learning objectives

  • Ingest external data to Fabric lakehouses using Spark
  • Configure external source authentication and optimization
  • Load data into lakehouse as files or as Delta tables

Chapters

  • 00:00 - Introduction
  • 05:48 - Learning objectives
  • 06:21 - Context in Fabric
  • 10:31 - What is Spark?
  • 14:54 - An introduction to notebooks
  • 16:42 - Explore Fabric notebooks
  • 22:28 - What is a lakehouse?
  • 25:44 - Fabric lakehouse vs Data warehouse
  • 28:47 - Write data into a lakehouse
  • 30:57 - Write to a Delta table
  • 34:13 - Consider uses for ingested data
  • 42:01 - Exercise - Ingest data with Spark and Microsoft Fabric notebooks
  • 01:04:51 - Knowledge check
  • 01:13:24 - Summary

Connect

Intermediate
Data Analyst
Data Engineer
Data Scientist
Microsoft Fabric