Episode
Change Data Capture and Managed Airflow in Azure Data Factory
with Scott Hanselman, Abhishek Narain, Mark Kromer
Mark Kromer and Abhishek Narain join Scott Hanselman to talk about two new capabilities in Azure Data Factory: Change Data Capture (CDC) and Managed Airflow. Change Data Capture in Azure Data Factory automatically detects data changes at the source without requiring complex designing or coding. Managed Airflow in Azure Data Factory is a managed orchestration service for Apache Airflow that simplifies the creation and management of Airflow environments on which you can operate end-to-end data pipelines at scale.
Chapters
- 00:00 - Introduction
- 00:40 - Change Data Capture
- 01:17 - CDC demo
- 07:59 - Managed Airflow
- 09:06 - Managed Airflow demo
- 14:33 - Wrap-up
Recommended resources
- Change data capture resource overview
- How to capture changed data from ADLS Gen2 to Azure SQL DB using a Change Data Capture (CDC) resource
- What is Azure Data Factory Managed Airflow?
- How does Azure Data Factory Managed Airflow work?
- Azure Data Factory
- Create a Pay-as-You-Go account (Azure)
- Create a free account (Azure)
Connect
- Scott Hanselman | Twitter: @SHanselman
- Mark Kromer | Twitter: @KromerBigData
- Abhishek Narain | Twitter: @NarainAbhishek
- Azure Data Studio | Twitter: @AzDataFactory
- Azure SQL | Twitter: @AzureSQL
- Azure Friday | Twitter: @AzureFriday
- Azure | Twitter: @Azure
Mark Kromer and Abhishek Narain join Scott Hanselman to talk about two new capabilities in Azure Data Factory: Change Data Capture (CDC) and Managed Airflow. Change Data Capture in Azure Data Factory automatically detects data changes at the source without requiring complex designing or coding. Managed Airflow in Azure Data Factory is a managed orchestration service for Apache Airflow that simplifies the creation and management of Airflow environments on which you can operate end-to-end data pipelines at scale.
Chapters
- 00:00 - Introduction
- 00:40 - Change Data Capture
- 01:17 - CDC demo
- 07:59 - Managed Airflow
- 09:06 - Managed Airflow demo
- 14:33 - Wrap-up
Recommended resources
- Change data capture resource overview
- How to capture changed data from ADLS Gen2 to Azure SQL DB using a Change Data Capture (CDC) resource
- What is Azure Data Factory Managed Airflow?
- How does Azure Data Factory Managed Airflow work?
- Azure Data Factory
- Create a Pay-as-You-Go account (Azure)
- Create a free account (Azure)
Connect
- Scott Hanselman | Twitter: @SHanselman
- Mark Kromer | Twitter: @KromerBigData
- Abhishek Narain | Twitter: @NarainAbhishek
- Azure Data Studio | Twitter: @AzDataFactory
- Azure SQL | Twitter: @AzureSQL
- Azure Friday | Twitter: @AzureFriday
- Azure | Twitter: @Azure
Have feedback? Submit an issue here.