Azure Fabric - Airflow feature in vnet internal mode

Mohammed Thahif BK 346 Reputation points Microsoft Employee
2025-03-24T05:35:09.1+00:00

Hello, we have a requirement to deploy Apache Airflow on Azure and considering couple of options as part of this.

  1. Airflow on AKS
  2. Airflow on WebApp
  3. managed Airflow on fabric or ADF

Our requirement is to run workflows (tasks as part of DAG) on series of Azure VMs. Does Azure managed Airflow ( Datafactory or fabric based) supports running DAG workflows on Azure VMs?

I know PaaS offering being public in nature, does it allow us to connect to Azure VMs? can someone please help?

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,667 questions
{count} votes

1 answer

Sort by: Most helpful
  1. Amira Bedhiafi 34,651 Reputation points Volunteer Moderator
    2025-03-24T11:27:03.73+00:00

    Hi Mohammed,

    Thank you for posting on Microsoft Learn.

    Managed Airflow in ADF or Microsoft Fabric is a PaaS offering. That means it runs in Microsoft-managed infrastructure, but it can be integrated with a VNet using Managed VNet integration.

    The main pros :

    • Fully managed, scalable, and monitored.
    • Integrated into the Azure ecosystem (ADF pipelines, triggers, logging).
    • Lower operational overhead.

    Can Managed Airflow run tasks on Azure VMs ?

    • Direct execution on VMs: Not out-of-the-box.
    • Remote execution via SSH, APIs, or agents: Yes, but you must configure DAGs to connect (for example you can use BashOperator or SSHOperator).

    ADF Managed Airflow can run in a managed VNet, which can be linked to your own VNet using Private Endpoints or VNet peering.

    Once integrated, your DAGs can securely connect to Azure VMs, databases, or other resources inside the VNet.

    You must ensure DNS resolution and routing is set up properly, and the VMs must allow access from the ADF Managed VNet.

    If you want maximum control, Airflow on AKS gives:

    • Full VNet integration
    • Control over how and where the tasks run (you can use KubernetesPodOperator or SSH)
    • Easier to use workers inside your VNet, including Azure VMs.

    But, don't forget that it requires managing and maintaining AKS, Airflow Helm chart, scaling...


Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.