Azure Fabric - Airflow feature in vnet internal mode

Question

Azure Fabric - Airflow feature in vnet internal mode

Mohammed Thahif BK 346 Microsoft Employee

Hello, we have a requirement to deploy Apache Airflow on Azure and considering couple of options as part of this.

Airflow on AKS
Airflow on WebApp
managed Airflow on fabric or ADF

Our requirement is to run workflows (tasks as part of DAG) on series of Azure VMs. Does Azure managed Airflow ( Datafactory or fabric based) supports running DAG workflows on Azure VMs?

I know PaaS offering being public in nature, does it allow us to connect to Azure VMs? can someone please help?

Venkat Reddy Navari 3,630 Reputation points Microsoft External Staff Moderator

2025-03-25T09:52:23.0233333+00:00

@Mohammed Thahif BK Just checking in to see if the below answer provided by @Amira Bedhiafi helped.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

1 answer

Your answer

Venkat Reddy Navari 3,630 Reputation points Microsoft External Staff Moderator

2025-03-25T09:52:23.0233333+00:00

@Mohammed Thahif BK Just checking in to see if the below answer provided by @Amira Bedhiafi helped.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Answer 1

Amira Bedhiafi 34,651 Volunteer Moderator

Hi Mohammed,

Thank you for posting on Microsoft Learn.

Managed Airflow in ADF or Microsoft Fabric is a PaaS offering. That means it runs in Microsoft-managed infrastructure, but it can be integrated with a VNet using Managed VNet integration.

The main pros :

Fully managed, scalable, and monitored.
Integrated into the Azure ecosystem (ADF pipelines, triggers, logging).
Lower operational overhead.

Can Managed Airflow run tasks on Azure VMs ?

Direct execution on VMs: Not out-of-the-box.
Remote execution via SSH, APIs, or agents: Yes, but you must configure DAGs to connect (for example you can use BashOperator or SSHOperator).

ADF Managed Airflow can run in a managed VNet, which can be linked to your own VNet using Private Endpoints or VNet peering.

Once integrated, your DAGs can securely connect to Azure VMs, databases, or other resources inside the VNet.

You must ensure DNS resolution and routing is set up properly, and the VMs must allow access from the ADF Managed VNet.

If you want maximum control, Airflow on AKS gives:

Full VNet integration
Control over how and where the tasks run (you can use KubernetesPodOperator or SSH)
Easier to use workers inside your VNet, including Azure VMs.

But, don't forget that it requires managing and maintaining AKS, Airflow Helm chart, scaling...

Venkat Reddy Navari 3,630 Reputation points Microsoft External Staff Moderator

2025-03-26T09:17:06.17+00:00

@Mohammed Thahif BK Following up to see if the above answer was helpful. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.
Assaf L 292 Reputation points

2025-04-23T15:48:07.46+00:00

@Amira Bedhiafi hi,

ADF Managed Airflow can run in a managed VNet, which can be linked to your own VNet using Private Endpoints or VNet peering.

Is there a similar feature support for the Fabric Airflow job? I couldn't find any documentation regarding it
Venkat Reddy Navari 3,630 Reputation points Microsoft External Staff Moderator

2025-04-23T16:12:28.7333333+00:00
@Assaf L As of now, Airflow jobs in Microsoft Fabric do not support VNet integration similar to Azure Data Factory’s Managed Airflow. There is currently no public documentation or feature availability confirming support for Private Endpoints, VNet peering, or Managed VNet integration within Microsoft Fabric's Airflow offering.

If secure communication with Azure VMs or other resources inside your private VNet is a requirement, you may consider:

Using ADF Managed Airflow, which supports Managed VNet and private connectivity.

Deploying Airflow on AKS, which gives full control over networking and access to resources in your VNet.

We recommend keeping an eye on the Microsoft Fabric documentation for future updates, as networking capabilities may evolve over time.

Hope this helps. Do let us know if you any further queries.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.
Venkat Reddy Navari 3,630 Reputation points Microsoft External Staff Moderator

2025-04-24T09:52:26.3833333+00:00

@Assaf L Following up to see if the above answer was helpful. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.
Assaf L 292 Reputation points

2025-04-24T21:57:01.22+00:00

The answer was helpful, but I cannot accept the answer since I didn't ask the root question

Thanks,

Share via

Azure Fabric - Airflow feature in vnet internal mode

1 answer

Your answer