How to move Apache Airfow Dags to Azure ?

Question

How to move Apache Airflow Dags to Azure ?

I have Apache Airflow DAGs which are running on GCP and have python code in it and would like to know how can I move this to Azure .

Is there a ETL tool which will convert DAG to some other ETL program?

Should the solution be shifted as is or must be re-architected if so what would be tool of choice?

Answer 1

Hi @Mudassar ,

Welcome to Microsoft Q&A forum and thanks for posting your query.

In Azure, ADF (Azure Data Factory) is the cloud-based ETL and data integration service permitting data-driven work processes for arranging data development and changing data at scale. With Azure Data Factory, pipelines (schedule data-driven workflows) can ingest data from unique data stores. You can assemble complex ETL measures that change data outwardly with information streams or by utilizing register administrations like Azure HDInsight Hadoop, Azure Databricks, and Azure SQL Database.

As per your requirement, I believe the service you are looking for might be Azure Databricks as Airflow provides tight integration between Azure Databricks and Airflow. The Airflow Azure Databricks integration lets you take advantage of the optimized Spark engine offered by Azure Databricks with the scheduling features of Airflow.

For more details about how to install the Airflow Azure Databricks integration, configuring a Databricks connection, please refer to this article : Managing dependencies in data pipelines

Also please refer to this article and see if it can be of help for your requirement : Deploying Apache Airflow in Azure to build and run data pipelines

Hope this info helps.

----------

Please don’t forget to Accept Answer and Up-Vote wherever the information provided helps you, this can be beneficial to other community members.