ELT frameworks (Azure)

Question

Generally speaking, which Azure data platform is most typically utilized for data processing in an ELT framework?
Azure Data Factory or Azure Databricks...

Answer 1

ADF and Databricks are commonly used for ETL or ELT solutions, and the decision-making process depends on the specific scenarios that need to be implemented. In most cases, a hybrid approach is utilized, combining ADF with custom code using Azure Databricks or Azure Functions/Batch for certain functionalities that are not yet available in ADF. However, in some exceptional cases where dedicated clusters are necessary, ADB may be preferred over ADF since ADF dataflows generate clusters at runtime with each execution. Over the next few years, ADLA and ADLS will be phased out.