Transition Azure Data Factory Mapping Data Flows to Azure Databricks (Scala?)

Question

As a result of some infrastructure changes that are outside of my control, my organization is shifting from the use of Azure Data Factory (including Mapping Data Flows) to Azure Databricks.

It is my understanding that ADF Mapping Data Flows functionally run on a Databricks or otherwise Spark cluster in the background. The Data Flow GUI allows for codeless development, while the Mapping Data Flow script is a dataflow-specific scripting language that allows programmatic development of these Mapping Data Flows.

However, we have considerable resources that are already built in Mapping Data Flows that need to be transitioned. To smoothly conduct this transition, is there a way to expose the language/logic that is being sent to the Spark compute cluster? Ideally in a way that can be migrated to the Azure Databricks platform easily, such as the raw Scala.

When debugging the data flow and errors occur, there's some small leakthrough and insight into this language (usually Scala errors), but I haven't seen anywhere to see the Scala in full.

Answer 1

Hello Todd Lazure,

Your understanding is correct that Azure Data Factory Mapping Data Flows functionally run on a Databricks or otherwise Spark cluster in the background. However, the Mapping Data Flow script is a dataflow-specific scripting language that allows programmatic development of these Mapping Data Flows, and this is different from the Scala language used in Databricks.

While it is possible to see some of the generated Spark code in Scala when debugging Mapping Data Flows in Azure Data Factory, this code is not publicly available in full. so, it is not possible to directly convert Mapping Data Flow code to Scala on Azure Databricks.

To transition your Mapping Data Flows to Azure Databricks, you will need to manually convert the logic and syntax used in your Mapping Data Flows to the appropriate Databricks libraries and syntax. This may involve significant modifications to your existing code, depending on the complexity of your Mapping Data Flows.

I hope this answers your question.