When you create a process for a Databricks python script processe in Data Factory Pipeline there's only option for dbfs source. Futher, If you have a python scripts from repos, Data Factory only reconize if you change to notebook type. Doc: https://learn.microsoft.com/en-us/azure/data-factory/transform-data-databricks-python#add-a-python-activity-for-azure-databricks-to-a-pipeline-with-ui

Repos is relatively new compared to DBFS. Microsoft and Databricks might be in the process of improving the integration, but it can take time for new features to be supported in integrated products like ADF. We all agree that the primary use case for ADF is data orchestration. Directly running scripts or notebooks might be a common task, but navigating and deciding which version of the script to run from a Git repo could introduce complexities that don't align with the primary purpose of ADF.

Why Data Factory only recognize Python scripts from dbfs into Databricks, and not from repos?

Accepted answer

Amira Bedhiafi 20,571 Reputation points

2023-09-20T19:10:36.38+00:00

Repos is relatively new compared to DBFS. Microsoft and Databricks might be in the process of improving the integration, but it can take time for new features to be supported in integrated products like ADF.

We all agree that the primary use case for ADF is data orchestration. Directly running scripts or notebooks might be a common task, but navigating and deciding which version of the script to run from a Git repo could introduce complexities that don't align with the primary purpose of ADF.
Please sign in to rate this answer.
Bruna dos Santos Almeida 25 Reputation points

2023-09-21T12:42:32.6366667+00:00

Hi Amira, thanks for the explanation.

The curious point is that if you create a new notebook instead of script python, ADF reconize from the repos folder into Databricks worspace. So, the complexities to managed version has already been set with the notebook.

Also for running a notebook file requires a jupyter and python interpretador, Although for running a python script only requires python interpretador, which cause less complexities than running a notebook file.

Amira Bedhiafi 20,571 Reputation points

2023-09-21T14:13:50.8633333+00:00

The fact that ADF recognizes new notebooks from the Databricks Repos folder but not Python scripts indicates that some integration and version management has been implemented for notebooks. I think it is a feature gap in the integration between ADF and Databricks or an intentional design choice, although the reasoning behind such a choice would be subject to speculation.
Sign in to comment

Share via

Why Data Factory only recognize Python scripts from dbfs into Databricks, and not from repos?

0 additional answers