Why Data Factory only recognize Python scripts from dbfs into Databricks, and not from repos?

Bruna dos Santos Almeida 25

When you create a process for a Databricks python script processe in Data Factory Pipeline there's only option for dbfs source. Futher, If you have a python scripts from repos, Data Factory only reconize if you change to notebook type.

User's image

Doc: https://learn.microsoft.com/en-us/azure/data-factory/transform-data-databricks-python#add-a-python-activity-for-azure-databricks-to-a-pipeline-with-ui

phemanth 10,250 Reputation points Microsoft Vendor

2023-09-21T08:27:00.1066667+00:00

@Bruna dos Santos Almeida Just checking in to see if the below answer provided by @Amira Bedhiafi helped.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Accepted answer

Amira Bedhiafi 23,486 Reputation points

2023-09-20T19:10:36.38+00:00

Repos is relatively new compared to DBFS. Microsoft and Databricks might be in the process of improving the integration, but it can take time for new features to be supported in integrated products like ADF.

We all agree that the primary use case for ADF is data orchestration. Directly running scripts or notebooks might be a common task, but navigating and deciding which version of the script to run from a Git repo could introduce complexities that don't align with the primary purpose of ADF.
Please sign in to rate this answer.
Bruna dos Santos Almeida 25 Reputation points

2023-09-21T12:42:32.6366667+00:00

Hi Amira, thanks for the explanation.

The curious point is that if you create a new notebook instead of script python, ADF reconize from the repos folder into Databricks worspace. So, the complexities to managed version has already been set with the notebook.

Also for running a notebook file requires a jupyter and python interpretador, Although for running a python script only requires python interpretador, which cause less complexities than running a notebook file.

Amira Bedhiafi 23,486 Reputation points

2023-09-21T14:13:50.8633333+00:00

The fact that ADF recognizes new notebooks from the Databricks Repos folder but not Python scripts indicates that some integration and version management has been implemented for notebooks. I think it is a feature gap in the integration between ADF and Databricks or an intentional design choice, although the reasoning behind such a choice would be subject to speculation.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Your answer

Share via

Why Data Factory only recognize Python scripts from dbfs into Databricks, and not from repos?

0 additional answers

Your answer