ADF Not Passing Parameters to Databricks Job as Expected

Question

ADF Not Passing Parameters to Databricks Job as Expected

Dima Frank 0

Hello,

I'm encountering an issue where Azure Data Factory (ADF) does not seem to pass parameters correctly to a Databricks job.

Here's the situation:

I have an ADF pipeline that uses the "Databricks Job activity to trigger a Databricks job.
I am passing a parameter called entity_name in the jobParameters section of the ADF activity JSON, like so:

User's image

On the Databricks side, the job has:

A defined job parameter named entity_name with a default value.
Despite this setup:
- The job starts, but the parameter does not appear to be passed through.
- I see the default value in the “Run parameters” on the Databricks run page.
- Is there a required setting in either ADF or Databricks that ensures parameters are correctly passed and resolved? Thanks in advance for your help. Best regards,

J N S S Kasyap 3,780 Reputation points Microsoft External Staff Moderator

2025-05-20T04:24:29.8533333+00:00

@Dima Frank
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

1 answer

Your answer

J N S S Kasyap 3,780 Reputation points Microsoft External Staff Moderator

2025-05-20T04:24:29.8533333+00:00

@Dima Frank
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

Answer 1

J N S S Kasyap 3,780 Microsoft External Staff Moderator

Hi @Dima Frank

Is there a required setting in either ADF or Databricks that ensures parameters are correctly passed and resolved?

There are required settings in both ADF and Databricks to ensure that parameters are correctly passed and resolved. These settings must align on both sides for parameter passing to work.

The parameter name in ADF (entity_name) must exactly match the widget name in Databricks notebook should be case-sensitive.

dbutils.widgets.text("entity_name", "")
entity_name = dbutils.widgets.get("entity_name")

Please refer the below Microsoft document for more information
https://learn.microsoft.com/en-us/azure/data-factory/transform-data-using-databricks-notebook

I hope this information helps. Please do let us know if you have any further queries.

Kindly consider upvoting the comment if the information provided is helpful. This can assist other community members in resolving similar issues.

Thank you.

Dima Frank 0 Reputation points

2025-05-19T10:25:13.13+00:00
Hi J N S S Kasyap,

Thank you for your response.

However, I believe there may have been a misunderstanding. My question is about the new Databricks Job activity in Azure Data Factory (ADF), not the traditional Notebook activity using widgets (dbutils.widgets). I am triggering a Databricks Job — not a notebook directly — and I'm using the jobParameters property in the ADF activity to pass parameters to the job.

To clarify:

In ADF, I'm using the Databricks Job activity, where I provide jobId and jobParameters in the activity configuration.

In Databricks, the Job has a parameter called entity_name (exactly the same name as in ADF Job Activity).

The parameter is not passed through — the job runs with the default value instead.

My question is whether there is a required configuration or known issue that would prevent jobParameters from being passed correctly from ADF to Databricks when using the Databricks Job activity (not notebook widgets).

I appreciate your support, but could you please advise specifically in the context of the Databricks Job activity in ADF and Databricks job parameters, not widgets?

Thank you again, Dima Frank
J N S S Kasyap 3,780 Reputation points Microsoft External Staff Moderator

2025-05-19T11:20:06.29+00:00
@Dima Frank

Based on the information the ADF job is correctly passing the entity_name parameter as bm20_equipment, but the Databricks job is still using the default value bm20_site.

Your Azure Data Factory (ADF) pipeline is correctly passing the parameter entity_name with the value bm20_equipment, as confirmed in the input JSON of the pipeline run. The parameter name matches exactly, including case sensitivity, and the Databricks job itself executes successfully.
However, the expected value is not being used during execution, indicating a likely issue in how the job task is referencing the parameter.

By defining job parameters in Databricks, referencing them appropriately within each task using {{job.parameters.<parameter_name>}}, and passing the desired values from ADF using the jobParameters property, you ensure that your Databricks tasks receive and utilize the correct parameter values provided by ADF.

In Databricks:

Edit the task inside your job

Under "Base Parameters", make sure that value set to:
entity_name = {{job.parameters.entity_name}}
This tells the task to use the job parameter passed at runtime, not a hardcoded default.
https://docs.databricks.com/aws/en/jobs/job-parameters?

I hope this information helps. Please do let us know if you have any further queries. I am happy to help further your queries.
Valentin GAILLEZ 5 Reputation points

2025-05-27T21:34:05.2066667+00:00

I'm having exactly the same problem trying to test the new azure data factory activity, which lets you run a databricks job instead of a notebook. Impossible to pass parameters each time they are empty on the databricks side whereas adf seems to retrieve the right values. Modifying the json of the adf activity by replacing job parameters by based parameters has changed nothing, still empty.
J N S S Kasyap 3,780 Reputation points Microsoft External Staff Moderator

2025-05-28T08:13:30.0833333+00:00

@Valentin GAILLEZ, Please share the new thread with all the details, We are happy help on your task.
Pablo Aschieri 0 Reputation points

2025-06-02T12:44:44.9266667+00:00

Hi, same problem here. Any update on this? is it a bug?
Valentin GAILLEZ 5 Reputation points

2025-06-03T07:57:38.21+00:00

@J N S S Kasyap I can't share it here because it's for a private company, but the problem is exactly the same as the one presented: I can't pass parameters to the new adf activity that lets you run a databricks job. The job launches correctly but the parameters are empty every time. On the other hand, by using the databricks api from adf and the post method via an adf web activity, I'm able to pass parameters that aren't empty.
Tony Gao 5 Reputation points

2025-06-03T12:09:12.2666667+00:00

I faced the same issue here, I can see job parameters have been passed from the adf databricks job acitivty input, however, my databricks job being triggered doesn't recieve the parameters. I suspect it's a bug, please look into this issue.

Share via

ADF Not Passing Parameters to Databricks Job as Expected

1 answer

Your answer