ADF to execute dynamic databricks notebook

Question

Hello,

in ADF, I have a pipeline which executes a databricks notebook.
In the settings tab of the notebook in ADF, in the notebookPath I have placed the path as follows:
/Users/myusername/company/xyz/notebookName

The above path works fine and the ADF pipeline executes fully.

But if I change the above path to have a variable for the notebookPath as follows then ADF pipeline errors and says notebook not found...

@concat('/Users/myusername/company/xyz/',pipeline().parameters.p_table_name)

Answer 1

Hi @arkiboys

Thanks for the question and using MS Q&A platform.

It looks like you're trying to execute a dynamic Databricks notebook in your ADF pipeline.

The issue you're facing is due to the way you're referencing the parameter in the notebookPath field.

Instead of using @concat, try using the @{...} syntax to reference the parameter and concatenate it with the rest of the path. Here's the correct way to do it:

In the Notebook Path field, enter:

@{concat('/Users/myusername/company/xyz/', pipeline().parameters.p_table_name)}

This should fix the issue, and the ADF pipeline should be able to execute the dynamic Databricks notebook.

Additionally, make sure that the p_table_name parameter is defined in your pipeline parameters, and you're passing the correct notebook name as a value when you execute the pipeline.

If you're still facing issues, please check the following:

1. Verify that the notebook exists in the specified path with the correct name.
2. Ensure that the ADF pipeline has the necessary permissions to access the notebook.
3. Check the pipeline execution logs for more detailed error messages.

Hope this helps. Do let us know if you any further queries.

---

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.