Support of Multiselect List in Azure Data Factory parameters

Question

Support of Multiselect List in Azure Data Factory parameters

Ravindra Raghuvanshi (Consultant) 20

Hello Team,

We are working on a scenario where we want to provide a capability to user to select some values (Multiple) from a predefined list of values. And as a result, the data factory should run only for the selected values and not for all the values present in the list.

For example: I have a list of fruits like Apple, Banana, Papaya, Orange etc.

Now user is triggering ADF pipeline, and he/she can select 1 or more fruits from above list and pipeline will work only for that list of selection.

We are using databricks too in this pipeline and need to pass the selection to databricks notebook as well. Do we have support for the same in ADF?

Something like below:

User's image

Accepted answer

1 additional answer

Your answer

Answer 1

ShaikMaheer-MSFT 38,546 Microsoft Employee Moderator

Hi Ravindra Raghuvanshi (Consultant),

Thank you for posting query in Microsoft Q&A Platform.

Please note, ADF cannot provide UI to popup values to select. So you need consider writing your own code to create a UI to show up values. And then In the code behind of UI you need to capture selected values and then consider making API call to ADF pipeline to start running pipeline. Here, your ADF pipeline should have parameters in it to take the values from REST API. And in REST API you should pass selected values to API while making ADF pipeline run call.

Please check below documentation of ADF REST APIs.

https://learn.microsoft.com/en-us/rest/api/datafactory/pipelines

Hope this helps. Please let me know if any further queries.

Please consider hitting Accept Answer button. Accepted answers help community as well. Thank you.

Ravindra Raghuvanshi (Consultant) 20 Reputation points

2023-07-10T07:16:26.71+00:00

Hi @ShaikMaheer-MSFT

Thanks for responding i got your answer however i just want to check any specific reason this support is not available in ADF or any future plan to have this in ADF.
ShaikMaheer-MSFT 38,546 Reputation points Microsoft Employee Moderator

2023-07-10T07:34:18.9333333+00:00

Hi Ravindra Raghuvanshi (Consultant), technically ADF is ETL cloud tool. So it will not have any UI related development capabilities. Thank you.

Hope this helps.

Please consider hitting Accept Answer button. Accepted answers help community as well. Thank you.

Answer 2

RevelinoB 3,675

Hi Ravindra,

Yes, Azure Data Factory (ADF) provides support for parameterizing and passing values to activities, including passing values to Databricks notebooks. You can achieve the scenario you described by using parameters and expressions in ADF.

Here's how you can implement it:

Define a parameter in your ADF pipeline to capture the user's fruit selection. Let's call it SelectedFruits. Set the type of the parameter to Array.
Create a dataset that contains the list of fruits (e.g., Apple, Banana, Papaya, Orange). This dataset can be an input dataset or a linked service.

Configure the activity in your ADF pipeline (such as Databricks notebook activity) to use the SelectedFruits parameter as a filter. You can use the @activity('PreviousActivityName').output.value expression to reference the output of a previous activity and pass it as a parameter value.

In your Databricks notebook, you can access the selected fruits using the dbutils.widgets.get method. You can retrieve the parameter value using dbutils.widgets.get('SelectedFruits').

Make sure you specify the appropriate paths and names for your Databricks notebook and linked service.

By using this approach, the user can select one or more fruits as parameters in the ADF pipeline, and the selected fruits will be passed to the Databricks notebook for further processing.

I hope this helps?

Ravindra Raghuvanshi (Consultant) 20 Reputation points

2023-07-07T07:00:54.12+00:00

Thanks, Revoline for quick answer.

Let me try if this is possible.
Ravindra Raghuvanshi (Consultant) 20 Reputation points

2023-07-07T07:05:55.01+00:00

Thanks Revoline let me try this.
RevelinoB 3,675 Reputation points

2023-07-07T07:14:41.7633333+00:00
Hi Ravindra,

Apologies for any confusion. In the context of your scenario, you don't necessarily need a dataset to capture the list of fruits for user selection. Instead, you can define the list of fruits as an array parameter directly in the ADF pipeline.

Here's an updated approach:

Define an array parameter in your ADF pipeline to capture the user's fruit selection. Let's call it SelectedFruits.

Configure the activity in your ADF pipeline (such as Databricks notebook activity) to use the SelectedFruits parameter as a filter. You can use the @pipeline().parameters.SelectedFruits expression to reference the parameter value.

In your Databricks notebook, you can access the selected fruits using the dbutils.widgets.get method. You can retrieve the parameter value using dbutils.widgets.get('SelectedFruits').

Here's an example of how you can set up the parameterized activity in ADF:

Define the Pipeline parameter

"parameters": { "SelectedFruits": { "type": "Array" } }

Configure the activity (e.g., Databricks notebook activity) to use the SelectedFruits parameter as a filter:

"activities": [ { "name": "DatabricksActivity", "type": "DatabricksNotebook", "linkedServiceName": { "referenceName": "DatabricksLinkedService", "type": "LinkedServiceReference" }, "typeProperties": { "rootPath": "/path/to/notebook", "notebookPath": "/path/to/notebook", "baseParameters": { "SelectedFruits": { "value": "@pipeline().parameters.SelectedFruits" } } } } ]

By using this approach, you can directly capture the user's fruit selection as an array parameter in the ADF pipeline, and pass it to the Databricks notebook for further processing. You don't need to create a separate dataset to define the list of fruits.

I hope this helps?
Ravindra Raghuvanshi (Consultant) 20 Reputation points

2023-07-07T07:22:45.79+00:00

Hi Revelino, in above mentioned approach the SelectedFruits will be a kind of static array and user will not be able to see all the possible values to select. He is not at all aware about what all values he can choose because we are not showing all values instead, we are asking user to set fruit values in parameters and passing it to databricks.

So, this might not solve the selection problem.
RevelinoB 3,675 Reputation points

2023-07-07T07:37:19.87+00:00
Hi Ravindra,

Apologies for any confusion caused. To clarify, you don't need to create a separate dataset to define the list of fruits for user selection. Instead, you can directly define the list of fruits as an array parameter in your Azure Data Factory (ADF) pipeline.

Here's an example of how you can set it up:

Define an array parameter in your ADF pipeline to capture the user's fruit selection. Let's call it SelectedFruits. You can define this parameter in the parameters section of your pipeline's JSON definition.

"parameters": { "SelectedFruits": { "type": "Array" } }

Configure the activity in your ADF pipeline (such as Databricks notebook activity) to use the SelectedFruits parameter as a filter. You can reference the parameter value using the @pipeline().parameters.SelectedFruits expression.

For example, in the typeProperties section of your Databricks notebook activity, you can define a baseParameters section and provide the value of the SelectedFruits parameter:

"typeProperties": { "rootPath": "/path/to/notebook", "notebookPath": "/path/to/notebook", "baseParameters": { "SelectedFruits": { "value": "@pipeline().parameters.SelectedFruits" } } }

In your Databricks notebook, you can access the selected fruits using the dbutils.widgets.get method. This method allows you to retrieve the value of the SelectedFruits parameter.

For example, in a Python cell within your Databricks notebook, you can retrieve the selected fruits as follows:

selected_fruits = dbutils.widgets.get("SelectedFruits")

By following this approach, you can directly capture the user's fruit selection as an array parameter in the ADF pipeline, and pass it to the Databricks notebook for further processing. You do not need to create a separate dataset or lookup activity for the user to select the fruits. The user will provide the fruit selection when triggering the pipeline, and that selection will be passed as a parameter to the Databricks notebook.

I hope this answers your query?
Ravindra Raghuvanshi (Consultant) 20 Reputation points

2023-07-07T11:26:55.7766667+00:00

Hi Revelino, in above mentioned approach the SelectedFruits will be a kind of static array and user will not be able to see all the possible values to select. He is not at all aware about what all values he can choose because we are not showing all values instead, we are asking user to set fruit values in parameters and passing it to databricks.

So, this might not solve the selection problem.

Share via

Support of Multiselect List in Azure Data Factory parameters

1 additional answer

Your answer