Passing Parameters from ADF to ML Designer Pipeline

Question

Passing Parameters from ADF to ML Designer Pipeline

Ajay Jeevan Jose 0

I have created an ML pipeline using Azure ML Designer, which includes an Execute Python Script component. This component is linked to a script bundle that processes a blob from Azure Blob Storage and saves the output back to the storage.

The ML pipeline is triggered using an Azure Data Factory (ADF) pipeline, and we are planning to move it to a production environment. Currently, details such as the container name, connection string, and credentials are hardcoded in the script, but I want to dynamically pass these values from ADF to the ML pipeline.

My Questions:

How can I define parameters in ML Designer and pass them from ADF?
What is the best way to access these parameters inside the Python script in the Execute Python Script component?
Are there any best practices for securely passing connection strings and credentials (e.g., via Azure Key Vault or Linked Services)?

Would appreciate any guidance or best practices on achieving this. Thanks in advance!

J N S S Kasyap 1,065 Reputation points Microsoft External Staff

2025-02-20T05:18:21.0266667+00:00
Hi @Ajay Jeevan Jose,

Thanks for posting your query!

You can integrate Azure Data Factory (ADF) with an Azure Machine Learning (AML) Designer pipeline and pass parameters dynamically by following approach to parameterize the Python script in the "Execute Python Script" component.

Defining Parameters in ML Designer and Passing Them from ADF

In Azure Machine Learning Designer, you can create parameters by going to the Parameters section in your pipeline. Give each parameter a name (like container_name or connection_string) and choose the correct data type. These parameters can then be used in components like the Execute Python Script.

To pass these parameters from Azure Data Factory (ADF) to Azure ML, use the AzureMLExecutePipeline activity in ADF. In the activity settings, map ADF parameters to the ML pipeline parameters.

For example, you can pass container_name and connection_string from ADF to the ML pipeline like this:

"Parameters": { "container_name": "@pipeline().parameters.container_name", "connection_string": "@pipeline().parameters.connection_string" }

Access these parameters inside the Python script in the Execute Python Script component

Accessing parameters inside the Python script in the Execute Python Script component is done by retrieving them through the Run Context object, where they are available as input datasets or variables when passed to the pipeline

from azureml.core import Run # Get the run context run = Run.get_context() # Access the pipeline parameters using run.input_params container_name = run.input_params.get('container_name') connection_string = run.input_params.get('connection_string')

Best practices for securely passing connection strings and credentials

To securely pass connection strings and credentials, store them in Azure Key Vault. This keeps sensitive information safe. In Azure Data Factory (ADF), you can create a Linked Service to Azure Key Vault, which lets you retrieve these secrets during pipeline execution, avoiding hardcoding sensitive data.
In the Azure ML pipeline, you can access these secrets using environment variables or the Azure SDK. This method ensures that your credentials are securely handled and not exposed in scripts or pipeline settings.

Ref thread: https://learn.microsoft.com/en-us/answers/questions/855597/accepting-adf-parameters-in-azure-ml-pipeline-pyth

Ref Docs: https://learn.microsoft.com/en-us/azure/data-factory/transform-data-machine-learning-service?source=recommendations
https://learn.microsoft.com/en-us/azure/data-factory/how-to-expression-language-functions

I hope this information helps. Please do let us know if you have any further queries.

Kindly consider upvoting the comment if the information provided is helpful. This can assist other community members in resolving similar issues.

Thank you.
J N S S Kasyap 1,065 Reputation points Microsoft External Staff

2025-02-21T01:41:52.4966667+00:00

@Ajay Jeevan Jose
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
J N S S Kasyap 1,065 Reputation points Microsoft External Staff

2025-02-24T00:39:25.0233333+00:00

@Ajay Jeevan Jose
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

Your answer

J N S S Kasyap 1,065 Reputation points Microsoft External Staff

2025-02-21T01:41:52.4966667+00:00

@Ajay Jeevan Jose
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
J N S S Kasyap 1,065 Reputation points Microsoft External Staff

2025-02-24T00:39:25.0233333+00:00

@Ajay Jeevan Jose
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

Share via

Passing Parameters from ADF to ML Designer Pipeline

Your answer