Update Machine Learning models by using Data Factory

Question

Update Machine Learning models by using Data Factory

PONS LOPEZ Antonio 0

Hi

With Machine Learning Studio (Classic) you could update the model of a prediction pipeline in Data Factory using the Machine Learning Update Resource activity (as indicated at https://learn.microsoft.com/en-us/azure/data- factory/update-machine-learning-models). But for pipelines developed with Designer, I don't know how something equivalent can be done using the Machine Learning Execute Pipeline activity in Data Factory. Is there a way to do something similar? Does it have to be done another way?

dupammi 8,615 Reputation points Microsoft External Staff

2023-10-16T11:59:48.1966667+00:00

Hi @PONS LOPEZ Antonio ,

Thanks for reaching out to us. Yes, you are correct. The Machine Learning Update Resource activity (in Data Factory) is used for pipelines developed with Machine Learning Studio (Classic). While the Machine Learning Execute Pipeline activity is used for pipelines developed with Designer.

To create a Machine Learning model using ML Designer Studio and execute from Data Factory, you can follow the below documentation:

Execute Azure Machine Learning pipelines - Azure Data Factory & Azure Synapse | Microsoft Learn

I hope this helps.
dupammi 8,615 Reputation points Microsoft External Staff

2023-10-17T11:40:38.4733333+00:00

Hi @PONS LOPEZ Antonio ,

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others.

Thanks!
dupammi 8,615 Reputation points Microsoft External Staff

2023-10-18T09:52:16.63+00:00

Hi @PONS LOPEZ Antonio ,

Following up to see if the above suggestion was helpful. Was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others.

Thanks!
PONS LOPEZ Antonio 0 Reputation points

2023-10-19T09:25:59.18+00:00

Hi,

I don't have the solution yet. I am clear that with Machine Learning Studio (Classic) the Machine Learning Update Resource activity can be used and that the Machine Learning Execute Pipeline activity is used for pipelines developed with Designer in Data Factory.

But my question is more specific, how can I update the model for pipelines developed with Designer in Data Factory.

Thanks!
dupammi 8,615 Reputation points Microsoft External Staff

2023-10-19T10:29:44.75+00:00

Hi @PONS LOPEZ Antonio ,

Thanks for your response.

Updating models in pipelines developed with Designer in Azure Data Factory involves a different approach compared to using Machine Learning Studio (Classic) and the Machine Learning Update Resource activity. It might not have a direct equivalent.

I do not have much experience using ADF activities, but I think you can achieve a similar outcome by incorporating Azure Machine Learning services. Here's how you can update the model in pipelines developed with Designer:

Azure Machine Learning Service: First, ensure that your training pipeline in Designer is set up to save the trained model as an artifact or register the model in the Azure Machine Learning service.

Use Azure ML SDK: Leverage the Azure Machine Learning Python SDK within your Data Factory pipeline to interact with the Azure Machine Learning workspace. You can use the SDK to access your trained model or any relevant artifacts.

Custom Python Activity: Within your Data Factory pipeline, add a custom Python activity that uses the Azure ML SDK to retrieve the latest model or relevant data from the Azure Machine Learning workspace.

Pass Model to Pipeline: After retrieving the model or data, you can pass it as a parameter or key-value pair to your pipeline within Data Factory. You may need to customize your pipeline to accept this parameter and use it accordingly.

By using the Azure Machine Learning Python SDK, you can access the models and artifacts stored in your Azure Machine Learning workspace and incorporate them into your Data Factory pipeline.

This approach requires some custom scripting, it allows you to update the model for pipelines developed with Designer in Data Factory.

Try this and hopefully it should help.
dupammi 8,615 Reputation points Microsoft External Staff

2023-10-22T01:03:44.1533333+00:00

Hi @PONS LOPEZ Antonio ,

Following up to see if the below suggestion given by @Janarthanan S was helpful. Was also checking, if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others.

Thanks!
dupammi 8,615 Reputation points Microsoft External Staff

2023-10-24T07:21:16.07+00:00

Hi @PONS LOPEZ Antonio ,

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others.

Thanks!
PONS LOPEZ Antonio 0 Reputation points

2023-10-24T07:59:33.03+00:00

Hi,

I don't know how this information can help me since the activity I use, Machine Learning Execute Pipeline, does not appear in the list of activities in which the dynamic content flow is supported.

Thanks!
dupammi 8,615 Reputation points Microsoft External Staff

2023-10-24T13:26:43.87+00:00

Hi @PONS LOPEZ Antonio ,

Have you tried the approach mentioned in my latest reply above i.e., on Oct 19, 2023, 3:59 PM

If not, kindly follow the steps which I mentioned.

Thanks!
PONS LOPEZ Antonio 0 Reputation points

2023-10-24T14:50:00.61+00:00

Hi @dupammi ,

What was indicated in your last reply on Oct 19, 2023, 3:59 PM are very general indications since many of these steps are not clear to me as to how to approach them. I have another query open related to this, https://learn.microsoft.com/en-us/answers/questions/1391949/pass-a-dataset-to-a-machine-learning-execute-pipel, where I detail the problem and what I have done.

Thanks!
dupammi 8,615 Reputation points Microsoft External Staff

2023-10-25T13:35:14.6+00:00

Hi @PONS LOPEZ Antonio ,

Thanks for your response and details on the other thread.

Regarding your question on other thread, "To register the trained best model of the Tune Model Hyperparameters module as an output of the ML training pipeline in the Designer", please try the Register Model module.

For more information on how to use the Register Model module, see the docs

Regarding your question "How to pass the dataset generated in the training pipeline to the prediction pipeline", you were rightly using the "Output" activity in the training pipeline and store the model dataset in a storage account or Azure Blob Storage. Then, in the prediction pipeline, you can try the "Lookup" activity to retrieve the model dataset from the storage account or Azure Blob Storage.

After retrieving the model dataset, you can use the "Web" activity in the prediction pipeline to call the Azure Machine Learning Designer's REST API endpoint to execute the prediction pipeline. Please pass the retrieved model dataset as a parameter to the REST API endpoint.

For more information on how to use the "Web" activity to call REST API endpoints, please refer to the following Azure documentation:

Web activity in Azure Data Factory

For more information on how to use the "Lookup" activity to retrieve data from a storage account or Azure Blob Storage, please refer to the following Azure documentation:

Lookup activity in Azure Data Factory

Regarding REST API error:

Based on the error message you provided, it seems that the API version '1.0' does not support the HTTP method 'GET'. Therefore, you should use the HTTP method 'POST' instead of 'GET' to call the REST service.
Web Activity - Azure Data Factory & Azure Synapse | Microsoft Learn

I hope this helps you resolve the issue. Thank you!
PONS LOPEZ Antonio 0 Reputation points

2023-10-25T15:51:13.4966667+00:00

Hi @dupammi

Thanks for your response.

Regarding the question "To register the trained best model of the Tune Model Hyperparameters module as an output of the ML training pipeline in the Designer", I think I have not explained well. I already have the model registered as indicated in https://learn.microsoft.com/en-us/azure/machine-learning/how-to-deploy-model-designer?view=azureml-api-1#register-the-model using . What I don't know is how to update that model dynamically, not manually, updating the version when the training pipeline is executed from the Data Factory.

If it achieved that, it would already have it because in the predictive pipeline it would always take the latest version of the model.

Thanks!

dupammi 8,615 Microsoft External Staff

Hi @PONS LOPEZ Antonio ,

Thank you for providing more information.

To implement the dynamic model versioning and updating of a machine learning training pipeline in Azure Data Factory, you'll need to incorporate Azure Data Factory (ADF) activities and Azure Functions or other methods to achieve the dynamic model versioning. Below are the steps you can try:

Determine the New Model Version

from azureml.core import Model 

# List all registered models with the same name 

registered_models = Model.list(workspace=ws, name='your_model_name') 

# Find the latest version number 

latest_version = max([int(model.version) for model in registered_models]) 

# Increment the version for the new model 

new_version = str(latest_version + 1)

Update the Training Step with the New Version

train_step = PythonScriptStep(
    name='train_step',
    script_name='train.py',
    arguments=[
        '--input_data', input_data.as_named_input('input_data'),
        '--output_data', output_data,
        '--model_version', new_version  # Pass the new version as an argument
    ],
    inputs=[input_data],
    outputs=[output_data],
    compute_target='your_compute_target_name',
    source_directory='your_source_directory',
    allow_reuse=False
)

For more information, please refer to this technical document for implementing the PythonScriptStep

parse the 'model_version' argument and use it when registering the new model version.

import argparse
from azureml.core import Workspace, Model

# Parse the arguments
parser = argparse.ArgumentParser()
parser.add_argument('--model_version', type=str, help='New model version')
args = parser.parse_args()

# Connect to your Azure ML workspace
ws = Workspace.from_config()

# Register the new model with the specified version
model = Model.register(workspace=ws,
                       model_path='path_to_new_model',
                       model_name='your_model_name',
                       tags={'area': 'your_model_area'},
                       properties={'version': args.model_version})

Submit the Pipeline

With these modifications in place, submit your pipeline as before, and it will dynamically update the model version each time it's run.

You can set up a trigger in Azure Data Factory to execute your training pipeline when the Azure Function provides the new model version. The trigger can be event driven.

Please refer to the below document which explains about StepSequence:
azureml.pipeline.core.builder.StepSequence class - Azure Machine Learning Python | Microsoft Learn

I hope this helps. Thank you!

PONS LOPEZ Antonio 0 Reputation points

2023-10-28T17:44:55.57+00:00
Hi @dupammi

Thanks for your response.

I think I'm almost there. I just think I have an authentication problem left.

I really think that I don't have to register a new version of a model, but of a dataset. I have registered the "Trained best model" output dataset of the "Tune Model Hyperparameters" component in the training pipeline, as you can see in the image.

This dataset is the one I use as a model in the predictive pipeline (see image) which always uses the latest version of the dataset:

Now what I have done is a python script that takes that dataset and registers it by incrementing the version number by 1. (see script).

Running the script from my local computer works.

The idea would be to be able to execute it from a Data Factory pipeline at the end of the training pipeline execution activity (see image).

For that I am trying to use a "Custom" activity (I don't know if there is another easier way to execute the script within the Data Factory pipeline), but I have authentication problems.

I have done a test creating a pipeline in Data Factoy only with a "Custom" activity for which I have created a "Batch account" with a "Pool"

For the "Custom" activity I have created a linked service with the Blob Storage where I have uploaded the script. The connection works correctly as you can see in the image.

If I leave the script as it worked for me locally, and run the pipeline in Data Factory only with the "Custom" activity, it does not work for me because it waits for interactive authentication through the browser and that is not possible if the script is executed from a "Pool". " from a "Batch account".

I have tried modifying the script to use MsiAuthentication as you can see in the image

but it gives me the following error:

"mistake": {

"message": "Failed to connect to MSI. Please make sure MSI is configured correctly.\nGet Token request returned http error: 400, reason: Bad Request" }

I have created a "Managed Identities"

and I have tried using the "Identity type" "System assigned" and "User-assigned" for the "Batch account" to assign permissions in the "Azure Machine Learning workspace"

But it still gives me the same error:
dupammi 8,615 Reputation points Microsoft External Staff

2023-10-29T02:57:34.69+00:00

Hi @PONS LOPEZ Antonio ,

By incorporating the version incrementing step within the Azure Machine Learning Studio Designer pipeline, you can create a more streamlined end-to-end workflow without involving additional components in Azure Data Factory.

It's a good practice to handle the version incrementing within the Azure Machine Learning Studio Designer pipeline whenever possible. This approach simplifies the workflow and improves overall manageability. If you're using Azure Data Factory to orchestrate tasks, my suggestion is to let the ML pipeline handle the data versioning as well. Thanks!

1 answer

Your answer

dupammi 8,615 Reputation points Microsoft External Staff

2023-10-16T11:59:48.1966667+00:00

Hi @PONS LOPEZ Antonio ,

Thanks for reaching out to us. Yes, you are correct. The Machine Learning Update Resource activity (in Data Factory) is used for pipelines developed with Machine Learning Studio (Classic). While the Machine Learning Execute Pipeline activity is used for pipelines developed with Designer.

To create a Machine Learning model using ML Designer Studio and execute from Data Factory, you can follow the below documentation:

Execute Azure Machine Learning pipelines - Azure Data Factory & Azure Synapse | Microsoft Learn

I hope this helps.
dupammi 8,615 Reputation points Microsoft External Staff

2023-10-17T11:40:38.4733333+00:00

Hi @PONS LOPEZ Antonio ,

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others.

Thanks!
dupammi 8,615 Reputation points Microsoft External Staff

2023-10-18T09:52:16.63+00:00

Hi @PONS LOPEZ Antonio ,

Following up to see if the above suggestion was helpful. Was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others.

Thanks!
PONS LOPEZ Antonio 0 Reputation points

2023-10-19T09:25:59.18+00:00

Hi,

I don't have the solution yet. I am clear that with Machine Learning Studio (Classic) the Machine Learning Update Resource activity can be used and that the Machine Learning Execute Pipeline activity is used for pipelines developed with Designer in Data Factory.

But my question is more specific, how can I update the model for pipelines developed with Designer in Data Factory.

Thanks!
dupammi 8,615 Reputation points Microsoft External Staff

2023-10-19T10:29:44.75+00:00

Hi @PONS LOPEZ Antonio ,

Thanks for your response.

Updating models in pipelines developed with Designer in Azure Data Factory involves a different approach compared to using Machine Learning Studio (Classic) and the Machine Learning Update Resource activity. It might not have a direct equivalent.

I do not have much experience using ADF activities, but I think you can achieve a similar outcome by incorporating Azure Machine Learning services. Here's how you can update the model in pipelines developed with Designer:

Azure Machine Learning Service: First, ensure that your training pipeline in Designer is set up to save the trained model as an artifact or register the model in the Azure Machine Learning service.

Use Azure ML SDK: Leverage the Azure Machine Learning Python SDK within your Data Factory pipeline to interact with the Azure Machine Learning workspace. You can use the SDK to access your trained model or any relevant artifacts.

Custom Python Activity: Within your Data Factory pipeline, add a custom Python activity that uses the Azure ML SDK to retrieve the latest model or relevant data from the Azure Machine Learning workspace.

Pass Model to Pipeline: After retrieving the model or data, you can pass it as a parameter or key-value pair to your pipeline within Data Factory. You may need to customize your pipeline to accept this parameter and use it accordingly.

By using the Azure Machine Learning Python SDK, you can access the models and artifacts stored in your Azure Machine Learning workspace and incorporate them into your Data Factory pipeline.

This approach requires some custom scripting, it allows you to update the model for pipelines developed with Designer in Data Factory.

Try this and hopefully it should help.
dupammi 8,615 Reputation points Microsoft External Staff

2023-10-22T01:03:44.1533333+00:00

Hi @PONS LOPEZ Antonio ,

Following up to see if the below suggestion given by @Janarthanan S was helpful. Was also checking, if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others.

Thanks!
dupammi 8,615 Reputation points Microsoft External Staff

2023-10-24T07:21:16.07+00:00

Hi @PONS LOPEZ Antonio ,

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others.

Thanks!
PONS LOPEZ Antonio 0 Reputation points

2023-10-24T07:59:33.03+00:00

Hi,

I don't know how this information can help me since the activity I use, Machine Learning Execute Pipeline, does not appear in the list of activities in which the dynamic content flow is supported.

Thanks!
dupammi 8,615 Reputation points Microsoft External Staff

2023-10-24T13:26:43.87+00:00

Hi @PONS LOPEZ Antonio ,

Have you tried the approach mentioned in my latest reply above i.e., on Oct 19, 2023, 3:59 PM

If not, kindly follow the steps which I mentioned.

Thanks!
PONS LOPEZ Antonio 0 Reputation points

2023-10-24T14:50:00.61+00:00

Hi @dupammi ,

What was indicated in your last reply on Oct 19, 2023, 3:59 PM are very general indications since many of these steps are not clear to me as to how to approach them. I have another query open related to this, https://learn.microsoft.com/en-us/answers/questions/1391949/pass-a-dataset-to-a-machine-learning-execute-pipel, where I detail the problem and what I have done.

Thanks!
dupammi 8,615 Reputation points Microsoft External Staff

2023-10-25T13:35:14.6+00:00

Hi @PONS LOPEZ Antonio ,

Thanks for your response and details on the other thread.

Regarding your question on other thread, "To register the trained best model of the Tune Model Hyperparameters module as an output of the ML training pipeline in the Designer", please try the Register Model module.

For more information on how to use the Register Model module, see the docs

Regarding your question "How to pass the dataset generated in the training pipeline to the prediction pipeline", you were rightly using the "Output" activity in the training pipeline and store the model dataset in a storage account or Azure Blob Storage. Then, in the prediction pipeline, you can try the "Lookup" activity to retrieve the model dataset from the storage account or Azure Blob Storage.

After retrieving the model dataset, you can use the "Web" activity in the prediction pipeline to call the Azure Machine Learning Designer's REST API endpoint to execute the prediction pipeline. Please pass the retrieved model dataset as a parameter to the REST API endpoint.

For more information on how to use the "Web" activity to call REST API endpoints, please refer to the following Azure documentation:

Web activity in Azure Data Factory

For more information on how to use the "Lookup" activity to retrieve data from a storage account or Azure Blob Storage, please refer to the following Azure documentation:

Lookup activity in Azure Data Factory

Regarding REST API error:

Based on the error message you provided, it seems that the API version '1.0' does not support the HTTP method 'GET'. Therefore, you should use the HTTP method 'POST' instead of 'GET' to call the REST service.
Web Activity - Azure Data Factory & Azure Synapse | Microsoft Learn

I hope this helps you resolve the issue. Thank you!
PONS LOPEZ Antonio 0 Reputation points

2023-10-25T15:51:13.4966667+00:00

Hi @dupammi

Thanks for your response.

Regarding the question "To register the trained best model of the Tune Model Hyperparameters module as an output of the ML training pipeline in the Designer", I think I have not explained well. I already have the model registered as indicated in https://learn.microsoft.com/en-us/azure/machine-learning/how-to-deploy-model-designer?view=azureml-api-1#register-the-model using . What I don't know is how to update that model dynamically, not manually, updating the version when the training pipeline is executed from the Data Factory.

If it achieved that, it would already have it because in the predictive pipeline it would always take the latest version of the model.

Thanks!
dupammi 8,615 Reputation points Microsoft External Staff

2023-10-29T02:57:34.69+00:00

Hi @PONS LOPEZ Antonio ,

By incorporating the version incrementing step within the Azure Machine Learning Studio Designer pipeline, you can create a more streamlined end-to-end workflow without involving additional components in Azure Data Factory.

It's a good practice to handle the version incrementing within the Azure Machine Learning Studio Designer pipeline whenever possible. This approach simplifies the workflow and improves overall manageability. If you're using Azure Data Factory to orchestrate tasks, my suggestion is to let the ML pipeline handle the data versioning as well. Thanks!

Answer 1

Janarthanan S 700

Hi @PONS LOPEZ Antonio

But my question is more specific, how can I update the model for pipelines developed with Designer in Data Factory.

You can find the solution in the detailed documentation provided by Microsoft

Please find below link

https://learn.microsoft.com/en-us/azure/data-factory/how-to-manage-studio-preview-exp#pipeline-experimental-view

I hope the solution will be useful and if it's helpful please accept the answer.

Regards,

Janarthanan S

0 comments