Pass a dataset to a Machine Learning Execute Pipeline in Data Factory activity

Question

Pass a dataset to a Machine Learning Execute Pipeline in Data Factory activity

PONS LOPEZ Antonio 0

Hello,

I have a training pipeline developed with the Azure MAchine Learning Designer and another for predictions. I want to update the model used in the prediction pipeline with the one generated with the training pipeline. In the predictions pipeline I have several parameters.

User's image

One of these parameters is the model dataset.

In Data factory I have created a pipeline with two Machine Learning Execute Pipeline activities, one that executes the training pipeline and the other the predictions pipeline. But in the prediction pipeline I don't know how to pass the dataset with the model generated in the training pipeline, since it does not appear as a parameter even though it is defined in the prediction pipeline.

romungi-MSFT 48,906 Reputation points Microsoft Employee Moderator

2023-10-16T10:26:24.6366667+00:00
@PONS LOPEZ Antonio I think you are referring to the ML pipeline when you mean a pipeline parameter is already defined in the prediction pipeline.

I do not have much experience using ADF activities but going by the reference of ML execute pipeline activity you can add your Azure ML prediction pipeline parameter as key value pair, I think the following should help to update your prediction pipeline with required values.

Ensure the required data from training pipeline is saved as output or registered as a model in the ML workspace.

In the Machine Learning Execute Pipeline activity for the predictions pipeline, add a "Web" activity before the Machine Learning Execute Pipeline activity. In the "Web" activity, you can use the Azure Machine Learning REST API to get the pipeline job data of the training pipeline.

Using the output of this web activity you can pick up the required data and pass the same as key/value pair as input parameters of prediction pipeline.
PONS LOPEZ Antonio 0 Reputation points

2023-10-25T14:59:20.6833333+00:00
Thanks romungi-MSFT for response

But I have several doubts:

How do I register as output of the ML training pipeline in the Designer the Trained best model of the Tune Model Hyperparameters module?

Once this is achieved, from what I understand, I would have to add a Web activity to the Data Factory pipeline in the following way:

Where in the URL parameter I have put the address of the REST service of the training pipeline) as you can see in the image)

But I must point out that when I tried to call the URL of the REST service I got the following error:

"Code": "UnsupportedApiVersion", "Message": "The HTTP resource that matches the request URI 'https://westeurope.api.azureml.ms/pipelines/v1.0/subscriptions/0bfb3599-e1ef-4cef-9273-6a52efc32643/resourceGroups/aplchurn4/providers/Microsoft.MachineLearningServices/workspaces/amlaplchurn4/PipelineRuns/PipelineEndpointSubmit/Id/15368957-a5e1-40e9-854c-6f819737f989' with API version '1.0' does not support HTTP method 'GET'.",

Assuming that the previous problem was solved, in the predictive pipeline I have defined the model as input, as seen in the image,

And finally, in the prediction pipeline activity in Data Factory, I assign the output of the Web activity as an input parameter.

What I had done until now is save the output of the Tune Model Hyperparameters module in the training pipeline in a certain path as you can see in the image:

which coincides with that of an already registered model which is then the one I use in the prediction pipeline.

I don't know if it would also be worth it

1 answer

Your answer

romungi-MSFT 48,906 Reputation points Microsoft Employee Moderator

2023-10-16T10:26:24.6366667+00:00

@PONS LOPEZ Antonio I think you are referring to the ML pipeline when you mean a pipeline parameter is already defined in the prediction pipeline.

I do not have much experience using ADF activities but going by the reference of ML execute pipeline activity you can add your Azure ML prediction pipeline parameter as key value pair, I think the following should help to update your prediction pipeline with required values.

Ensure the required data from training pipeline is saved as output or registered as a model in the ML workspace.

In the Machine Learning Execute Pipeline activity for the predictions pipeline, add a "Web" activity before the Machine Learning Execute Pipeline activity. In the "Web" activity, you can use the Azure Machine Learning REST API to get the pipeline job data of the training pipeline.

Using the output of this web activity you can pick up the required data and pass the same as key/value pair as input parameters of prediction pipeline.
PONS LOPEZ Antonio 0 Reputation points

2023-10-25T14:59:20.6833333+00:00

Thanks romungi-MSFT for response

But I have several doubts:

How do I register as output of the ML training pipeline in the Designer the Trained best model of the Tune Model Hyperparameters module?

Once this is achieved, from what I understand, I would have to add a Web activity to the Data Factory pipeline in the following way:

Where in the URL parameter I have put the address of the REST service of the training pipeline) as you can see in the image)

But I must point out that when I tried to call the URL of the REST service I got the following error:

"Code": "UnsupportedApiVersion", "Message": "The HTTP resource that matches the request URI 'https://westeurope.api.azureml.ms/pipelines/v1.0/subscriptions/0bfb3599-e1ef-4cef-9273-6a52efc32643/resourceGroups/aplchurn4/providers/Microsoft.MachineLearningServices/workspaces/amlaplchurn4/PipelineRuns/PipelineEndpointSubmit/Id/15368957-a5e1-40e9-854c-6f819737f989' with API version '1.0' does not support HTTP method 'GET'.",

Assuming that the previous problem was solved, in the predictive pipeline I have defined the model as input, as seen in the image,

And finally, in the prediction pipeline activity in Data Factory, I assign the output of the Web activity as an input parameter.

What I had done until now is save the output of the Tune Model Hyperparameters module in the training pipeline in a certain path as you can see in the image:

which coincides with that of an already registered model which is then the one I use in the prediction pipeline.

I don't know if it would also be worth it

Answer 1

Thanks romungi-MSFT for response

But I have several doubts:

How do I register as output of the ML training pipeline in the Designer the Trained best model of the Tune Model Hyperparameters module? User's image

Once this is achieved, from what I understand, I would have to add a Web activity to the Data Factory pipeline in the following way:

User's image

Where in the URL parameter I have put the address of the REST service of the training pipeline) as you can see in the image)

User's image

But I must point out that when I tried to call the URL of the REST service I got the following error:

"Code": "UnsupportedApiVersion",
		"Message": "The HTTP resource that matches the request URI 'https://westeurope.api.azureml.ms/pipelines/v1.0/subscriptions/0bfb3599-e1ef-4cef-9273-6a52efc32643/resourceGroups/aplchurn4/providers/Microsoft.MachineLearningServices/workspaces/amlaplchurn4/PipelineRuns/PipelineEndpointSubmit/Id/15368957-a5e1-40e9-854c-6f819737f989' with API version '1.0' does not support HTTP method 'GET'.",

Assuming that the previous problem was solved, in the predictive pipeline I have defined the model as input, as seen in the image, User's image

And finally, in the prediction pipeline activity in Data Factory, I assign the output of the Web activity as an input parameter.

User's image

What I had done until now is save the output of the Tune Model Hyperparameters module in the training pipeline in a certain path as you can see in the image:

User's image

which coincides with that of an already registered model which is then the one I use in the prediction pipeline.

User's image

I don't know if it would also be worth it

Share via

Pass a dataset to a Machine Learning Execute Pipeline in Data Factory activity

1 answer

Your answer