AzureML pipeline not working

Question

I have designed a pipeline that makes predictions and saves the results to a blob container.
The pipeline works fine after submitting the experiment. However, after I publish it and call it via its REST endpoint, it does not work (I don't get my results). The portal shows that the job has been completed, without any error.
Can someone enlighten me on how to use a publish pipeline?

Answer 1

Thanks for the details. All published pipelines have a REST endpoint. With the pipeline endpoint, you can trigger a run of the pipeline from external systems, such as non-Python clients. For information about how to authenticate when calling REST endpoints, see https://aka.ms/pl-restep-auth.

Using the endpoint enables "managed repeatability" in batch scoring and retraining scenarios, for example. For more information, see https://aka.ms/pl-first-pipeline.

https://learn.microsoft.com/en-us/python/api/azureml-pipeline-core/azureml.pipeline.core.graph.publishedpipeline?view=azure-ml-py

Here is the sample and document for publish.

Answer 2

There are 2 versions in Azure ML studio and here are the links by which you can develop the ML pipelines in both of them. I have used version 1 long before and I am continuing the same.

• version 1 : https://learn.microsoft.com/en-us/azure/machine-learning/v1/how-to-create-machine-learning-pipelines?view=azureml-api-1
• version 2 : https://learn.microsoft.com/en-us/azure/machine-learning/tutorial-pipeline-python-sdk#create-a-job-environment-for-pipeline-steps

Please read this if you are facing issue in version 1, I haven't explored version 2 yet

After searching for a while I was able to find a solution here: https://github.com/Azure/MachineLearningNotebooks/issues/298

• Short Answer: I have used Python Script Step functionality to build my ML pipeline. This is a version 1 python sdk functionality which have allow_reuse parameter set to True, change it to False
• Long Answer: The default behavior of a Step execution in Pipelines is that when the script specified in the Step using script_name, inputs, and the parameters of a step remains the same, the output of a previous step run will be reused instead of running the step again. When a step is reused, the job is not submitted to the compute, instead, the results from the previous run are immediately available to the next step runs.
  Azure Machine Learning Pipelines provide ways to control and alter this behavior. ### allow_reuse Flag
  You can specify allow_reuse=False as a parameter of the Step. When allow_reuse is set to False, the step run won’t be reused, and a new run will always be generated for the step during pipeline execution. Default behavior of Pipelines is to set allow_reuse=True for steps.

step = PythonScriptStep(name="Hello World",
                         script_name="hello_world.py", 
                         compute_target=aml_compute, 
                         source_directory= source_directory,
                         allow_reuse=False
                        )