Running pipeline job from batch endpoint with multiple instances

Question

I currently deployed a pipelineJob to a batch endpoint. In the definition of my pipelineJob I have specified that I want the job to run in 5 instances. However when I call the job from the batch endpoint it runs only with one instance. I tried to pass the number of instances as a parameter, calling the endpoint through the CLI with the parameter --instance-count 5, and nothing seems to work. How can I call the endpoint and run that job with multiple node instances?

Answer 1

Hello Oriol Varela Fernandez,

Welcome to the Microsoft Q&A and thank you for posting your questions here.

I understand that you would like to run pipeline job from batch endpoint with multiple instances.

Regarding your question and explanation:

• Modify your pipeline definition to include ParallelRunStep or parallel_for_each - https://learn.microsoft.com/en-us/azure/machine-learning/how-to-use-parallel-job-in-pipeline
• Ensure your compute cluster supports multiple nodes and is set to autoscale.
• Remove --instance-count from CLI commands, as it does not apply to pipeline jobs.
• Monitor pipeline execution using Azure ML Studio logs to confirm parallel execution.

Example of deploying pipelines and using parallel execution:

https://github.com/azure/azureml-examples/blob/main/sdk/python/jobs/pipelines/1g_pipeline_with_parallel_nodes/pipeline_with_parallel_nodes.ipynb

I hope this is helpful! Do not hesitate to let me know if you have any other questions. or clarifications.

---

Please don't forget to close up the thread here by upvoting and accept it as an answer if it is helpful.