How to create CI/CD pipelines for scripts that run on Azure Batch

Question

Hello,

I was going thru Azure Batch and thought this would be a good fit for one of my solutions. I have thousands of python scripts with some dependencies on pegasus which takes in some file as input and generates a report(pdf or txt) as an output. I would like to manage the execution of

1. Running all these scripts along with dependencies on Azure (Probably Batch or some cheap offering)
2. They are not compute intensive but the scripts may keep growing and I would made the output reports available for other systems to easily consume with an endpoint.
3. I would like to have the development done on VS Code with Azure Devops as source control along with a deployment pipeline to be pushed to Azure Batch directly(If at all this exists as a solution today).

Could you please point me in a right direction on the above 3 key points?

Thanks

Answer 1

Hello @Pavan Puligandla ,

Regarding

1.Running all these scripts along with dependencies on Azure (Probably Batch or some cheap offering) 2.They are not compute intensive but the scripts may keep growing and I would made the output reports available for other systems to easily consume with an endpoint.

Azure Batch is generally used where there is large/variable concurrency and explicit control over pools/VM size/OS is required or complex task dependencies.

How to use for your use case:

1. Containerize your runtime (Python + Pegasus + fonts/libcairo/ghostscript etc. needed for PDFs) -> publish to Azure Container Registry (ACR).
2. Input -> Task mapping: one task per input file (or batch of files) so Batch can parallelize.
3. State & artifacts: mount or stage Azure Blob Storage for inputs; write outputs to reports/{jobId}/{taskId}/....
4. Cost controls: use Spot VMs in pools with autoscale. (Low‑priority VMs were retired; Spot is the supported path.)
5. Scale model: Chose either a long‑lived pool that autos-scales on queue depth or an autopool per job (clean isolation, no pool drift)

Alternative to Azure Batch, you can try Azure Container Apps.
Azure Container Apps are generally used for jobs that aren’t compute‑heavy or long‑running, and you want minimal ops and you need pay‑per-execution model.

How to use it for your use case:

• Containerize once (same as Azure Batch) -> push to ACR.
• Use event‑driven jobs: trigger on Service Bus or Storage Queue messages that point to the input file URI; job runs a single script instance and exits.
• Outputs go to Blob Storage;
• Expose HTTP API for retrieval (Azure Functions or a small Container App)

---

Regarding,

3. I would like to have the development done on VS Code with Azure Devops as source control along with a deployment pipeline to be pushed to Azure Batch directly(If at all this exists as a solution today).

Unfortunately, there is not a pre-made Azure Devops solution yet but here are few links that help you prepare for it. You can use quick starts available over Microsoft.learn articles for Azure Batch for example this link : https://learn.microsoft.com/en-us/azure/batch/quick-run-python

This has the source code available (https://github.com/Azure-Samples/batch-python-quickstart) and this executes the jobs and shares the output in a text based file over to your storage account.

To automate this in Azure DevOps pipeline, I was not able to find the readymade pipeline however we have the steps outlined in detail over here: https://learn.microsoft.com/en-us/azure/batch/batch-ci-cd, kindly refer the steps and test out Azure Batch if it meets your requirement.