The costing for the Batch services and I have trouble running my Python job in Azure Batch because the required directories are not present on the Batch node.

Question

1. I would like to know how Batch service is charged.
2. When I run my Python jobs through ADF's Batch service, I get som troubles.

For example: My test.py script and the utils folder are located in the run_python folder. The test.py script calls the test1.py script in the utils folder. But when running the script using ADF, there is an issue where it cannot find the utils folder.

How I can keep the structure in Batch service's node.

Answer 1

Azure Batch service charges are based on the following components:

1. Compute costs: Charges for VMs you use in your pool and the cost depends on the VM size and the number of VMs.
2. Storage costs: Charges for storing files in Azure Storage, including input data, application packages, and output data.
3. Data transfer costs: Charges for data transfer between the Azure Batch service and Azure Storage, as well as between different regions.

Regarding your issue with running the Python job, it looks like the problem is with the directory structure itself and the way paths are referenced in your scripts.

You can create a zip file of your run_python folder, including both the test.py script and the utils folder. This will ensure that the directory structure is preserved when the files are uploaded to the Batch node.

Then, upload the zip file to an Azure Storage account that is accessible by your Azure Batch account.

In your Azure Batch task, include a startup script that will unzip the package in the correct location on the Batch node.

Answer 2

Hello @谢志青 Zhiqing Xie

Azure Batch is charged based on the number of nodes, the size of the nodes, and the duration of the nodes.

You can find more information about pricing on the Azure Batch pricing page.

Regarding your issue, it seems like the utils folder is not being copied to the node where the job is running.

You can try adding the utils folder to your input files in the Batch job. This will ensure that the folder is copied to the node where the job is running.

To do this, you can add the following code to your Python script:

import os
import shutil
# Copy the utils folder to the working directory
src_dir = os.path.join(os.getcwd(), "utils")
dst_dir = os.path.join(os.getcwd(), "run_python", "utils")
shutil.copytree(src_dir, dst_dir)

This code will copy the utils folder to the working directory of the Batch task. You can then use the utils folder in your Python script as usual. That should help

---

I hope that this response has addressed your query and helped you overcome your challenges. If so, please mark this response as Answered. This will not only acknowledge our efforts, but also assist other community members who may be looking for similar solutions.