How to move files from Batch Account Pool to Blob Storage using Data Factory ?

Question

Hi,

In Azure Data factory pipline, we have a custom activity who use Batch Account to process a python script. This script call an external API who's returns informations stocked in a log file. The files that result from the execution are stocked on the pool job :

I want to move this files to my blob storage. The Batch Account is connected to my Storage Account.
The problem is that there are only 2 files in my blob storage :

How to move the content of wd folder of the job into the blob storage like stderr.txt and stdout.txt ?

Answer 1

Hello @kouceila ouali ,

Thanks for the question and using MS Q&A platform.

If my understanding is correct, you are invoking a rest api and results in downloading a file that gets stored in the WD Folder. Now you would like to move this file to your storage account.

From my research, the wd stands for the working directory of the job that has been triggered.

You could run a small snippet to copy the files in the working directory (./ - or the environment variable - AZ_BATCH_TASK_WORKING_DIR- recursively) and upload to the blob storage using python blob sdk or rest apis using the put operation.

Hope this will help. Please let us know if any further queries.

------------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how