question

kouceilaouali-4436 avatar image
0 Votes"
kouceilaouali-4436 asked SathyamoorthyVijayakumar-MSFT commented

How to move files from Batch Account Pool to Blob Storage using Data Factory ?

Hi,

In Azure Data factory pipline, we have a custom activity who use Batch Account to process a python script. This script call an external API who's returns informations stocked in a log file. The files that result from the execution are stocked on the pool job :
144213-image.png


I want to move this files to my blob storage. The Batch Account is connected to my Storage Account.
The problem is that there are only 2 files in my blob storage :
144261-image.png

How to move the content of wd folder of the job into the blob storage like stderr.txt and stdout.txt ?



azure-data-factoryazure-blob-storageazure-batch
image.png (6.5 KiB)
image.png (3.1 KiB)
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

SathyamoorthyVijayakumar-MSFT avatar image
1 Vote"
SathyamoorthyVijayakumar-MSFT answered SathyamoorthyVijayakumar-MSFT commented

Hello @kouceilaouali-4436,

Thanks for the question and using MS Q&A platform.

If my understanding is correct, you are invoking a rest api and results in downloading a file that gets stored in the WD Folder. Now you would like to move this file to your storage account.

From my research, the wd stands for the working directory of the job that has been triggered.

You could run a small snippet to copy the files in the working directory (`./` - or the environment variable - AZ_BATCH_TASK_WORKING_DIR- recursively) and upload to the blob storage using python blob sdk or rest apis using the put operation.


Hope this will help. Please let us know if any further queries.


  • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how



· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @kouceilaouali-4436,

Just checking in to see if the above answer helped. If this answers your query, do click Accept Answer and Up-Vote for the same. And, if you have any further query do let us know.

0 Votes 0 ·