ADF and azure-batch incorrectly uploads batch task dependency files when stored inside a directory

Question

ADF and azure-batch incorrectly uploads batch task dependency files when stored inside a directory

Dave 1

Does anyone know to correctly preserve the directory structure for supporting files stored on an ADLSGen2 storage account that gets used in a custom ADF batch Python activity?

I have an ADF pipeline that executes a Python script (called run_script.py) on an Ubuntu Linux batch pool using the custom batch activity.

The script depends on multiple supporting .py modules that are stored in a lib directory.
The lib directory is located at the same level as the python script like so:
run_script.py
lib/lib_file1.py
lib/lib_file2.py

The same structure is uploaded to a location on my data lake storage, which gets linked to the batch activity in the ADF pipeline.

The problem is the pipeline fails to run - Python throws a ModuleNotFound error (no module named lib).
When I inspect the working directory for the job, the directory structure looks something like this:
run_script.py
lib_file1.py
lib_file2.py
lib (empty file)

The files are getting copied to the same level as the main script and the directory structure of the lib folder is not being preserved. Because of this, the script cannot find the corresponding lib module in the correct place, which throws the error.

I'd be very grateful if someone could help me to resolve this issue or provide more insight. I'm unsure if there is a way to instruct azure-batch to preserve the file structure?

I've simplified the structure to make it easier to explain, but in reality, there are more files and they are also nested in sub-modules.
It is not easy to modify the main script to use a flat file structure.

1 answer

Your answer

Answer 1

KranthiPakala-MSFT 46,737 Microsoft Employee Moderator

Hi @Dave ,

Welcome to Microsoft Q&A forum and thanks for posting your query.

Thanks for the details provided but I'm still trying to understand from where (Source - Is it a Azure blob storage/On Prem location or something else) the .py and dependency files ( lib directory ) are copied to ADLS Gen2 location and what is process that is being used to copy the files and lib folder.

I'm assuming that you are using ADF copy activity to copy those files and directory from x source to ADLS gen2 and then pointing ADF Custom Batch activity to the new ADLS Gen2 location. Please correct if I misunderstood your requirement.

If you agree with my understanding, then I would suggest you to please check this doc which has detailed info on how to preserve the directory structure using Copy activity in ADF: ADF Copy activity - recursive and copyBehavior examples

When you select the recursive property to true under copy activity source settings and copyBehavior as preserveHierarchy under sink settings then the target folder is created with the same structure as the source.

Copy source settings:

Copy sink settings:

Hope this info helps. Do let us know if you have further query.

----------

Please don't forget to click on and upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
Want a reminder to come back and check responses? Here is how to subscribe to a notification
If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators

KranthiPakala-MSFT 46,737 Reputation points Microsoft Employee Moderator

2021-12-24T10:03:09.69+00:00

Hi @Dave ,

Just checking in to see if the above suggestion was helpful. If it answers your query, please do click “Accept Answer” and/or Up-Vote, as it might be beneficial to other community members reading this thread. And, if you have any further query do let us know.
KranthiPakala-MSFT 46,737 Reputation points Microsoft Employee Moderator

2021-12-27T23:14:54.667+00:00

Hi @Dave ,

We still have not heard back from you. Just wanted to check if the above suggestion was helpful? If it answers your query, please do click “Accept Answer” and/or Up-Vote, as it might be beneficial to other community members reading this thread. And, if you have any further query do let us know.
Dave 1 Reputation point

2021-12-28T08:06:59.117+00:00

Hi, Sorry for the late response.

I'm manually copying the files into the ADLS Gen2 location. and not using a copy activity.

The problem is not the copy - as I can do that myself. The problem is when a batch task is created, the files used to run the .py file are copied to the working directory on the node but when this happens the file structure is not preserved in the working directory on the node.
KranthiPakala-MSFT 46,737 Reputation points Microsoft Employee Moderator

2022-01-05T00:13:24.023+00:00

@Dave - When you are copying the files manually to the ADLS Gen2 location, could you please make sure to create the directory hierarchy as required to run the .py scripts. As per my understanding, I don't see any issue with ADF process here. My understanding is that the issue is with the directory structure which is making the batch task to fail because of the incorrect folder structure. Please correct if I miss anything or misunderstood the ask so that we can better assist you.
KranthiPakala-MSFT 46,737 Reputation points Microsoft Employee Moderator

2022-01-07T07:04:38.357+00:00

Hello @Dave ,

Following up to see if your issue was resolved? If you still have questions do let us know, we will be happy to assist.

Thank you
Chengyue Pan 1 Reputation point

2023-05-13T19:23:09.48+00:00

I have the same problem right now. We try to use ADF custom activity to call Azure Batch Service to run python code. And the python script is on ADLS Gen2 container "scripts" location with folder structure

run_script.py
lib/lib_file1.py
lib/lib_file2.py

Then the batch service will import these script to the batch service node wd folder as following structure

run_script.py

lib

lib_file1.py

lib_file2.py

The question is how to make the Azure batch service batch node task wd folder structure same as the existing folder structure in container "scripts"

Thanks

Jerry

Share via

ADF and azure-batch incorrectly uploads batch task dependency files when stored inside a directory

1 answer

Your answer