How to mount connect file share folder in Azure Batch container

Chris M 30

I have created a batch pool/jobs/tasks, uploaded container image and connected to an Azure File Share.

The container image, is a python program which should read data from ./data/inputs

...and output everything in ./data/outputs.

In my connected storage file share, I have folders ./data/inputs and ./data/outputs in the root of the storage file share. I have added the storage to the pool, and everything is reporting as successfully connected (I have created all roles, images etc).

When I run locally on my desktop docker using the options docker run --mount type=bind,source="$(pwd)"/data,target=/data -it

Nikhil Duserla 1,780 Reputation points Microsoft Vendor

2024-07-25T07:09:53.9966667+00:00

Hi Chris M,

Just checking in to see if the below answer provided by Vinodh247 helped.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.
Chris M 30 Reputation points

2024-07-25T07:28:54.93+00:00

Hi Nikhil. We have a bit of a time difference so have not had a chance to try out the suggestions yet.

The suggestion is definitely helpful and comprehensive and gave me some things to try out.

Will report back on progress as soon as possible.

Regards

Accepted answer

Vinodh247 16,116 Reputation points

2024-07-25T00:18:30.91+00:00
Hi Chris Martis,

Thanks for reaching out to Microsoft Q&A.

Please do not respond with CLI. I first need to understand how to do this from the Azure portal website

To mount an Azure File Share to an Azure Batch container using the Azure portal, follow these steps:

Create or Select an Azure Batch Pool:

Navigate to your Azure Batch account in the Azure portal.

Select "Pools" from the left navigation pane.

Create a new pool or select an existing one.

Configure File Share Mounting:

Go to the "Configuration" section of your pool.

Under "Mount configuration," select "Add" to mount an Azure File Share.

Fill in the required details:

Storage account name: The name of your storage account.

File share name: The name of your Azure File Share.

Mount path: The path where the file share will be mounted on the compute nodes (e.g., '/mnt/data').

Account key: The key for accessing the storage account.

Verify Mount Configuration:

Ensure the file share is correctly added to the mount configuration.

Save the changes and ensure the pool is updated.

Submit a Job to the Pool:

Navigate to the "Jobs" section and create a new job.

Configure the job to use the pool with the mounted file share.

Configure Task to Use Mounted File Share:

When creating tasks for the job, ensure the task command references the mounted path (ex: '/mnt/data/inputs' and '/mnt/data/outputs').

Run and Monitor the Job:

Submit the job and monitor its progress.

Verify that the tasks can access the mounted file share paths.

By following the above steps, the Azure File Share will be mounted to the specified path on each compute node in your Azure Batch pool, allowing your container to access the required directories.

Let me know if this worked for you!

Please 'Upvote'(Thumbs-up) and 'Accept' as an answer if the reply was helpful. This will benefit other community members who face the same issue.
Please sign in to rate this answer.

1 person found this answer helpful.
Chris M 30 Reputation points

2024-07-25T13:52:16.89+00:00

Thanks. I did not resolve the issue with this.

in the mount options, I put

Azure File Share

the url from the storage for that File Share

for "relative mount path "/mnt/data"

I leave mounting options empty

account name and key for the storage account

When the pool is created with 2 notes, I get an error on both nodes.

Message:

An error was encountered while configuring specified mount(s)

Values:

Message - [Errno 2] No such file or directory: 'mnt/batch/tasks/fsmounts/fshare-mnt/data.log'

MountiConfiguationPath - Status.TvmIntervalError.

I get the same error when I choose /data/, or /.

I understand that mounting means set folder A of the EFS to be folder A of the container in the node , but this is clearly not how it works in Azure. The documentation is very thin on this, and if I cannot resolve this, then I will not be able to proceed with Azure.

What is the relative path mount? Relative to What? How do you express it? Do I need to create some folders in the EFS first?

Any help would be appreciated.

Chris M 30 Reputation points

2024-07-25T15:40:02.83+00:00

My 2nd comment was duplicating the above, so I removed.

Vinodh247 16,116 Reputation points

2024-07-25T15:50:27.0466667+00:00

I suspect your configuration might be at fault. Can you check the below and confirm?

Before mounting, make sure that the directory structure within the container exists. You may need to create the directory where you intend to mount the Azure File Share.

Ensure the relative path is correctly specified. For example, if you want to mount the Azure File Share to /mnt/data within the container, you should specify mnt/data as the relative mount path (without a leading slash).

Double-check that the source URL, account name, and account key are correctly specified.

Ensure that the container has the necessary permissions to access and mount the Azure File Share.

Chris M 30 Reputation points

2024-07-25T15:56:58.2766667+00:00

Thanks Vinodh247.Ah....but of course. I think I may have missed the directory in the container. I will check this and then go down the list if that didn't fix it. Will report back shortly.

Chris M 30 Reputation points

2024-07-25T17:50:10.84+00:00

okay, so I added the directory in the container. I tried various options, but I settled with the following in the dockerfile:

RUN bash -c 'mkdir -p mymountpoint/data/{Inputs,Outputs}'

I then used mymountpoint as the relative folder, so I no longer get the "directory doesn't exist" - (good!) , but now I get a new error .

Code: MountConfigurationError

Message: An error was encountered while configuring specified mount(s)

Values: Message - Incorrect invocation or permissions MountConfigurationPath - mymountpoint

The EFS is mounted as Z on my local system, so I tried Z too, and curiously I get the same error as above

I also tried mnt/mymountpoint and mymountpoint/ and I got the previous "directory doesn't exist" error.

The storage access keys etc are correct (i picked them up from the Access keys in the storage container, and I triple-checked them!).

So, unless something else is happening, (i.e. the mounting happening somewhere where the container doesn't have write access), it could be container access permissions.

I have created a "system managed identity" for the batch account.

I then granted the batch account the role of "Storage Blob Data Contributor" in storage account and tested, which I now removed. This did not make a difference.

I am not sure how to proceed. Documentation is really thin on this. I'd be grateful for any help/pointers.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

1 additional answer

Chris M 30 Reputation points

2024-07-25T19:56:28.1033333+00:00
right, so I managed to fix this. The issue was

as you pointed out, the container had to have the folders created inside it to map to it

the mounting options were left blank. I had to set up permissions dir_mode, file_mode etc.

Thanks for the help.
Please sign in to rate this answer.

1 person found this answer helpful.
Vinodh247 16,116 Reputation points

2024-07-26T05:26:08.2766667+00:00

glad that this helped you.

Sreenidhi Palimar 20 Reputation points

2024-08-09T16:45:58.3366667+00:00

Hi Chris M,
Could you please let me know in the field "mount option", what are the inputs that we can provide

Chris M 30 Reputation points

2024-08-09T17:20:39.15+00:00

@Sreenidhi Palimar , really depends on what you want to do. You can use something like -o dir_mode=0777,file_mode=0777 etc, which tell you the permissions to read/write etc.

Sreenidhi Palimar 20 Reputation points

2024-08-12T06:53:08.5733333+00:00

Hi Chris M,
I have updated all the required field and ran the job, job ran successfully but i could see no logs getting pushed for Azure file share ,
let me provide you the screenshots for the reference,

1.Screenshot of Mount configuration

2.Screenshot of the job ran and got successful (here you could see where the files are getting created that is in which path)

3.Screenshot where my mount is created on the node

Screenshot of workitems folder in the node(this is different from the above attached path) here i could see all the details of job ran like job number, job task and the .out and .err file

Let me know how to correct this if you aware of any methods to troubleshoot this issue
and let me know if i need to provide any further details

Chris M 30 Reputation points

2024-08-12T08:05:03.76+00:00

See here https://learn.microsoft.com/en-us/azure/batch/files-and-directories.

All mounted folders are created inside fsmounts. So, when you defined workitems in mount options, this was (correctly) created inside fsmounts....and has nothing to do with the workitems system folder.

If you want to access the content of other system folders in your node, you will need to use environment variable references inside your app.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

How to mount connect file share folder in Azure Batch container

1 additional answer

Your answer