How to set permissions on a cluster node's scratch disk filesystem so that users can write to it?

Gary Mansell 111

Hi,

I have created a custom cluster that builds a nodearray with an attached local disk and mounts it to the node successfully on node startup and formats it (it is not persistent).

How do I now set permissions on the /scratch filesystem - so that my user's compute jobs can write to it?

There is no doco that I can find explaining how to do this...

Thanks

Gary

Anveshreddy Nimmala 2,465 Reputation points Microsoft Vendor

2024-01-10T06:41:25.97+00:00

Hi Gary Mansell,

Welcome to Microsoft Q&A Platform, thanks for posting your query here.

To set permissions on the /scratch filesystem, you can use the chmod command..

chmod 755 /scratch

This command sets the permissions to rwxr-xr-x, which means that the owner has read, write, and execute permissions, and all other users have read and execute permissions.

You can also set the ownership of the /scratch directory using the chown command.

chown myuser /scratch

This command sets the owner of the /scratch directory to myuser.

A better approach would be to create a group for your users and give that group permission to write to the /scratch directory. To do this, you can follow these

Create a group for your users:

sudo groupadd mygroup

Add your users to the group:

sudo usermod -a -G mygroup user1

sudo usermod -a -G mygroup user2

Change the group ownership of the /scratch directory to the group you just created:

sudo chgrp mygroup /scratch

Give the group write permission on the /scratch directory:

sudo chmod g+w /scratch

Now, any user who is a member of the mygroup group will be able to write to the /scratch directory.

hope this helps you.
Gary Mansell 111 Reputation points

2024-01-10T08:57:32.4533333+00:00

Anveshreddy Nimmala thanks very much for taking the time to write such a detailed answer, please accept my sincere apologies for not making my question detailed enough to recieve the response that I was looking for.

It is a non-persistent disk that I have specified in the CycleCloud cluster template for this node - so CycleCloud creates and initialises the disk every time the cluster nodes are created and deleted. Hence, any permissions directly made to the node/filesystem will be lost when the node is deleted/re-created by CycleCloud/Slurm.

Hence, I would like to know the CycleCloud configuration (or maybe the Slurm config?) in order to set the permissions on the node's newly created disk & filesystem every time the node is created.
Gary Mansell 111 Reputation points

2024-01-10T10:20:25.1666667+00:00
I have made some progress on this after some pointers from an "AI friend" - it seems that the answser is to use a cluster init script in the CycleCloud Slurm config to do the permissions setting.

In my case, I am already using a YAML format cloud-init script, so it seems the syntax I need to add to it is something like this:

#cloud-config runcmd: - chown -R :users /scratch - chmod -R 777 /scratch

Hope that helps someone...
Gary Mansell 111 Reputation points

2024-01-10T17:03:18.99+00:00

So, I have found that the cloud-config file is run before the CycleCloud Slurm config creates the scratch disk and mount point - so this does not work!

After some further research, I believe that it is possible to create a cluster-init script, I will try this next:

https://learn.microsoft.com/en-us/azure/cyclecloud/cluster-references/cluster-init-reference?view=cyclecloud-8

Will report back on progress

2 answers

Anveshreddy Nimmala 2,465 Reputation points Microsoft Vendor

2024-01-10T12:06:27.2266667+00:00

Hi Gary Mansell,

Thankyou for responding back on this and sharing briefing more details.

The init_script parameter allows you to specify a script that will be run on the node when it is created.

You can use this script to set the permissions on the disk and filesystem as required.

you can use the init_script parameter to set permissions on the disk and filesystem.

[[node]

type = compute

count = 1

storage_mounts = /mnt/resource

storage_size = 100

storage_type = Standard_LRS

init_script = /opt/cycle/jobs/init_script.sh

]

[cluster-init]

default_user = myuser

the init_script parameter is set to /opt/cycle/jobs/init_script.sh. This script will be run on the node when it is created.

You can create the init_script.sh file in the specified location and add the commands to set the permissions on the disk and filesystem.

#!/bin/bash

# Set permissions on the disk and filesystem

chmod 755 /mnt/resource

chown myuser:mygroup /mnt/resource

the chmod command sets the permissions on the /mnt/resource directory to 755, and the chown command sets the owner and group of the directory to myuser:mygroup.

Hope this helps you. please like the answer if it helps you or else please get back with any queries.
Please sign in to rate this answer.
Gary Mansell 111 Reputation points

2024-01-10T17:07:15.36+00:00

Hi Anveshreddy Nimmala thanks for a further response - an init_script seems like a simple solution to my problem, but I don't see how to get it on the node, as the node is built from a fresh image every time it is started?

Do I perhaps have to store it on a shared NFS partition from an external server, so that it is visible to the node? I am wondering if the NFS share is mounted before the init_script is called - do you have any advice on this? Presumably NFS export permissions need to be considered too?

Please could you elaborate?

Anveshreddy Nimmala 2,465 Reputation points Microsoft Vendor

2024-01-11T03:59:26.49+00:00

Hi Gary Mansell, Thankyou for responding back on this and sharing briefing more details. You are correct that the init_script needs to be available on the node when it is created. You can configure CycleCloud to mount an NFS share on the node by adding a storage_mounts parameter to the node definition in the cluster template.

Anveshreddy Nimmala 2,465 Reputation points Microsoft Vendor

2024-01-11T04:02:29.0633333+00:00

Hi Gary Mansell, Thankyou for responding back on this . You are correct that the init_script needs to be available on the node when it is created. You can configure CycleCloud to mount an NFS share on the node by adding a storage_mounts parameter to the node definition in the cluster template. [[node] type = compute count = 1 storage_mounts = /mnt/nfs storage_size = 100 storage_type = Standard_LRS init_script = /mnt/nfs/set_permissions.sh ] You will also need to configure CycleCloud to mount the NFS share on the node. You can do this by adding a cluster-init section to the cluster template. [cluster-init mount_nfs] mounts = /mnt/nfs:/nfs nfs_server = <server-ip> nfs_options = rwx You will need to ensure that the NFS share is exported with the appropriate permissions to allow the CycleCloud nodes to mount it. You can configure the NFS export permissions on the NFS server. Hope this helps you.

Gary Mansell 111 Reputation points

2024-01-11T11:25:21.62+00:00

Hi I can't get your suggestion to work - the /scratch filesystem mounts, but the permissions are not changed. This is what I put for the node array config (I added your suggested init_script line):

[[nodearray optistruct]] Extends = nodearraybase MachineType = $optistructMachineType ImageName = $optistructImageName MaxCoreCount = $MaxoptistructExecuteCoreCount [[[volume scratch]]] Size = 1500 StorageAccountType = Premium_LRS Mount = scratch Persistent = false [[[configuration cyclecloud.mounts.scratch]]] mountpoint = /scratch fs_type = xfs Interruptible = $optistructUseLowPrio MaxPrice = $optistructSpotMaxPrice AdditionalClusterInitSpecs = $optistructClusterInitSpecs EnableNodeHealthChecks = $EnableNodeHealthChecks init_script = /shared/chmod-scratch.sh [[[configuration]]] slurm.hpc = false slurm.partition = optistruct # set pcpu = false for all hyperthreaded VMs slurm.use_pcpu = false

And this is the script on the /shared folder - you can see it is executable and accessible via root from a remote machine:

Anveshreddy Nimmala 2,465 Reputation points Microsoft Vendor

2024-01-12T04:26:23.8633333+00:00

Hi Gary Mansell, Thankyou for responding back on this. Please add below chmod command in the script to change the permissions on the /scratch directory. # Mount the NFS share mount -t nfs nfs_server:/nfs_share /mnt/nfs_share # Change the permissions on the /scratch directory chmod 777 /scratch Also, make sure that the script is executable by running below command. chmod +x /shared/chmod-scratch.sh Hope this helps you.
Sign in to comment
Gary Mansell 111 Reputation points

2024-01-12T11:46:33.7933333+00:00
Hi, I managed to get this working with a CycleCloud Cluster-Init Scipt which is run after the cluster is built. This script has to be created in project on the CycleCloud server and then uploaded to the locker (CycleCloud Storage account). Then, using the CycleCloud UI, you edit the cluster config and in the advanced setttings section select the cluster-init script for the node type in question. To create the script and upload it to the locker:

$ cyclecloud project init scratch $ cd scratch/ $ cyclecloud project add_spec execute $ vi specs/execute/cluster-init/scripts/000_install.sh #!/bin/sh chmod -R 777 /scratch [ricadmin@azu-nemo-srv ~]$ cyclecloud project upload

Then, go to the CycleCloud UI and select to edit the cluster and add the cluster-init script:
Please sign in to rate this answer.
Anveshreddy Nimmala 2,465 Reputation points Microsoft Vendor

2024-01-17T02:10:24.4766667+00:00

Hi Gary Mansell, Thankyou for posting the solution here, it helps the community if any having same issues further.
Sign in to comment