The azure cli command "az ml attach folder" is directly adding .azureml directory to .amlignore , so where to put config.json when using Azure devops pipeline to submit script to aml workspace?

Question

Hello MS team,

I am using Azure devOps pipeline to submit a control script to the Azure-ML workspace. This control script in turn kicks off the Azure-ML pipeline containing pythonscriptsteps and hyperdrive step.

My directory structure:

.
├───.vscode
├───Automation
├───Build
└───Source
├───.azureml
├───.vscode
├───amlcode
│ ├───projectcode
│ └───pycache
├───config
├───Data
├───setup
├───tests
│ ├───.pytest_cache
│ │ └───v
│ │ └───cache
│ └───pycache
└───pycache

So here one of the azure cli task in Azure DevOps pipeline uses:

az ml folder attach -w $(azureml.workspaceName) -g $(azureml.resourceGroup)

This command attaches my whole directory to the AML workspace and automatically creates ".amlignore" and ".azureml" is automatically added to that.

So it is throwing an authentication error as the config.json() is not found because it is generally put in the path /.azureml.

Where to put the config.json() then? What is the best practice?

Answer 1

@Shivapriya Katta The command az ml folder attach will create the directories and add the config file to .azureml to ensure the workspace resources are easily accessible. You can lookup the note section of the command for reference.

This command creates a .azureml subdirectory that contains example runconfig and conda environment files. It also contains a config.json file that is used to communicate with your Azure Machine Learning workspace.

The authentication error in your case could be because az login command might have been missed which allows the cli to authenticate interactively or service principal or MI and then run rest of the commands. You can try to run this and check if the attach works successfully.

Also, with the devops pipeline I am not sure if az devops login is required to be run but if the above command fails even after az login authentication you can try az devops login.

If an answer is helpful, please click on or upvote which might help other community members reading this thread.

Answer 2

Thanks a lot!... I understood what "az ml folder attach" is doing (have been reading Azure docs), what I wanted is when I do the following:

1. Submit a script using Service principal from Azure git repo (where my code lies), what is the point of using "az ml folder attach"?

because anyway when I am submitting script from my local PC or from Azure DevOps (without "az ml folder attach"), the script runs.

Answer 3

Thank you...Got the issue resolved! The command "az ml folder attach" won't override the ".amlignore" if the user creates it beforehand. So I just created ".amlignore" how I want it to be and removed ".azureml" from it.