Registering Data Asset from Job with v2 SDK/CLI

Question

Registering Data Asset from Job with v2 SDK/CLI

Tomas Van Pottelbergh 21

Following this example I tried to register the output of a job as a Data Asset, by using the azureml:<my_data>:<version> syntax for the path. The following minimal example shows what I'm trying to achieve:

from azure.ai.ml import command  
from azure.ai.ml.entities import Data  
from azure.ai.ml import Input, Output  
from azure.ai.ml.constants import AssetTypes  
  
my_job_outputs = {  
    "prep_data": Output(type=AssetTypes.URI_FOLDER, path="azureml:test_output:1")  
}  
  
job = command(  
    command="echo 'test' > ${{outputs.prep_data}}",  
    outputs=my_job_outputs,  
    environment="test-env@latest",  
    compute="cpu-cluster",  
)  
  
# submit the command  
returned_job = ml_client.create_or_update(job)

Unfortunately, this fails. The job submission works, but the Job is immediately going to the Failed status with the following error message:

Invalid output uri azureml:test_output:1/ found for output prep_data of run , the list of supported uri formats are ["wasb://", "wasbs://", "adl://", "abfs", "abfss://", "azureml://"]

What is the correct syntax for registering a Data Asset from a Job output?

romungi-MSFT 48,801 Reputation points Microsoft Employee

2022-09-19T06:02:45.463+00:00
@Tomas Van Pottelbergh The data asset can be registered after your output is registered on any of the valid paths i.e the following:

# local: './<path>/<file>' # blob: 'https://<account_name>.blob.core.windows.net/<container_name>/<path>/<file>' # ADLS gen2: 'abfss://<file_system>@<account_name>.dfs.core.windows.net/<path>/<file>' # Datastore: 'azureml://datastores/<data_store_name>/paths/<path>/<file>'

Since you are using it as output instead of input and since there is no previous data asset set as reference for input, the error message is seen.
Once the output data is registered in any of the above paths you can add a reference to the data, this is the data asset that can be referenced later. Please refer this document for creating data asset.

By creating a data asset, you create a reference to the data source location, along with a copy of its metadata.

my_path = '<path>' my_data = Data( path=my_path, type=AssetTypes.URI_FILE, description="<description>", name="<name>", version="<version>" ) ml_client.data.create_or_update(my_data)

Once the reference is created, you can try to read the asset as input in the corresponding command or job. I hope this clarifies!!
Tomas Van Pottelbergh 21 Reputation points

2022-09-19T13:47:49.54+00:00

Thanks for your answer @romungi-MSFT .

I tried using the azureml://datastores/<data_store_name>/paths/<path>/<file> syntax for the path and that indeed works. However, this registers the data asset under a random name such as azureml:azureml_mighty_iron_xs3ntws5p6_output_data_test_output:1.

How can I register such an output under a user-friendly data asset name such as azureml:test_output:1? In the v1 SDK this is possible using the register_on_complete method (https://learn.microsoft.com/en-us/azure/machine-learning/v1/how-to-create-machine-learning-pipelines#configure-data-with-dataset-and-outputfiledatasetconfig-objects), but I don't see how to do this using the v2 SDK/CLI.
romungi-MSFT 48,801 Reputation points Microsoft Employee

2022-09-21T13:09:31.327+00:00

@Tomas Van Pottelbergh I can confirm that the friendly name and version is yet to be added for output class which provides parity with register_on_complete from v1. There isn't an ETA yet since v2 is still experimental.
Tomas Van Pottelbergh 21 Reputation points

2022-09-23T13:24:32.343+00:00

Thanks for the clarification @romungi-MSFT . I really look forward to this feature being added. Is there a GitHub issue on the SDK repo to track this so I can be informed when it is available?
romungi-MSFT 48,801 Reputation points Microsoft Employee

2022-09-26T09:59:29.263+00:00

@Tomas Van Pottelbergh There isn't a public repo for tracking this feature. This is an internal spec that should be available when v2 goes GA.
You could add a feature request on azure SDK for python though.
mangesh vk 0 Reputation points

2023-02-16T05:12:48.97+00:00
hi @romungi-MSFT

I have the exact same requirement, please let us know if this feature is available now in SDK V2. Also if there is any way to get the "Run Name" while in the pipeline , using which the path to the output can be formulated. The job runs on a compute cluster, I couldnt find any way to get access to the System Identity assigned to the computer cluster, the "client" and "object id" mentioned in the error is the Principal ID (system assigned managed identity) of the cluster

when I run it now, I get following error

Message: The client 'xxxxxxxxxxxxxxxx' with object id ' xxxxxxxxxxxxxxxx' does not have authorization to perform action 'Microsoft.MachineLearningServices/workspaces/datastores/read' over scope '/subscriptions/xxxxxxxxxxxxxxx/resourceGroups/xxxxxxxxx/providers/Microsoft.MachineLearningServices/workspaces/xxxx/datastores/workspaceblobstore' or the scope is invalid. If access was recently granted, please refresh your credentials.
Umang Salgia 0 Reputation points

2023-08-14T06:04:17.12+00:00

I am facing this error too. Is there any plan to add this to the SDK? As it is almost a year, does not look like this is on the roadmap

Your answer

romungi-MSFT 48,801 Reputation points Microsoft Employee

2022-09-19T06:02:45.463+00:00

@Tomas Van Pottelbergh The data asset can be registered after your output is registered on any of the valid paths i.e the following:

# local: './<path>/<file>' # blob: 'https://<account_name>.blob.core.windows.net/<container_name>/<path>/<file>' # ADLS gen2: 'abfss://<file_system>@<account_name>.dfs.core.windows.net/<path>/<file>' # Datastore: 'azureml://datastores/<data_store_name>/paths/<path>/<file>'

Since you are using it as output instead of input and since there is no previous data asset set as reference for input, the error message is seen.
Once the output data is registered in any of the above paths you can add a reference to the data, this is the data asset that can be referenced later. Please refer this document for creating data asset.

By creating a data asset, you create a reference to the data source location, along with a copy of its metadata.

my_path = '<path>' my_data = Data( path=my_path, type=AssetTypes.URI_FILE, description="<description>", name="<name>", version="<version>" ) ml_client.data.create_or_update(my_data)

Once the reference is created, you can try to read the asset as input in the corresponding command or job. I hope this clarifies!!
Tomas Van Pottelbergh 21 Reputation points

2022-09-19T13:47:49.54+00:00

Thanks for your answer @romungi-MSFT .

I tried using the azureml://datastores/<data_store_name>/paths/<path>/<file> syntax for the path and that indeed works. However, this registers the data asset under a random name such as azureml:azureml_mighty_iron_xs3ntws5p6_output_data_test_output:1.

How can I register such an output under a user-friendly data asset name such as azureml:test_output:1? In the v1 SDK this is possible using the register_on_complete method (https://learn.microsoft.com/en-us/azure/machine-learning/v1/how-to-create-machine-learning-pipelines#configure-data-with-dataset-and-outputfiledatasetconfig-objects), but I don't see how to do this using the v2 SDK/CLI.
romungi-MSFT 48,801 Reputation points Microsoft Employee

2022-09-21T13:09:31.327+00:00

@Tomas Van Pottelbergh I can confirm that the friendly name and version is yet to be added for output class which provides parity with register_on_complete from v1. There isn't an ETA yet since v2 is still experimental.
Tomas Van Pottelbergh 21 Reputation points

2022-09-23T13:24:32.343+00:00

Thanks for the clarification @romungi-MSFT . I really look forward to this feature being added. Is there a GitHub issue on the SDK repo to track this so I can be informed when it is available?
romungi-MSFT 48,801 Reputation points Microsoft Employee

2022-09-26T09:59:29.263+00:00

@Tomas Van Pottelbergh There isn't a public repo for tracking this feature. This is an internal spec that should be available when v2 goes GA.
You could add a feature request on azure SDK for python though.
mangesh vk 0 Reputation points

2023-02-16T05:12:48.97+00:00

hi @romungi-MSFT

I have the exact same requirement, please let us know if this feature is available now in SDK V2. Also if there is any way to get the "Run Name" while in the pipeline , using which the path to the output can be formulated. The job runs on a compute cluster, I couldnt find any way to get access to the System Identity assigned to the computer cluster, the "client" and "object id" mentioned in the error is the Principal ID (system assigned managed identity) of the cluster

when I run it now, I get following error

Message: The client 'xxxxxxxxxxxxxxxx' with object id ' xxxxxxxxxxxxxxxx' does not have authorization to perform action 'Microsoft.MachineLearningServices/workspaces/datastores/read' over scope '/subscriptions/xxxxxxxxxxxxxxx/resourceGroups/xxxxxxxxx/providers/Microsoft.MachineLearningServices/workspaces/xxxx/datastores/workspaceblobstore' or the scope is invalid. If access was recently granted, please refresh your credentials.
Umang Salgia 0 Reputation points

2023-08-14T06:04:17.12+00:00

I am facing this error too. Is there any plan to add this to the SDK? As it is almost a year, does not look like this is on the roadmap

Share via

Registering Data Asset from Job with v2 SDK/CLI

Your answer