Azure ML Dynamic Output Path

Sten Schneider 0

Hi there,

relating to this example: "https://learn.microsoft.com/en-us/azure/machine-learning/how-to-manage-inputs-outputs-pipeline?view=azureml-api-2&tabs=python#how-to-customize-output-path"

Is it possible to dynamically set the output path but not using the name of the command step (as shown in the example with ${{name}} but the name/id of the pipeline or some timestamp?

I tried things like ${{run_id}}, ${{parent.name}} , ... but nothing seems to be allowed except for "name".

Is the a documentation about allowed keywords using ${{}} notation? Unfortunately, I did not find anything on this.

Also I don't understand where the value of "name" is coming from. If it is from the components definition or environment variables, why are no other values allowed?

In our project this is critically important because we parallelized many steps that must write to the exact same location. Thank you very much in advance for your help.

Best regards ,

Sten

VasaviLankipalle-MSFT 14,256 Reputation points

2024-04-17T22:07:05.7133333+00:00

Hello @Sten Schneider , Thanks for using Microsoft Q&A Platform.

The {name} keyword refers to the job name, which will be resolved at job execution time but for the pipeline it's the step job name , and the {output_name} keyword refers to the output name that is defined in the component YAML.

Did you get a chance to check this documentation? https://learn.microsoft.com/en-us/azure/machine-learning/reference-yaml-component-command?view=azureml-api-2

For Binding inputs and outputs between steps in a pipeline job you can refer this: https://learn.microsoft.com/en-us/azure/machine-learning/reference-yaml-core-syntax?view=azureml-api-2#binding-inputs-and-outputs-between-steps-in-a-pipeline-job
Sten Schneider 0 Reputation points

2024-04-23T14:23:35.73+00:00

@VasaviLankipalle-MSFT Thanks for your reply. Is it possible to access something like run_id or start_timestamp via $ notation in the future? So it must be something that is changing with each batch endpoint execution but the same for all pipeline steps. This feature would be highly appreciated!
Right now the workaround is to explicitly send Output() objects for every step output with specified paths when invoking the batch endpoint. But I don't think, this is an ideal solution, as the caller have to know how much each step is parallelized.