Azure SDK previous step output to multiple steps as input

Mia17 41 Reputation points
2022-04-06T03:05:11.427+00:00

I have a output from previous step and want to use it as input to multiple steps.
But, when I run the experiment, the pipeline looks like this

Here is my code

source_directory="./test"  
designer1_config = ScriptRunConfig(source_directory=source_directory,  
                                 command=["python", "designer1.py",   
                                          "--output_test1", output_test1], #   
                                 compute_target=aml_compute,  
                                 environment=env_py)  
  
designer1_step = CommandStep(name="designer_step",   
                           inputs=[input_dept_fun_d],  
                           outputs=[output_test1], #  
                           runconfig=designer1_config,  
                           allow_reuse=True)  


source_directory="./test"  
designer2_config = ScriptRunConfig(source_directory=source_directory,  
                                 command=["python", "designer2.py"], #   
                                 compute_target=aml_compute,  
                                 environment=env_py)  
  
designer2_step = CommandStep(name="designer2_step",   
                           inputs=[input_dept_fun_d, output_test1],  
                           outputs=[], #  
                           runconfig=designer2_config,  
                           allow_reuse=True)  

Use 2 steps, it shows picture 1

step_sequence = [designer1_step, designer2_step]  

Use 1 step, it shows picture 2

step_sequence = [designer2_step]  
  

Submit the experiement

pipeline = Pipeline(workspace=ws, steps=step_sequence)  
pipeline_run = Experiment(workspace=ws, name='pipeline').submit(config=pipeline, regenerate_outputs=True)  

190374-image.png

190373-image.png

Here's what I want

190392-image.png

Azure Machine Learning
Azure Machine Learning
An Azure machine learning service for building and deploying models.
2,720 questions
{count} votes

Accepted answer
  1. romungi-MSFT 43,691 Reputation points Microsoft Employee
    2022-04-07T06:19:24.293+00:00

    @MiaZhangWHQWistron-2092 Based on the setup for designer2_step the inputs are the original input of step1 and the output of step1. The second screen shot seems appropriate and the designer has just replicated the original input dataset for step2. The connection that you are referring to is irrelevant because the same dataset is used, and designer only displays it for simplicity.

    If an answer is helpful, please click on 130616-image.png or upvote 130671-image.png which might help other community members reading this thread.

    1 person found this answer helpful.

0 additional answers

Sort by: Most helpful