I'm having problems with the flow of inputs/outputs dependencies in a pipeline.
Im trying to run a ipynb file in a step (NotebookRunnerStep) and later use that notebook
(the ipynb file, including all the cell outputs generated when running) as an input of the next step, but i keep getting errors.
I would greatly appreciate a 10-minute video call session to resolve doubts.
output_from_notebook = PipelineData(name="notebook_output",
datastore= def_blob_store,
output_path_on_compute= 'notebook_output')
step1 = NotebookRunnerStep(name="processing_step",
notebook_run_config=cfg,
params={"param_name": param_variable},
allow_reuse=False,
compute_target=aml_compute,
output_notebook_pipeline_data_name='notebook_output',
outputs = [output_from_notebook])
html_output = PipelineData(name="html_output", datastore=def_blob_store, output_path_on_compute='html_output', is_directory=False)
step2 = PythonScriptStep(name="convert_step",
arguments=[
'--notebook', output_from_notebook.as_mount(),
'--output_html', source_directory
],
inputs=[output_from_notebook],
outputs = [html_output],
script_name='convert_notebook_to_html.py',
compute_target=aml_compute,
source_directory=source_directory,
allow_reuse=False,
runconfig = run_config_cd)