ParallelRunStep doesn´t support multiple input datasets

Question

Currently we are using the parallelrunstep in AML, Is there any workaround/other approach to support distribute datasets in the parallel run step.

Answer 1

@AI Thanks for the question. An Azure ML dataset is just metadata pointing to a path or collection of paths in an Azure storage account. You should first "merge" those datasets into a collection of adjacent folders (e.g. root/dataset1/, root/dataset2/, ...) and then run PRS against root/**.