Run parallel databricks in a foreach

Question

Hello The last action in my pipeline is a foreach that invokes another pipeline (execute load files). The pipeline in question is a copy data with databricks.

Invoked Pipeline:

I'd like to parallelize this action. The result of my look up is a list of databricks path. Even though I have unchecked the sequential from foreach activity but still its running in sequential. I execute this pipeline on trigger mode too. This is my pipeline expression in foreach.

@activity('Get ControlFlow').output.value

Any help would be appriciated.

Answer 1

Hi Thales Berti, Thank you for posting query in Microsoft Q&A Platform.

Inside your execute pipeline activity, could you please check if wait for execution complete option is checked?

If yes, remove that and try. Hope this helps. Please let me know how it goes.