How to Pause ADF Flow activity

Question

I have created a dataflow with Azure synapse the flow has got multiple sources, is it possible for me to run pipeline to selected data sources and not all sources in the pipeline.

Please see image below.

Is it possible to Run just the Contact without running the Account when i trigger the pipeline.

Thanks

Kelvin

Answer 1

It is not possible to run only a subset of a data flow's transformations.

Given that you have two completely independent flows of data (Contact to sink1, Account to sink2), a cleaner implementation would be to separate the two into different ADF data flows. This would also permit you to run each flow in isolation as you wish.

Answer 2

@Kelvin Utuedor

Thanks for reaching out to Microsoft Q&A.

In Azure Synapse, each data source in a pipeline is typically associated with a specific activity. If you want to run the pipeline for selected data sources, you would need to control the execution of the corresponding activities.

However, Azure Synapse Pipelines does not natively support selective execution of activities within a single pipeline run. The pipeline, when triggered, will execute all activities defined within it.

Here are a couple of workarounds you might consider:

1. Separate Pipelines: Create separate pipelines for each data source. This way, you can trigger the pipeline corresponding to the specific data source you want to process.

Parameterized Pipeline: Use parameters to control the execution of activities within the pipeline. You can define a parameter at the pipeline level and use it in an ‘If Condition’ activity to decide whether to execute the associated activities.

Remember to debug and publish the pipeline after making these changes.

Please note that these are general suggestions, and the exact implementation might vary based on your specific use case and the structure of your dataflow and pipeline. It’s always a good idea to refer to the Azure Synapse documentation or consult with an Azure Synapse expert for detailed guidance.

Hope this helps. Do let us know if you any further queries.

---

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.