Copy Activity Dynamic Source Connection

Question

Hi

Currently my pipelines on dev and prod have the same source. So basically my ingestion pipeline + EDW pipeline run daily. Reads same source twice. By prod and then by Dev.

In order to avoid double read of source twice one by dev and one by prod,

Is there a way to dynamically point sources based on flag on copy activity?

Below is diagram of existing set up and the 2nd part is thinking to break apart ingestion pipeline and dynamically decide where to pick up data from.

If not, is there any other way to achieve this?

Answer 1

Hi RJ,

It looks like you're trying to streamline your Azure Data Factory pipelines by dynamically selecting the source based on a flag in your copy activity, to avoid double reading from the same source. Here's how you can potentially set this up:

Solution Overview

You can achieve dynamic source selection in Azure Data Factory by using parameters in your copy activity. Here’s a step-by-step guide on how to do that:

Create Parameters:

• When defining your pipeline, create a parameter that will hold the value of the flag (e.g., sourceFlag) to determine which source to use.

Dynamic Content in Copy Activity:

  - In your copy activity, where you specify the source dataset, you can use the `@pipeline().parameters.sourceFlag` expression to dynamically set the source.
  
     - This will allow you to switch between different sources based on the value of `sourceFlag`.
     
     **Modify Pipeline Logic**:
     
        - Use a conditional activity (e.g. If Condition activity) before your copy activity to set the `sourceFlag` based on your desired logic.
        
        **Maintaining Datasets**:
        
           - Ensure you have datasets created for all possible sources you want to include. The copy activity will reference these datasets dynamically based on the parameter value.
           

Example

Here's a brief example using pseudo-code:

{
    "copyActivity": {
        "source": {
            "type": "DatasetReference",
            "referenceName": "@{pipeline().parameters.sourceFlag}"
        },
        "sink": {
            "type": "DatasetReference",
            "referenceName": "YourSinkDataset"
        }
    }
}

Notes

• This method not only reduces redundancy but also makes your ingestion process more flexible and easier to manage.
• If you're unfamiliar with parameters or how to set dynamic content, take a look at the documentation on Copy Activity for more guidance.

Feel free to share any more details, and I’ll be glad to help you further!