Azure Data Factory: How to avoid redundant activities ?

Question

Hi,

Only 40 activities are allowed in a pipeline. But in our case around 40% activities are duplicated.

For e.g. We have to copy delete file activity multiple times. The reason is the file should get delete if any of the activities in file fails.

Can you please help on how I handle this scenario.

Answer 1

Hello @Abhishek Vyas and welcome to Microsoft Q&A. Thank you for this question, pipeline logic like this is my specialty.

With some clever usage of 'skipped' and 'completion' dependencies , it is possible to route failures in different points of the pipeline to the same activity. I have made a mock-up of your pipeline, with such alterations. (see below picture, and pipeline JSON at end of article)

If you wish to learn the subtle art of reducing redundant pipeline activities through logic, there are several base principles you need to keep in mind.

Multiple dependencies with the same source are OR'ed together.
Multiple dependencies with different sources are AND'ed together.
Together this looks like (Xfails OR XSkipped) AND (Ysucceeds OR Yskipped)

For activities in series ( not parallel), failure is not the only case to handle. For a sequence of activities linked my success dependencies, when an early activity fails, later activities are skipped. When a late activity fails, the early activities succeeded. Thus, to handle both cases, you must handle success, failure, and skipped.

Completion is the opposite of skipped. Completion encompasses both success and failure.

Overall pipeline success/failure is not determined by the failure of a single activity. A pipeline fails when either:

1. the last activity in a pipeline fails
2. an activity (X) has a on-success dependant activity (Y), and that activity (Y) is not run.

Please let me know if you have any questions.

----------

{  
    "name": "ReduceRedundantActivities",  
    "properties": {  
        "description": "Write 'go' or 'stop' in the parameters to simulate success / fail of an activity.",  
        "activities": [  
            {  
                "name": "MockGetCredential",  
                "type": "SetVariable",  
                "dependsOn": [],  
                "userProperties": [],  
                "typeProperties": {  
                    "variableName": "ThrowException",  
                    "value": {  
                        "value": "@if(equals(pipeline().parameters.GetCredential,'go'),true,'stop')",  
                        "type": "Expression"  
                    }  
                }  
            },  
            {  
                "name": "MockDeleteIncomingOnFailure",  
                "description": "Consolidate redundant activities",  
                "type": "Wait",  
                "dependsOn": [  
                    {  
                        "activity": "MockCopy",  
                        "dependencyConditions": [  
                            "Failed",  
                            "Skipped"  
                        ]  
                    },  
                    {  
                        "activity": "MockGetCredential",  
                        "dependencyConditions": [  
                            "Completed"  
                        ]  
                    },  
                    {  
                        "activity": "MockGetColumnMapping",  
                        "dependencyConditions": [  
                            "Skipped",  
                            "Completed"  
                        ]  
                    }  
                ],  
                "userProperties": [],  
                "typeProperties": {  
                    "waitTimeInSeconds": 1  
                }  
            },  
            {  
                "name": "MockGetColumnMapping",  
                "type": "SetVariable",  
                "dependsOn": [  
                    {  
                        "activity": "MockImportHeader",  
                        "dependencyConditions": [  
                            "Succeeded"  
                        ]  
                    }  
                ],  
                "userProperties": [],  
                "typeProperties": {  
                    "variableName": "ThrowException",  
                    "value": {  
                        "value": "@if(equals(pipeline().parameters.ColumnMapping,'go'),true,'stop')",  
                        "type": "Expression"  
                    }  
                }  
            },  
            {  
                "name": "MockImportHeader",  
                "type": "Wait",  
                "dependsOn": [  
                    {  
                        "activity": "MockGetCredential",  
                        "dependencyConditions": [  
                            "Succeeded"  
                        ]  
                    }  
                ],  
                "userProperties": [],  
                "typeProperties": {  
                    "waitTimeInSeconds": 1  
                }  
            },  
            {  
                "name": "MockCopy",  
                "type": "SetVariable",  
                "dependsOn": [  
                    {  
                        "activity": "MockGetColumnMapping",  
                        "dependencyConditions": [  
                            "Succeeded"  
                        ]  
                    }  
                ],  
                "userProperties": [],  
                "typeProperties": {  
                    "variableName": "ThrowException",  
                    "value": {  
                        "value": "@if(equals(pipeline().parameters.Copy,'go'),true,'stop')",  
                        "type": "Expression"  
                    }  
                }  
            },  
            {  
                "name": "MockDelete",  
                "type": "Wait",  
                "dependsOn": [  
                    {  
                        "activity": "MockCopy",  
                        "dependencyConditions": [  
                            "Succeeded"  
                        ]  
                    }  
                ],  
                "userProperties": [],  
                "typeProperties": {  
                    "waitTimeInSeconds": 1  
                }  
            },  
            {  
                "name": "MockTransform",  
                "type": "Wait",  
                "dependsOn": [  
                    {  
                        "activity": "MockDelete",  
                        "dependencyConditions": [  
                            "Succeeded"  
                        ]  
                    }  
                ],  
                "userProperties": [],  
                "typeProperties": {  
                    "waitTimeInSeconds": 1  
                }  
            }  
        ],  
        "parameters": {  
            "GetCredential": {  
                "type": "string",  
                "defaultValue": "go"  
            },  
            "ColumnMapping": {  
                "type": "string",  
                "defaultValue": "go"  
            },  
            "Copy": {  
                "type": "string",  
                "defaultValue": "go"  
            }  
        },  
        "variables": {  
            "ThrowException": {  
                "type": "Boolean"  
            }  
        },  
        "annotations": []  
    }  
}