Best Practice for Handling Multiple Excel Files with Different Schemas and Schedules in Azure Data Factory

Question

I’m working on an Azure Data Factory (ADF) solution where I receive four different Excel files daily as input data for order processing. Each of these files has a unique and fixed schema that will not change over time.

In terms of scheduling:

Two files need to be processed twice a day

Two files need to be processed four times a day

Given this setup, I’m trying to decide between two design options:

Create separate ADF pipelines for each file, each with its own data flow and scheduled trigger

Design a single, parameterized pipeline that can handle all four files using dynamic logic and different trigger schedules

Which approach is considered best practice in this scenario — especially in terms of maintainability, performance, monitoring, and scalability?

Any guidance or architectural suggestions would be greatly appreciated!

Answer 1

Ideally a single generic file for similar actions are preferred rather than individual pipelines per files assuming all you need to do is follow the same flow of actions for every file.

Then please follow the below approach :

1. Create a single pipeline scheduled for 4 times a day

Flow within the pipeline :

1. check the execution time, if it matches the schedule of 4 times , only append the last 2 excel file names as an array else include all 4 files
2. pass that variable as an input for for each activity for iterations
3. within foreach, orchestrate your flow
4. In case of copy activity, your source dataset can be parameterized within the itertaion item() file name and in your mapping section you can have dynamic mapping.
5. now for sink and dynamic mapping you need to have a configuration mapping somewhere that maps your excel file name to the corresponding sink name and mapping