Data Flow activity with different transformations dynamically

Question

Hi

I have a dataflow where the datasets are coming dynamically through pipeline in foreach loop. Now in this dataflow I want to use filter,derived column transformations etc. But it shows error while validating . How do I mitigate this? there must be an option.

Following is the dataset where folder name is dynamically which is used in dataflow

Dataflow uses source options for taking the file names

As the file names are dynamic, now there is no projection of header fields, So all transformations like filter, derived will give error .

How do I mitigate this and validate it successfully.

Answer 1

If your data flow does not have a defined projection, then you will use "late binding" in your transformations where you can refer to specific column names using the byName() metafunction. Alternatively, you can use column matching in Derived Columns to define a canonical model that is disconnected from the hardened source projection.

https://mssqldude.wordpress.com/2020/01/06/late-binding-vs-early-binding-in-adf-data-flows/

https://www.youtube.com/watch?v=K5tgzLjEE9Q