Dynamic Source prosjection when using Data flow on Azure Data Factory

Nuno Trigo 51 Reputation points

I have an data flow that compares two similar tables( the original and the new one), the new rows from the new file are added (appended) on the original one.

It works fine. My issue is that this Dataflow will be reused by 10 different files using parameters that are in a json file. 255125-image.png

It works almost fine.

My big issue is that metadata differs from each table and data factory doesnt adapt accordingly.

As an example. file1 : file projection is perfect.

When data factory iterates to the second table, data factory still have the same projection even though metadata differs. the result is : empty columns ....

I need to data factory to dynamically find out metadata from each table on the source of my dataflow and adapt accordingly.

How can I do that ? there is no option under "optimize" to set some dynamic parameters.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
7,201 questions
{count} vote

Accepted answer
  1. AnnuKumari-MSFT 23,506 Reputation points Microsoft Employee

    Hi @Nuno Trigo ,

    Thankyou for using Microsoft Q&A platform and thanks for posting your question here.

    As I understand your query, you are trying to create a dynamic dataflow that can be utilized to perform data transformation and data movement for multiple tables. Kindly let me know if that is not the ask.

    You will need to use a parameterized dataset that uses a dataset parameters for the name of the table. Then, pass a string parameter from the Foreach activity that contains the table name into the dataset parameter for that data flow activity.

    In dataflow, you need to use schema less datasets for your Source and Sink transformations and either use "automap" on the Sink or write a rule using rule-based mapping to map columns based on patterns.

    This video helps explain how to do it: ADF Transform Multiple Tables in Single Pipeline

    Hope this will help. Please let us know if any further queries.


    • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you.
      Original posters help the community find answers faster by identifying the correct answer. Here is how
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification
    • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators
    0 comments No comments

0 additional answers

Sort by: Most helpful