Dynamic Schema / ADF

Raffy Comendador 6 Reputation points
2021-11-29T00:46:08.717+00:00

Hi, am trying to ingest text files going to parquet table dynamically from different source files (column names and count differ per each file). Thus, am not defining the table schema anywhere (source, sink, dataset) to make this work. but due to this, the parquet table have Prop_0, prop_1 etc. as column names as a result (as expected :-) ).

How can I make the ADF use the column names included in the source file to have the schema of both dataset, source, sink get updated accordingly so that the target parquet table will have proper column names?

Thank you in advance.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,343 questions
No comments
1 vote

3 answers

Sort by: Most helpful
  1. KranthiPakala-MSFT 33,146 Reputation points Microsoft Employee
    2021-11-30T23:01:44.9+00:00

    Hi @Raffy Comendador ,

    Thanks for your query and using Microsoft Q&A forum.

    Have you tried enabling the first row as header property in your source dataset? If not could you please try it and see if that helps.

    153854-image.png

    In case if that is not helpful, the alternate would be to go with parameterized mapping for each source file. To explore more about dynamic column mapping please refer to this video - Dynamic Column mapping in Copy Activity in Azure Data Factory

    Here is the public doc related to parameterized mapping: Parameterize mapping

    Hope this info helps. Do let us know if you have further query.

    ----------

    • Please don't forget to click on 130616-image.png and upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification
    • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators

  2. Raffy 1 Reputation point
    2021-12-05T23:24:59.797+00:00

    Hi, I have yet to apply the resolution provided. will try to revert within the week. thank you and God bless


  3. Raffy 1 Reputation point
    2021-12-11T09:15:32.023+00:00

    Hi, cant find the accept button :) but this is helpful and is accepted. thanks much!

    No comments