Dynamic Schema / ADF

Question

Hi, am trying to ingest text files going to parquet table dynamically from different source files (column names and count differ per each file). Thus, am not defining the table schema anywhere (source, sink, dataset) to make this work. but due to this, the parquet table have Prop_0, prop_1 etc. as column names as a result (as expected :-) ).

How can I make the ADF use the column names included in the source file to have the schema of both dataset, source, sink get updated accordingly so that the target parquet table will have proper column names?

Thank you in advance.

Answer 1

Hi @Raffy Comendador ,

Thanks for your query and using Microsoft Q&A forum.

Have you tried enabling the first row as header property in your source dataset? If not could you please try it and see if that helps.

In case if that is not helpful, the alternate would be to go with parameterized mapping for each source file. To explore more about dynamic column mapping please refer to this video - Dynamic Column mapping in Copy Activity in Azure Data Factory

Here is the public doc related to parameterized mapping: Parameterize mapping

Hope this info helps. Do let us know if you have further query.

----------

• Please don't forget to click on and upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators

Answer 2

Hi, I have yet to apply the resolution provided. will try to revert within the week. thank you and God bless

Answer 3

Hi, cant find the accept button :) but this is helpful and is accepted. thanks much!