Is there any way to do Custom dynamic mapping of different number of columns in dataflow or any other options to achieve this?

Question

My source (CSV file in ADLS) has header record(3 columns) , detail records(5 columns) and trailer record(2 columns) . The header record has less number of columns than the detail records. When I try to convert this csv file to parquet, i m getting the column count error using copy activity in ADF. So I tried using dataflow to do the mapping but still its considering only three columns and ignoring the other two columns in the detail records.
So please let me know how to achieve this using dataflow or any other azure services.

Sample Data

1|~filename|~30122020
2|~Mark|~cse|~378|~2020
2|~John|~|~430|~2019
99|~3

Answer 1

Looks like this type of dynamic mapping won't work in ADF. It's better to ask source team or regenerate csv in fixed proper format.

This has been answered here - https://stackoverflow.com/questions/65503601/is-there-any-way-to-do-custom-dynamic-mapping-of-different-number-of-columns-in

---

Please don't forget to Accept Answer and Up-vote if the response helped -- Vaibhav