Process Multi-Line Fixed Width Input Files in ADF

Question

We need to process traditional mainframe batch files using ADF. These text files are fixed record length AND multi-line per logical record, meaning all the columns to be extracted span across multiple rows of the input file. Any way to do this in ADF natively without preprocessing the file?

Example:

HGAXX2021-02-23 // Header Row
GAAELEMENT1 ELEMENT2 ELEMENT3 ELEMENT4 // Record 1
GABELEMENT5 ELEMENT6 ELEMENT7 ELEMENT8 // Record 1
GAAELEMENT1 ELEMENT2 ELEMENT3 ELEMENT4 // Record 2
GABELEMENT5 ELEMENT6 ELEMENT7 ELEMENT8 // Record 2
GAAELEMENT1 ELEMENT2 ELEMENT3 ELEMENT4 // Record 3
GABELEMENT5 ELEMENT6 ELEMENT7 ELEMENT8 // Record 3
FGAXX000003 // Footer Row with Count of Records

Thank you!
Jon

Answer 1

Add a Data Flow to your Pipeline. In the Source transformation, select "Multiline rows".