BUG : Flatten transformation not reading the array column with multiple words in column name

Raghav Sharma 1 Reputation point
2021-07-08T10:54:21.303+00:00

I was trying to split the pipe delimited string column which had a column name {Bill Code} in Derived column transformation followed by Flatten transformation, Derived transformation which will split the string and convert it into an array and Flatten will convert them into individual rows.
Ex: B12|B13|B14 to ["B12", "B13", "B14"]

Then I passed this column with {Bill Code} name to flatten transformation and went to data preview it threw an error "Unroll value {Bill Code} must be an array", although the bill code column was an array but it was not honoring it, then I had to change the column name from bill code to code and then the flatten transformation honored the column as an array.

Conclusion : Flatten transformation is not honoring the array column with multiple words separated by space and enclosed by curly brackets .

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,045 questions
{count} votes

2 answers

Sort by: Most helpful
  1. MartinJaffer-MSFT 25,981 Reputation points
    2021-07-09T19:56:10.837+00:00

    I have reproduced this bug using data:

    Row,Bill Code
    1,A1|A2|A3
    2,B12|B13|B14
    

    and script:

    source(output(
            Row as short,
            {Bill Code} as string
        ),
        allowSchemaDrift: true,
        validateSchema: false,
        ignoreNoFilesFound: false,
        format: 'delimited',
        container: 'martin',
        fileName: 'sample.csv',
        columnDelimiter: ',',
        escapeChar: '\\',
        quoteChar: '\"',
        columnNamesAsHeader: true) ~> origin
    origgin derive(Code = split({Bill Code},'|')) ~> Split
    Split foldDown(unroll(Code),
        mapColumn(
            Row,
            Bill = Code
        ),
        skipDuplicateMapInputs: false,
        skipDuplicateMapOutputs: false) ~> Flatten1
    

    I tried several permutations of names and flatten settings, in case something else was going on. However it looks like you are right, it is the name. This seems odd to me. I will let the developers know about this bug. Thank you for bringing it to our attention.


  2. MartinJaffer-MSFT 25,981 Reputation points
    2021-09-15T17:57:49.13+00:00

    This bug should now be fixed.

    0 comments No comments