data flow flatten json file

Question

data flow flatten json file

arkiboys 9,706

Hello,
see screenshots below
1-
I receive this error when going to data preview of the flatten activity in the dataflow.
2-
The flatten activity does not show the columns array in the dropdown of the unroll by
any suggestions?

sample json file used

{
"Title":"Demo",
"Owner":"test2",
"EntityName":"tableName",
"Columns":
[
{
"SourceColumnName":"TableID",
"ColumnType":"INT",
"SourceFormat":""
},
{
"ColumnName":"Name",
"ColumnType":"VARCHAR(50)",
"SourceFormat":""
}
]
}

arkiboys 9,706 Reputation points

2021-11-04T14:34:10.283+00:00

Please note that the source does not have imported schema as the source is used with a parameter to load data dynamically
If I import schema from a sample source, then I will get the columns array in the dropdown list as you see in screenshot below.
How can I set the unroll by to the columns array using dynamic expression so that way I do not have to iport schema of the source?

Accepted answer

1 additional answer

Your answer

arkiboys 9,706 Reputation points

2021-11-04T14:34:10.283+00:00

Please note that the source does not have imported schema as the source is used with a parameter to load data dynamically
If I import schema from a sample source, then I will get the columns array in the dropdown list as you see in screenshot below.
How can I set the unroll by to the columns array using dynamic expression so that way I do not have to iport schema of the source?

Answer 1

MarkKromer-MSFT 5,226 Microsoft Employee Moderator

Click on "Add dynamic content" next to the Unroll by field an enter an expression like this: byName('Columns')

arkiboys 9,706 Reputation points

2021-11-04T21:48:02.483+00:00

I placed this as shown in screen-shot

The error in the flatten data preview is
Spark job failed: {
"text/plain": "{\"runId\":\"14b093ec-36bd-4734-b704-121582035a29\",\"sessionId\":\"97177c8e-3dd5-40a5-b772-d54536b1856e\",\"status\":\"Failed\",\"payload\":{\"statusCode\":400,\"shortMessage\":\"DF-DSL-001 Parameter stream has parsing errors\nLine 2 Position 13: extraneous input ''' expecting {DECIMAL_LITERAL,
arkiboys 9,706 Reputation points

2021-11-04T21:49:45.767+00:00

Is there something wrong with my json data?
Mark Kromer MSFT 1,146 Reputation points

2021-11-04T21:51:39.237+00:00

Try without the toString(), you can ignore the UI warning
arkiboys 9,706 Reputation points

2021-11-04T21:57:57.937+00:00

Error:
java.lang.Throwable: Structure pattern returns no matches for stream 'Flatten1'
Narayanaswamy, M (Manjunath) 15 Reputation points

2023-12-28T15:57:57.03+00:00

I have a similar situation, in my case the column that i want to unroll by is type array.

It fails with the error :

Job failed due to reason: Structure pattern returns no matches for stream 'flattenJson'.

Also it was suggested to use typecasting, when i wraparound with toString [toString(byName('value')) ]it fails with the following error:

Job failed due to reason: com.microsoft.dataflow.Issues: DF-FOLDWN-002 - unrollby should reference a array or map column - EXE-0001,Dataflow cannot be analyzed as a graph,[564 737],

readJson foldDown(

) ~> flattenJson

Answer 2

Atanu Das 1

Do we have any update on this - facing the same issue?

Lokesh lehkara 1 Reputation point

2022-09-06T13:33:44.783+00:00

byName('Columns') also doesn't work. Does anyone found any solution to it.

Share via

data flow flatten json file

1 additional answer

Your answer