ADF reading json array of documents in Data flow

Question

ADF reading json array of documents in Data flow

Jakub Ramut 96

Hi,

In Data flow object I'd like to read data from json files from blob storage (created in a separate pipeline), but when I try to click "Import projection" in "Projection" the result is only one column _corrupt_record:

The projection is successfully finished without any errors.
In "Source options" tab, Json settings we use "Array of documents" and the json is a valid array, e.g:
[
{
"PeriodId": 16,
"FactDdn": "account.status",
"FactValue": "2"
},
{
"PeriodId": 16,
"FactDdn": "ccm.rn.clear",
"FactValue": "0"
}
]

When I change the json file to the format:
{
"PeriodId": 16,
"FactDdn": "account.status",
"FactValue": "2"
},
{
"PeriodId": 16,
"FactDdn": "ccm.rn.clear",
"FactValue": "0"
}

and settings to "Document per line" everything is fine but it doesn't solve the problem because the results of the data flow should have the array of documents format (we need to read it from .net core and don't want to change anything there).

Is it some known issue or am I missing some settings?

HarithaMaddi-MSFT 10,156 Reputation points

2020-12-16T13:18:21.503+00:00

Hi @Jakub Ramut ,

Welcome to Microsoft Q&A Platform. Thanks for posting the query.

I was unable to reproduce the issue when I copied the content shared in a JSON file and used "Array of documents" option in source options of dataflow as below. Please share mock file if feasible for us to investigate it further.

Looking forward for your response!

Answer accepted by question author

Jakub Ramut 96

Hi,
The problem was solved in a different way, I read the JSON as a document per line, and then at the end of the data flow transform it into the standard JSON file.
Thanks for your help!

Regards,
Jakub Ramut

HarithaMaddi-MSFT 10,156 Reputation points

2020-12-22T12:15:55.917+00:00
Thanks @Jakub Ramut for sharing the resolution. I converted your comment to an answer so that it can be useful for community members.

---

Please accept an answer if correct. Original posters help the community find answers faster by identifying the correct answer. Here is how.

Want a reminder to come back and check responses? Here is how to subscribe to a notification.

1 additional answer

Your answer

HarithaMaddi-MSFT 10,156 Reputation points

2020-12-16T13:18:21.503+00:00

Hi @Jakub Ramut ,

Welcome to Microsoft Q&A Platform. Thanks for posting the query.

I was unable to reproduce the issue when I copied the content shared in a JSON file and used "Array of documents" option in source options of dataflow as below. Please share mock file if feasible for us to investigate it further.

Looking forward for your response!
HarithaMaddi-MSFT 10,156 Reputation points

2020-12-22T12:15:55.917+00:00

Thanks @Jakub Ramut for sharing the resolution. I converted your comment to an answer so that it can be useful for community members.

---

Please accept an answer if correct. Original posters help the community find answers faster by identifying the correct answer. Here is how.

Want a reminder to come back and check responses? Here is how to subscribe to a notification.

Answer 1

Hi @HarithaMaddi-MSFT ,

Thank you for your warm welcome.

The full content from the example file:
[{"PeriodId":16,"FactDdn":"account.status","FactValue":"2"}
,{"PeriodId":16,"FactDdn":"ccm.rn.clear","FactValue":"0"}
,{"PeriodId":16,"FactDdn":"ccm.rn.clprep","FactValue":"0"}
,{"PeriodId":16,"FactDdn":"ccm.rn.open","FactValue":"0"}
,{"PeriodId":16,"FactDdn":"ccm.rn.total","FactValue":"0"}
,{"PeriodId":16,"FactDdn":"diag.eftotdr","FactValue":"4"}
,{"PeriodId":16,"FactDdn":"diag.total","FactValue":"5"}
,{"PeriodId":16,"FactDdn":"efilesttap1","FactValue":"Success"}
,{"PeriodId":17,"FactDdn":"account.status","FactValue":"2"}
,{"PeriodId":17,"FactDdn":"ccm.rn.clear","FactValue":"0"}
,{"PeriodId":17,"FactDdn":"ccm.rn.clprep","FactValue":"0"}
,{"PeriodId":17,"FactDdn":"ccm.rn.open","FactValue":"0"}
,{"PeriodId":17,"FactDdn":"ccm.rn.total","FactValue":"0"}
,{"PeriodId":17,"FactDdn":"diag.eftotdr","FactValue":"4"}
,{"PeriodId":17,"FactDdn":"diag.total","FactValue":"7"}
,{"PeriodId":17,"FactDdn":"efilesttap1","FactValue":"Success"}
]

it's a json file with content-type "application/octet-stream", the file should be read from the blob storage.
I can't add it as attachment because json type is not supported here.

The dataset for us is the json data set which pointing into the blob storage:

HarithaMaddi-MSFT 10,156 Reputation points

2020-12-17T11:31:14.787+00:00

Hi @Jakub Ramut ,

Thanks for sharing the details. I am unable to reproduce the issue with the sample content provided. Apologies for inconvenience in adding the JSON file, it can be attached by saving as .txt file. However, for more deeper investigation, I would request you to send a mail to AzCommunity[at]microsoft[dot]com with subject "Attn:Haritha - ADF reading json array of documents in Data flow" and we will work with you in person to look at the issue.

Thanks for your support! Looking forward for your response.