ADF reading json array of documents in Data flow

Jakub Ramut 96 Reputation points
2020-12-16T08:31:17.843+00:00

Hi,

In Data flow object I'd like to read data from json files from blob storage (created in a separate pipeline), but when I try to click "Import projection" in "Projection" the result is only one column _corrupt_record:
48711-image.png

The projection is successfully finished without any errors.
In "Source options" tab, Json settings we use "Array of documents" and the json is a valid array, e.g:
[
{
"PeriodId": 16,
"FactDdn": "account.status",
"FactValue": "2"
},
{
"PeriodId": 16,
"FactDdn": "ccm.rn.clear",
"FactValue": "0"
}
]

When I change the json file to the format:
{
"PeriodId": 16,
"FactDdn": "account.status",
"FactValue": "2"
},
{
"PeriodId": 16,
"FactDdn": "ccm.rn.clear",
"FactValue": "0"
}

and settings to "Document per line" everything is fine but it doesn't solve the problem because the results of the data flow should have the array of documents format (we need to read it from .net core and don't want to change anything there).

Is it some known issue or am I missing some settings?

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,663 questions
{count} votes

Accepted answer
  1. Jakub Ramut 96 Reputation points
    2020-12-21T07:44:59.827+00:00

    Hi,
    The problem was solved in a different way, I read the JSON as a document per line, and then at the end of the data flow transform it into the standard JSON file.
    Thanks for your help!

    Regards,
    Jakub Ramut


1 additional answer

Sort by: Most helpful
  1. Jakub Ramut 96 Reputation points
    2020-12-16T14:17:46.257+00:00

    Hi @HarithaMaddi-MSFT ,

    Thank you for your warm welcome.

    The full content from the example file:
    [{"PeriodId":16,"FactDdn":"account.status","FactValue":"2"}
    ,{"PeriodId":16,"FactDdn":"ccm.rn.clear","FactValue":"0"}
    ,{"PeriodId":16,"FactDdn":"ccm.rn.clprep","FactValue":"0"}
    ,{"PeriodId":16,"FactDdn":"ccm.rn.open","FactValue":"0"}
    ,{"PeriodId":16,"FactDdn":"ccm.rn.total","FactValue":"0"}
    ,{"PeriodId":16,"FactDdn":"diag.eftotdr","FactValue":"4"}
    ,{"PeriodId":16,"FactDdn":"diag.total","FactValue":"5"}
    ,{"PeriodId":16,"FactDdn":"efilesttap1","FactValue":"Success"}
    ,{"PeriodId":17,"FactDdn":"account.status","FactValue":"2"}
    ,{"PeriodId":17,"FactDdn":"ccm.rn.clear","FactValue":"0"}
    ,{"PeriodId":17,"FactDdn":"ccm.rn.clprep","FactValue":"0"}
    ,{"PeriodId":17,"FactDdn":"ccm.rn.open","FactValue":"0"}
    ,{"PeriodId":17,"FactDdn":"ccm.rn.total","FactValue":"0"}
    ,{"PeriodId":17,"FactDdn":"diag.eftotdr","FactValue":"4"}
    ,{"PeriodId":17,"FactDdn":"diag.total","FactValue":"7"}
    ,{"PeriodId":17,"FactDdn":"efilesttap1","FactValue":"Success"}
    ]

    it's a json file with content-type "application/octet-stream", the file should be read from the blob storage.
    I can't add it as attachment because json type is not supported here.

    The dataset for us is the json data set which pointing into the blob storage:
    48756-image.png48757-image.png


Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.