question

PeterSharp-4330 avatar image
0 Votes"
PeterSharp-4330 asked SaurabhSharma-msft commented

Use Data Factory to generate 'Array of documents' JSON from non-JSON source

I'm loading some data (from a parquet file, as it happens), and after some filtering and preparation, I'm wanting to write the output back to a JSON sink.

I have no issues getting the data, or even writing it. My issue is that the output it generates is as below.

 {"meterId":"21IB04060","servicePointId":"47434","transmitterId":"4CAE000040606B07","rowTimestamp":"2021-09-28 13:00:00","rowType":"Index","rowValue":436574,"rowChange":2449}
 {"meterId":"21IB04060","servicePointId":"47434","transmitterId":"4CAE000040606B07","rowTimestamp":"2021-09-28 14:00:00","rowType":"Index","rowValue":437234,"rowChange":660}

I cant then use this JSON file because it's written in 'Document per line' format. Essentially, this string is invalid JSON (as tested with JSONLint).

Valid JSON would be the following.

 [{"meterId":"21IB04060","servicePointId":"47434","transmitterId":"4CAE000040606B07","rowTimestamp":"2021-09-28 13:00:00","rowType":"Index","rowValue":436574,"rowChange":2449},
 {"meterId":"21IB04060","servicePointId":"47434","transmitterId":"4CAE000040606B07","rowTimestamp":"2021-09-28 14:00:00","rowType":"Index","rowValue":437234,"rowChange":660}]

So, my question is, how can I produce this in a sensible way? I've read around the forums that running a second flow to read the JSON output I am generating and setting the format there, I can then output the JSON in the desired format. This seems like a crazy workaround however.

Any suggestions would be awesome.

Thanks

azure-data-factory
· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @petersharp-4330,
Thanks for using Microsoft Q&A !!

I have tested in my environment by importing data from Parquet to a Json File and I could see the same behavior.

136404-image.png


I am checking internally with the products team and get back to you on the same.

Thanks
Saurabh

0 Votes 0 ·
image.png (33.1 KiB)

1 Answer

SaurabhSharma-msft avatar image
0 Votes"
SaurabhSharma-msft answered SaurabhSharma-msft commented

Hi @petersharp-4330,
As per internal communication it looks like a limitation at sink side and the products team is constantly updating JSON and Parquet file processing in data flow and Copy JSON Sink has more options as of now.
So you need to go for workarounds like Copy Activity for now.

Thanks
Saurabh

· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @petersharp-4330,
Please provide your feedback at Azure Feedback site.

Thanks
Saurabh


0 Votes 0 ·