dataflow fails reading .json

arkiboys 9,701 Reputation points
2023-06-23T12:07:45.6733333+00:00

hello,

the dataflow in the ADF takes .json files and processes them.

Most of the time it works fine in all environments.

forexample, today, prod is fine, whereas other environments failed

Recently, sometimes, in different environments, I see this message as the dataflow job fails

Job failed due to reason: at Source 'source1': Malformed records are detected in schema inference. Parse Mode: FAILFAST. It could be because of a wrong selection in document form to parse json file(s). Please try a different 'Document form' (Single document/Document per line/Array of documents) on the json source

Any suggestions?

thank you

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,246 questions
{count} votes

2 answers

Sort by: Most helpful
  1. QuantumCache 20,346 Reputation points
    2023-06-23T19:19:50.2966667+00:00

    Hello @arkiboys

    There might be some malformed records in the JSON files that are causing the schema inference to fail.

    check if there are any changes in the JSON files that are causing the issue. You can compare the JSON files that are processed successfully with the ones that are failing to see if there are any differences.

    If the issue persists, you can try to isolate the problem by running the data flow with a smaller set of JSON files. This can help you identify if the issue is related to a specific file or a group of files.


  2. Subashri Vasudevan 11,221 Reputation points
    2023-06-25T01:33:24.0666667+00:00

    Hello @arkiboys

    Hello @arkiboys

    Thanks for your ask and using the MS q&A portal. Since you say that it gets automatically fixed , file should not have any issue. You may check if there are any changes to the other environment pipeline which is not yet pushed to prod. Or since it is lower lanes, is there a possibility of somebody modifying the file for testing purpose? Please check by comparing files in prod and lower lane and let us know.

    Thanks

    0 comments No comments

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.