copy and divide large json file

Gerald Rupp 130 Reputation points
2023-07-18T12:53:57.4533333+00:00

Hi everybody,
I want to store all JSON files from one container to another container in Azure Data Factory. The JSON files are very large, so I would like to divide one JSON file into several smaller JSON files. Could you tell me how to proceed?

with kind regards,

Gerald

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,222 questions
{count} votes

Accepted answer
  1. KranthiPakala-MSFT 46,602 Reputation points Microsoft Employee
    2023-07-19T20:38:35.43+00:00

    @Gerald Rupp Thanks for using Microsoft Q&A forum and posting your query.

    As per my understanding, you would like to split the large JSON files into smaller JSON files before copying them from Container A to Container B. Please correct me if I miss anything.

    At this point I'm not sure of the structure of your JSON file and how you are expecting the structure to be in your sink after splitting them. But I'm sharing a generic approach using Azure Data Factory mapping data flows. You can choose the specific options in Data flow setting based on your specific need.

    You can use mapping data flow to read through your source JSON files which are large and set the partition type on the sink transformation as shown below to copy them as smaller json files:

    User's image

    For more details about each of these partition types, please refer to this document: Data flow performance - Optimize tab

    Hope this info helps.


    Please don’t forget to Accept Answer and Yes for "was this answer helpful" wherever the information provided helps you, this can be beneficial to other community members.

    1 person found this answer helpful.
    0 comments No comments

0 additional answers

Sort by: Most helpful

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.