Merging millions of json files from azure blob

Jardar Maatje 1

Hi,

we have an Azure blob container with millions of small json files. I have successfully been able to setup a copy task in azure data factory for merging these files into one file that will be more manageable for further processing, preferably in data lake. Right now the destination is a CSV file in another container.

However this takes ages, so I need help in finding the most performant way to approach this.

So what is the most efficient way to do this? Both in terms of setup, but also i choosing sink and format.

In the end I would like to have the data stored in Azure Data Lake for analyis.

HimanshuSinha-msft 19,381 Reputation points Microsoft Employee

2020-09-14T23:43:10.527+00:00

Hello @Jardar Maatje ,

Thanks for the ask and also for using the forum .

My understanding is that your goal is to get all the data in these million JSON file to a single CSV file . You have not mentioned as what is the parallelism you are using in the copy activity . Please do take some time and request you to go through this ( if not doen already ) . From an implemenation point of view i could request you to please see if you can use mapping dataflow . Mapping DF runs on databricks which can very helps as different cluster can pick up different buckets of JSON file and process that . Also is there a way you can bucketize your 1 million json file , may be created year / month etc , this will help partioning the data and improve ingestion time .

Thanks
Himanshu
Jardar Maatje 1 Reputation point

2020-09-15T12:04:04.967+00:00

Hi,

thanks for the response. I have had a look at the link you sent me, and it seems that the problem is that when merging into a single file the amount of parallelism is limited. However when inserting into a CosmosDb I improved on performance significantly.
HimanshuSinha-msft 19,381 Reputation points Microsoft Employee

2020-09-15T23:14:01.957+00:00

Hello ,
You are right , the write operations on a single file on the sink will slow the operation down . One thing worth trying is the converge approach , 1 million file -> 1000(may be ) files - > 10 file - > 1 file .
Also if you can use a relation DB that should perform better the single file idea .

Let me know if you have further query .

Thanks
Himanshu
HimanshuSinha-msft 19,381 Reputation points Microsoft Employee

2020-09-21T20:18:08.443+00:00

Hello ,
We have not heard back from you on this and was just following up ,if the issue is resolved .In case if you have a resolution we request you to share that with the community .

Thanks
Himanshu
HimanshuSinha-msft 19,381 Reputation points Microsoft Employee

2020-09-24T17:33:31.707+00:00

Hello ,

We have not heard back from you yet on this and was just following up ,if the issue is resolved .In case if you have a resolution we request you to share that with the community .

Thanks
Himanshu
Dimeji Olayinka 37 Reputation points

2022-07-27T13:32:20.74+00:00

What link were you looking at?