Read large json from data lake storage

Question

I have few large json files stored into ADLS gen2 ranges from 50 MB to 10 GB. I need a way to read these json files and get number of records in these json file. What can be best way to read big json files? I believe function app may not work due to timeout limit.

Thanks

Answer 1

Hello @Vikas Tiwari ,

Thanks for the question and using MS Q&A platform.

Azure Data Lake Storage Gen2 is a set of capabilities dedicated to big data analytics, built on Azure Blob Storage.

Data Lake Storage gen2 supports several Azure services. You can use them to ingest data, perform analytics, and create visual representations. For a list of supported Azure services, see Azure services that support Azure Data Lake Storage Gen2.

Several open source platforms support Data Lake Storage Gen2. For a complete list, see Open source platforms that support Azure Data Lake Storage Gen2.

For the data processing, there are a number of possible technologies, including HDInsight, Databricks, Azure Stream Analytics, and Azure Synapse (Analytics).

Note: I believe that Serverless SQL pool in Azure Synapse Analytics can help you do this: Query JSON files using serverless SQL pool in Azure Synapse Analytics

You can process the Json perform your aggregation and then Save to your Data Lake.

Hope this will help. Please let us know if any further queries.

------------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators