ADF: Unable to load json file >100 MB into a database column

Question

Hi,

I have a requirement to load the json files (ADF Logs) into our database via Azure Data Factory (Copy Activity). The datatype of the column is "nvarchar". There will be around 20 files/day and the file at 21:00 hrs is having size more than 100 MB. Below is the folder structure.

I am able to load files less than 100 MB. When the copy activity come across files > 100 MB it throws an error as below. If I delete the file which is greater than 100 MB, then the copy activity succeeds.

ErrorCode=DelimitedTextIncorrectRowDelimiter,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The specified row delimiter is incorrect. Cannot detect a row after parse 100 MB data.,Source=Microsoft.DataTransfer.Common,'

Below is the "source", "source dataset" and "sink" in the copy activity

How to resolve this error? Can someone please help me.

Answer 1

Hello @Pashikanti Ranadheer ,

Thanks for the question and using MS Q&A platform.

For the source Dataset ( ScreenShot 2) , you could set the MaxRowSize and see whether this resolves your issue.

You could click on the code view of the source dataset and add the property MaxRowSize and set it to a number greater 100 MB in bytes - for instance, to set 200 MB

"MaxRowSize" :  209715200   

Hope this will help. Please let us know if any further queries.

------------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators