"Out of Memory" issue by using ADF copy activity process large size Excel file

Question

Our client is using ADF copy activity processing large size excel file( includes more than 30 columns, file size around 80-10MBs ) which meet "Out of Memory" error intermittently. We understand this is a known issue in ADF which cause the error. We applied workaround which bypass the issue. Our client asks when this known issue can be fixed. Is there any ETA for the permanent solution?

Answer 1

HI @Jinlei Wang ,

Thanks for your query. At this time we do not have an ETA for the permanent fix but product team recommends to use Mapping Data flow instead of Copy activity to avoid this limitation. You can use dataflow activity to move thousands of big excel files into other data store. In dataflow, the excel has supported streaming read with few CPU/Memory consumption. Based on internal perf test, it can move/transfer 1GB excel file in dataflow within 5 minutes while using Azure Integration runtime with 32 cores.

Also if you have any specific feedback with copy activity limitation of copying big excel files, I would recommend you to please log your feedback in Azure feedback forum here: https://feedback.azure.com/d365community/forum/1219ec2d-6c26-ec11-b6e6-000d3a4f032c

Here is an existing feedback logged by another user on the same request: https://feedback.azure.com/d365community/idea/22728378-6f26-ec11-b6e6-000d3a4f032c. I would encourage you to please up-vote and/or comment on it to increase the priority of it.

Hope this info helps. Do let us know if you have further query.

----------

• Please don't forget to click on and upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators