ADF copy operation for excel files is not working when file size is more than 100 mb. Is there a reason for this?

Question

I am trying to copy an excel file which is over 100 MB and it is not getting copied. I am currently on Visual Studio subscription for azure. Is there any specific reason for this?

Answer 1

Hello @Vishnu Peesapati ,

Thanks for the question and using MS Q&A platform.

Today ADF, Excel in copy activity does not support streaming read. It must load whole excel file into memory - read data row by row, cell by cell. For this reason - it may encounter Out of Memory / Low Performance when working with the big files. Unfortunately, this is current scenario with regards to the Excel files within Copy Activity.

Having said that, I had further researched and came up with the below workarounds . You could try use one of the workarounds at your end to handle this.

1. Use dataflow activity to move big excel file into other data store. In dataflow, Excel supports streaming read with a less CPU/Memory consumption. You could move files with sizes in GB.
2. Split big excel file into several small ones i.e one 100 MB File to 10 * 10 MB Files, then use copy activity to move the excel folder.
3. You can manually convert/save big excel file as csv format, then use copy activity to move it.
4. Register and host the self-hosted integration runtime (SHIR) with powerful machine, then use copy activity to move big excel file into other data store with that IR.

Hope this will help. Please let us know if any further queries.

------------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators