unable to load 500 MB xml file using Azure data factory

Question

Hi All,
I have been trying to import a xml file around 500 MB, I can see the right schema in copy activity of ADF, while writing to SQL server table or parquet file its taking ages, roughly after 48 mins I have to cancel the operation, I have to process 20 files of the same size and that's how I will get it from third part source, I don't want to add any other process which is beyond data engineering team skill set and the tool organisation has chosen for ETL is Data Factory .
Note: Data integration unit is set to "Auto"

Answer 1

Hi @Syed Rashid Nizam ,

Welcome to Microsoft Q&A forum and thanks for posting you query.

I see that you are copying from ADLS to ADLS/SQL and assume that you are using Azure IR. If that is correct, then as per my analysis, you pipeline run had hit the current limit on xml support. A single large XML file is not well supported for now in ADF, it will hit the limit on the memory of Azure IR machine since each single xml file is treated a single document and needs to be hold in memory. As your XML file used is large, it might have used too many memory and this may cause some unexpected behavior even it didn't fail,

Currently, if you have such xml files, we suggest customer to use SHIR with larger memory size to workaround,

BTW, we are sharing this feedback to ADF product team to treat better support of single large xml file as a new feature request, and monitoring the requirement on this, but no concrete ETA for now.

I would also recommend you to please log a feature request in IDEAS forum here : https://feedback.azure.com/d365community/forum/1219ec2d-6c26-ec11-b6e6-000d3a4f032c
All the feedback shared in this forum are actively monitored and reviewed by ADF engineering team. Please do share the feedback link once it is posted.

Hope this info helps.

----------

• Please don't forget to click on and upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators