Uploading 1000's of files to a website

Question

We are using multiple file upload control in asp.net and using jQuery ajax to send files. All working great. But the requirement is to allow up to a 1000 files to be uploaded. This has impact on server cpu and memory used. Also crashes the browser tab while doing it.

What are our options?

ASP.NET/C# .NET Framework 4.7

Answer 1

ftp is better suited, or zip the files first and extract on server.

Answer 2

Hi @Ali Khan ,
Maybe you can try splitting the file into chunks and uploading the checks to the server.
Or pass the file object to XMLHttpRequest without loading the file into memory so that the browser can stream the file contents in the upload form.
Best regards,
Lan Huang

---

If the answer is the right solution, please click "Accept Answer" and kindly upvote it. If you have extra questions about this answer, please click "Comment".
Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.

Answer 3

Gee, if you had said about 500?
Then I would consider the ajaxfile uploader in the ajaxtool kit.

Since you can set the packet size and that it chunks (up-loads in smaller parts), then I have found that:

Memory foot print is small.
No real limits on the size of the file being up-loaded.
Allows and handles say 100 files, even 200 with great ease.
Does not over load the server - since each of those chunks of a file setup are processed, and that tends to give the server a bit of time to gulp some air, or take a breath - and thus other users are not really much impacted by the up-loader.

And I have tested the ajaxfile file up-loader to about 800 files.

However, I think the UI would (should be) changed, since really, at the end of the day?

1000 files to be uploaded is not really a job for a web browser interface. I would suggest that you have them use some FTP client software, and ftp does allow wild cards etc., and it would quite much made for this type of transfer.

And, the other suggestion was to have that end user zip up the files they way, and thus you only up-load one zip file.

The ajaxtoolkit is for asp.net webforms, and you can install it by nuget. I think that toolkit up-loader is quite happy up to say 500 files. However, I just think that anything beyond 100 files is too much for a web interface anyway.