C# HttpClient and multighreading for millions of requests - fast

Question

I have an app where I need to make millions of requests as fast as possible each request using a different proxy and agent. Reading about HttpClient, its intended use is to create one instance for multiple requests, but I don’t see where/how a different proxy can be used for each request. Are there examples available on how to do this?

Or is there a better way to do this that’s thread safe?

Thank you.

Answer 1

@moondaddy

Maybe, the links would help you.

https://makolyte.com/csharp-how-to-make-concurrent-requests-with-httpclient/

You should look into the concerns of overloading an inbound port the a service is using.

https://codeburst.io/throttling-concurrent-outgoing-http-requests-in-net-core-404b5acd987b

You can take the code and make it work for a Windows desktop solution.

https://medium.com/streamwriter/managing-http-requests-in-asp-net-core-f2ee2fe46a1

Answer 2

HttpClient is not really designed for this case. you might be able to get to work with your own factory and careful setting of connection lifetimes. you also don't want a dictionary of pools. you will need the httpclient disposed, and short keepalives to actually close the connection (disposing a httpclient does not immediately close the connection due to keep-alive).

I'd probably use TCPClient instead. You will have much better control of the connection. You would also need your own http format handlers.