C# HttpClient and multighreading for millions of requests - fast

asked 2021-05-03T19:36:26.127+00:00
moondaddy 856 Reputation points

I have an app where I need to make millions of requests as fast as possible each request using a different proxy and agent. Reading about HttpClient, its intended use is to create one instance for multiple requests, but I don’t see where/how a different proxy can be used for each request. Are there examples available on how to do this?

Or is there a better way to do this that’s thread safe?

Thank you.

C#
C#
An object-oriented and type-safe programming language that has its roots in the C family of languages and includes support for component-oriented programming.
6,905 questions
No comments
{count} votes

2 answers

Sort by: Most helpful
  1. answered 2021-05-03T22:02:05.197+00:00
    Duane Arnold 3,211 Reputation points

    @moondaddy

    Maybe, the links would help you.

    https://makolyte.com/csharp-how-to-make-concurrent-requests-with-httpclient/

    You should look into the concerns of overloading an inbound port the a service is using.

    https://codeburst.io/throttling-concurrent-outgoing-http-requests-in-net-core-404b5acd987b

    You can take the code and make it work for a Windows desktop solution.

    https://medium.com/streamwriter/managing-http-requests-in-asp-net-core-f2ee2fe46a1


  2. answered 2022-11-15T18:43:00.283+00:00
    Bruce (SqlWork.com) 27,846 Reputation points

    HttpClient is not really designed for this case. you might be able to get to work with your own factory and careful setting of connection lifetimes. you also don't want a dictionary of pools. you will need the httpclient disposed, and short keepalives to actually close the connection (disposing a httpclient does not immediately close the connection due to keep-alive).

    I'd probably use TCPClient instead. You will have much better control of the connection. You would also need your own http format handlers.

    No comments