C# HttpClient and multighreading for millions of requests - fast

moondaddy 896 Reputation points
2021-05-03T19:36:26.127+00:00

I have an app where I need to make millions of requests as fast as possible each request using a different proxy and agent. Reading about HttpClient, its intended use is to create one instance for multiple requests, but I don’t see where/how a different proxy can be used for each request. Are there examples available on how to do this?

Or is there a better way to do this that’s thread safe?

Thank you.

C#
C#
An object-oriented and type-safe programming language that has its roots in the C family of languages and includes support for component-oriented programming.
9,940 questions
0 comments No comments
{count} votes

2 answers

Sort by: Most helpful
  1. Duane Arnold 3,221 Reputation points
    2021-05-03T22:02:05.197+00:00

    @moondaddy

    Maybe, the links would help you.

    https://makolyte.com/csharp-how-to-make-concurrent-requests-with-httpclient/

    You should look into the concerns of overloading an inbound port the a service is using.

    https://codeburst.io/throttling-concurrent-outgoing-http-requests-in-net-core-404b5acd987b

    You can take the code and make it work for a Windows desktop solution.

    https://medium.com/streamwriter/managing-http-requests-in-asp-net-core-f2ee2fe46a1


  2. Bruce (SqlWork.com) 52,496 Reputation points
    2022-11-15T18:43:00.283+00:00

    HttpClient is not really designed for this case. you might be able to get to work with your own factory and careful setting of connection lifetimes. you also don't want a dictionary of pools. you will need the httpclient disposed, and short keepalives to actually close the connection (disposing a httpclient does not immediately close the connection due to keep-alive).

    I'd probably use TCPClient instead. You will have much better control of the connection. You would also need your own http format handlers.

    0 comments No comments