Fatest way to access and REST API for 200K times using HTTPCLinet

Question

we are trying to access an API endpoint for 200K times. we are able to access it without an error with the following code snippet:

private static HttpClient clinet = new HttpClient();
public static async Task Main()
{
string auth_token = SASToken.CreateToken(resourceURi, Keyname, key);

            clinet.BaseAddress = new Uri(string.Format("https://{0}.servicebus.windows.net/", serviceNamespace));
            clinet.DefaultRequestHeaders.TryAddWithoutValidation("Authorization", auth_token);
            dynamic obj = new object();
            for (int i = 0; i < 50000; i++)
            {
                obj.randValues.Add(0.145f);
            }
            obj.sim_Id = "991521f2-632d-4bfe-af76-a216ff29a5ab";
            obj.startTime = "2020-11-18T06:10:58.325Z";
            obj.EventType = "OPT_Cycle";
            string Jdata= Newtonsoft.Json.JsonConvert.SerializeObject(obj);

            for (int i = 0; i < 200000; i++) 
            {
                HttpResponseMessage msg = await PostTelemetryAsync(Jdata);
                if (msg.StatusCode == System.Net.HttpStatusCode.Created)
                {
                    Console.WriteLine("SUcess Data :" + msg.StatusCode);
                }
                else 
                {
                    Console.WriteLine("fail :" + msg.StatusCode);

                }
            }
            Console.Write(testst);
            Console.ReadKey();
        }


private static async Task<HttpResponseMessage> PostTelemetryAsync(string data)
        {
            try
            {
                              var url = string.Format("{0}/messages", hubName);
System.IO.MemoryStream streamcontent = new System.IO.MemoryStream(Encoding.UTF8.GetBytes(data));
                    streamcontent.Position = 0;
                    HttpContent content = new StreamContent(streamcontent);`enter code here`
                    content.Headers.Add("ContentType", "application/json");
                    return await clinet.PostAsync(url, content);
}
catch (Exception ex)
            {
                throw ex;
            }
}

The above approach took a large time to complete(~180 mins).

is there any other approach to speed it up.

Answer 1

Hi @Ankarao ,

You can try to create a collection of tasks and add task(s) to that collection, then call WhenAll method to asynchronously wait for multiple Task or Task<TResult> objects to complete.

List<Task> all_tasks = new List<Task>();  
  
for (int i = 0; i < 200000; i++)  
{  
    all_tasks.Add(PostTelemetryAsync(Jdata));  
}  
  
await Task.WhenAll(all_tasks);  

---

If the answer is helpful, please click "Accept Answer" and upvote it.

Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.

With Regards,
Fei Han

Answer 2

Hi @Ankarao ,

So it's not really clear on what factors you are using along with the code, like the machine specs that this program is running etc, code improvement can only do so much but it also has to be matched with a relatively good server to do the processing.

A couple of things you can do besides what FeiHan already suggested is checking out the TPL pattern and Parallel.Invoke for async tasks to fully utilize the machine cores with parallel execution.

Another point and this is based purely based on the code you posted, why do you have to keep creating a stream in loop every time in post telemetry, you are anyways sending Json content type then why convert in memory stream, I believe it will save a lot of time there, use the var httpContent = new StringContent(stringPayload, Encoding.UTF8, "application/json"); this you already have available before the 200k loop is called and pass that in PostAsync instead of creating a stream every single time.

Quick Tip: Use .Net core JsonSerealizer over Newtonsoft.

Hope this helps.