Send parallel requests

Article
01/05/2023

When your application needs to send a large number of requests to Dataverse you can achieve much higher total throughput by sending requests in parallel using multiple threads. Dataverse is designed to support multiple concurrent users, so sending requests in parallel leverages this strength.

Note

Sending parallel requests within a plug-in is not supported. More information: Do not use parallel execution within plug-ins and workflow activities

Optimum degree of parallelism (DOP)

Part of the service that Dataverse provides is managing resource allocation for environments. Production environments that are heavily used by many licenced users will have more resources allocated to them. The number and capabilities of the servers allocated may vary over time, so there is no fixed number that you should apply to get the optimum degree of parallelism. Instead, use the integer value returned from the x-ms-dop-hint response header. This value provides a recommended degree of parallelism for the environment.

When using Parallel Programming in .NET the default degree of parallelism depends on the number of CPU cores on the client running the code. If the number CPU cores exceeds the best match for the environment you may be sending too many requests. You can set the ParallelOptions.MaxDegreeOfParallelism Property to define a maximum number of concurrent tasks.

Service protection limits

One of the three facets monitored for service protection limits is the number of concurrent requests. By default this value is 52 but it may be higher. An error will be returned if the limit is exceeded. If you are depending on the x-ms-dop-hint response header value to limit the degree of parallelism, you should rarely hit this limit. If you encounter this error, you should reduce the number of concurrent threads.

There is a specific error returned when this limit is reached:

Error code	Hex code	Message
`-2147015898`	`0x80072326`	`Number of concurrent requests exceeded the limit of 52.`

You can also mitigate the liklihood of this error occurring by sending your requests to all the servers that support the environment by disabling server affinity.

Server affinity

When you make a connection to a service on Azure a cookie is returned with the response and all your subsequent requests will attempt to be routed to the same server unless capacity management forces it to go to another server. Interactive client applications, especially browser clients, benefit from this because it allows for re-using data cached on the server. Web browsers always have server affinity enabled and it cannot be disabled.

When sending requests in parallel from your client application, you can gain performance benefits by disabling this cookie. Each request you send will be routed any of the eligible servers. Not only does this increase total throughput, it also helps reduce impact of service protection limits because each limit is applied per server.

Following are some examples showing how to disable server affinity with .NET.

SDK for .NET
Web API

If you are using the ServiceClient or CrmServiceClient classes, add the following to the AppSettings node in the App.config file.

<add key="PreferConnectionAffinity" value="false" />

You can also set the value of the EnableAffinityCookie property with either the ServiceClient or CrmServiceClient

This can also be set using the ServiceClient(ConnectionOptions, Boolean, ConfigurationOptions) constructor using the ConfigurationOptions.EnableAffinityCookie property.

For .NET applications using HttpClient with the Web API, the following code shows how to disable cookies using a custom HttpMessageHandler.

HttpMessageHandler messageHandler = new OAuthMessageHandler(
    config,
    new HttpClientHandler() { UseCookies = false }
    );
HttpClient httpClient = new HttpClient(messageHandler)

If you are using dependency injection, you can use the ConfigurePrimaryHttpMessageHandler method to set the HttpClientHandler.UseCookies property to false.

IHostBuilder builder = Host.CreateDefaultBuilder();

builder.ConfigureServices(services =>
{
   services.AddHttpClient(
      name: "ClientName",
      configureClient: ConfigureHttpClientDelegate
      )
   .ConfigurePrimaryHttpMessageHandler(() =>
      new HttpClientHandler
      {
            UseCookies = false
      }
   );
});

Optimize your connection

When using .NET and sending requests in parallel, apply configuration changes like the following so your requests are not limited by default settings:

// Bump up the min threads reserved for this app to ramp connections faster - minWorkerThreads defaults to 4, minIOCP defaults to 4 
ThreadPool.SetMinThreads(100, 100);
// Change max connections from .NET to a remote service default: 2
System.Net.ServicePointManager.DefaultConnectionLimit = 65000;
// Turn off the Expect 100 to continue message - 'true' will cause the caller to wait until it round-trip confirms a connection to the server 
System.Net.ServicePointManager.Expect100Continue = false;
// Can decrease overall transmission overhead but can cause delay in data packet arrival
System.Net.ServicePointManager.UseNagleAlgorithm = false;

ThreadPool.SetMinThreads

This sets the minimum number of threads the thread pool creates on demand, as new requests are made, before switching to an algorithm for managing thread creation and destruction.

By default, the minimum number of threads is set to the processor count. You can use SetMinThreads to increase the minimum number of threads, such as to temporarily work around issues where some queued work items or tasks block thread pool threads. Those blockages sometimes lead to a situation where all worker or I/O completion threads are blocked (starvation). However, increasing the minimum number of threads might degrade performance in other ways.

The numbers you should use can vary according to the hardware. The numbers you use would be lower for a consumption based Azure function than for code running on a dedicated host with high-end hardware.

More information: System.Threading.ThreadPool.SetMinThreads

System.Net.ServicePointManager settings

With .NET Framework, ServicePointManager is a static class used to create, maintain, and delete instances of the ServicePoint class. Use these settings with the ServiceClient or CrmServiceClient classes. These settings should also apply when using HttpClient with Web API in .NET Framework, but with .NET Core Microsoft recommends settings in HttpClient instead.

DefaultConnectionLimit

This value is ultimately limited by the hardware. If it is set too high, it will be throttled by other means. The key point is that it should be raised above the default value, and at least equal to the number of concurrent requests you intend to send.

With .NET Core using HttpClient, this is controlled by the HttpClientHandler.MaxConnectionsPerServer and the default value is int.MaxValue.

More information:

Expect100Continue

When this property is set to true, the client will wait for a round-trip confirms a connection to the server. For HttpClient the default value of HttpRequestHeaders.ExpectContinue is false.

More information:

UseNagleAlgorithm

The Nagle algorithm is used to reduce network traffic by buffering small packets of data and transmitting them as a single packet. This process is also referred to as "nagling"; it is widely used because it reduces the number of packets transmitted and lowers the overhead per packet. Setting this to false can decrease overall transmission overhead but can cause delay in data packet arrival.

More information: System.Net.ServicePointManager.UseNagleAlgorithm

Examples

The following .NET examples show use of Task Parallel Library (TPL) with Dataverse.

SDK for .NET
Web API

The x-ms-dop-hint response value is available via the RecommendedDegreesOfParallelism property in either ServiceClient or CrmServiceClient. You should use this value when setting ParallelOptions.MaxDegreeOfParallelism when you use Parallel.ForEach.

These examples also show setting the EnableAffinityCookie property to false.

In this the examples below, the id values of the responses are added to a ConcurrentBag of Guids. ConcurrentBag provides a thread-safe unordered collection of objects when ordering doesn't matter. The order of the Guids returned by this method cannot be expected to match the order of the items sent in the entityList parameter.

Using ServiceClient with .NET 6 or higher

With .NET 6 or higher you can use the Parallel.ForEachAsync method with the asychronous methods included with ServiceClient, such as CreateAsync.

/// <summary>
/// Creates records in parallel
/// </summary>
/// <param name="serviceClient">The authenticated ServiceClient instance.</param>
/// <param name="entityList">The list of entities to create.</param>
/// <returns>The id values of the created records.</returns>
static async Task<Guid[]> CreateRecordsInParallel(
    ServiceClient serviceClient, 
    List<Entity> entityList)
{
    ConcurrentBag<Guid> ids = new();

    // Disable affinity cookie
    serviceClient.EnableAffinityCookie = false;

    var parallelOptions = new ParallelOptions()
    { MaxDegreeOfParallelism = 
        serviceClient.RecommendedDegreesOfParallelism };

    await Parallel.ForEachAsync(
        source: entityList,
        parallelOptions: parallelOptions,
        async (entity, token) =>
        {
            ids.Add(await serviceClient.CreateAsync(entity, token));
        });

    return ids.ToArray();
}

Using CrmServiceClient with .NET Framework

When using .NET Framework, the Clone method available in CrmServiceClient allows duplicating an existing connection to Dataverse so that you can use the Parallel.ForEach method.

/// <summary>
/// Creates records in parallel
/// </summary>
/// <param name="crmServiceClient">The authenticated CrmServiceClient instance.</param>
/// <param name="entityList">The list of entities to create.</param>
/// <returns>The id values of the created records.</returns>
static Guid[] CreateRecordsInParallel(
    CrmServiceClient crmServiceClient, 
    List<Entity> entityList)
{
   ConcurrentBag<Guid> ids = new ConcurrentBag<Guid>();

    // Disable affinity cookie
    crmServiceClient.EnableAffinityCookie = false;

   Parallel.ForEach(entityList,
      new ParallelOptions()
      {
            MaxDegreeOfParallelism = crmServiceClient.RecommendedDegreesOfParallelism
      },
      () =>
      {
            //Clone the CrmServiceClient for each thread
            return crmServiceClient.Clone();
      },
      (entity, loopState, index, threadLocalSvc) =>
      {
            ids.Add(threadLocalSvc.Create(entity));

            return threadLocalSvc;
      },
      (threadLocalSvc) =>
      {
            //Dispose the cloned crmServiceClient instance
            threadLocalSvc?.Dispose();
      }
   );
   return ids.ToArray();
}

The following static method example shows the use of an authenticated HttpClient that has been configured with a BaseAddress Property set to the Dataverse Web API Uri.

This method first sends a request using the WhoAmI function, but the data in the body of the response is not used. The x-ms-dop-hint response header value is what is needed. This value is used to set the ParallelOptions.MaxDegreeOfParallelism Property. Then it uses Parallel.ForEachAsync to send the requests using the those options.

The method then parses the id values of the records and sets them into a ConcurrentBag of Guids. ConcurrentBag provides a thread-safe unordered collection of objects when ordering doesn't matter. The order of the Guids returned by this method cannot be expected to match the order of the items sent in the entityList parameter.

/// <summary>
/// Creates account records in parallel
/// </summary>
/// <param name="client">Authenticated HttpClient instance</param>
/// <param name="entityList">List of JObject representing account records</param>
/// <returns>The id values of the created records.</returns>
static async Task<Guid[]> CreateRecordsInParallel(HttpClient client, List<JObject> entityList)
{
   ConcurrentBag<Guid> ids = new ConcurrentBag<Guid>();

   HttpResponseMessage whoAmIResponse = await client.SendAsync(new HttpRequestMessage
   {
         Method = HttpMethod.Get,
         RequestUri = new Uri(
         uriString: "WhoAmI",
         uriKind: UriKind.Relative)
   });

   int recommendedDegreeOfParallelism = int.Parse(whoAmIResponse.Headers.GetValues("x-ms-dop-hint").FirstOrDefault());

   var parallelOptions = new ParallelOptions() { MaxDegreeOfParallelism = recommendedDegreeOfParallelism };

   // Send the requests in parallel
   await Parallel.ForEachAsync(entityList, parallelOptions, async (jObject, token) =>
   {
        // Send the request to create an account record.
         var createResponse = await client.SendAsync(new HttpRequestMessage()
         {
            Method = HttpMethod.Post,
            RequestUri = new Uri(
            uriString: "accounts",
            uriKind: UriKind.Relative),
            Content = new StringContent(
               content: jObject.ToString(),
               encoding: System.Text.Encoding.UTF8,
               mediaType: "application/json")
         });

         string? uri = createResponse.Headers.GetValues("OData-EntityId").FirstOrDefault();

         // Parse the Id from the URI returned
         int firstParen = uri.LastIndexOf('(') + 1;
         int lastParen = uri.LastIndexOf(')');

         if (Guid.TryParse(uri[firstParen..lastParen], out Guid id))
         {
            ids.Add(id);
         }

   });

   return ids.ToArray();
}