CSOM Throttle on websites having more than 3Lakhs lists

Amit 691 Reputation points
2021-02-24T06:30:04.253+00:00

I have a website having more than 3Lakhs lists. When I execute the below code, the user gets throttled. Please confirm how to fetch all lists in SharePoint Online CSOM.

Code:
Web web = clientContext.Web;
ListCollection targetlListCollection = web.Lists;
clientContext.Load(targetlListCollection, l => l.Include(x => x.Id, x => x.BaseTemplate, x => x.BaseType, x => x.DefaultViewUrl));
clientContext.RequestTimeout = -1;
clientContext.ExecuteQueryWithIncrementalRetry();

SharePoint Server Development
SharePoint Server Development
SharePoint Server: A family of Microsoft on-premises document management and storage systems.Development: The process of researching, productizing, and refining new or existing technologies.
1,591 questions
0 comments No comments
{count} votes

Accepted answer
  1. Jerryzy 10,566 Reputation points
    2021-02-25T05:48:34.987+00:00

    Hi @Amit ,

    Build a extension method like this in a static class:

       public static void ExecuteQueryWithIncrementalRetry(this ClientContext clientContext, int retryCount, int delay)  
            {  
                int retryAttempts = 0;  
                int backoffInterval = delay;  
                int retryAfterInterval = 0;  
                bool retry = false;  
                ClientRequestWrapper wrapper = null;  
                if (retryCount <= 0)  
                    throw new ArgumentException("Provide a retry count greater than zero.");  
                if (delay <= 0)  
                    throw new ArgumentException("Provide a delay greater than zero.");  
      
                // Do while retry attempt is less than retry count  
                while (retryAttempts < retryCount)  
                {  
                    try  
                    {  
                        if (!retry)  
                        {  
                            clientContext.ExecuteQuery();  
                            return;  
                        }  
                        else  
                        {  
                            //increment the retry count  
                            retryAttempts++;  
      
                            // retry the previous request using wrapper  
                            if (wrapper != null && wrapper.Value != null)  
                            {  
                                clientContext.RetryQuery(wrapper.Value);  
                                return;  
                            }  
                            // retry the previous request as normal  
                            else  
                            {  
                                clientContext.ExecuteQuery();  
                                return;  
                            }  
                        }  
                    }  
                    catch (WebException ex)  
                    {  
                        var response = ex.Response as HttpWebResponse;  
                        // Check if request was throttled - http status code 429  
                        // Check is request failed due to server unavailable - http status code 503  
                        if (response != null && (response.StatusCode == (HttpStatusCode)429 || response.StatusCode == (HttpStatusCode)503))  
                        {  
                            wrapper = (ClientRequestWrapper)ex.Data["ClientRequest"];  
                            retry = true;  
      
                            // Determine the retry after value - use the `Retry-After` header when available  
                            string retryAfterHeader = response.GetResponseHeader("Retry-After");  
                            if (!string.IsNullOrEmpty(retryAfterHeader))  
                            {  
                                if (!Int32.TryParse(retryAfterHeader, out retryAfterInterval))  
                                {  
                                    retryAfterInterval = backoffInterval;  
                                }  
                            }  
                            else  
                            {  
                                retryAfterInterval = backoffInterval;  
                            }  
      
                            // Delay for the requested seconds  
                            Thread.Sleep(retryAfterInterval * 1000);  
      
                            // Increase counters  
                            backoffInterval = backoffInterval * 2;  
                        }  
                        else  
                        {  
                            throw;  
                        }  
                    }  
                }  
                throw new MaximumRetryAttemptedException($"Maximum retry attempts {retryCount}, has be attempted.");  
            }  
      
            [Serializable]  
            public class MaximumRetryAttemptedException : Exception  
            {  
                public MaximumRetryAttemptedException(string message) : base(message) { }  
            }  
    

    Usage:

    5 retries in 10 seconds:

                        Web web = ctx.Web;  
                        ListCollection targetlListCollection = web.Lists;  
                        ctx.Load(targetlListCollection, l => l.Include(x => x.Id, x => x.BaseTemplate, x => x.BaseType, x => x.DefaultViewUrl));  
                        ctx.ExecuteQueryWithIncrementalRetry(5, 10);  
    

    Here is a official document about handing SharePoint Online Throttling in CSOM, I suggest you can have a reference:

    Avoid getting throttled or blocked in SharePoint Online

    Thanks
    Best Regards


    If an Answer is helpful, please click "Accept Answer" and upvote it.
    Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.

    0 comments No comments

0 additional answers

Sort by: Most helpful