Need assistance to cope with Azure Automation PnP PowerShell Runbook throttling

Question

Hello everybody,

I am in the middle of building a solution using SharePoint, Azure Logic Apps and Azure Automation PowerShell runbooks.

I have the following scenario:
We have this ERP kind-off system that allows my (B2C) customer (kitchen company) to register quotes, orders, and invoices. For document management reasons they wanted to use SharePoint Online.
For each customer they have, they want to have a SharePoint team site that allows them to store and tag (managed metadata) documents and pictures, and take notes.
They have more than 7.000 unique (B2C) customers. And because one of their registered customers can walk in anytime, they want to have a team site for all of the accounts registered within the system. That’s why the amount of 7.000+.

I have built a Logic App workflow that automatically creates a new team site if a CSV file contains a new customer, and additionally it will create document sets for each kitchen project. That is not done by hand, so from within the Logic App workflow I am running a Azure Automation PowerShell runbook containing PnP-PowerShell cmdlets to fully provision the site, think about:

• Enabling site and site collection features
• Adding libraries and navigation items
• Adding site columns and content types
• Adding the content types to the lists
• Configuring permissions
• Etc.

As you may notice this takes a little while. Actually all this works perfectly!

Now the real question.
Every hour I want to import a new CSV file that contains new invoices and orders, across all accounts in the system. And I would like to include ‘throttling prevention’ mechanisms but I really need some help with that because it is my first time dealing with such a large amount of team sites and data to be processed. I am hoping to find a jumpstart here.

• I did read several articles on the web and I did find some resources that may help:
• I did read there are concurrency control possibilities from within an Azure Automation Runbook
• o https://learn.microsoft.com/en-us/azure/automation/manage-runbooks#prevent-concurrent-jobs, and
• o https://social.msdn.microsoft.com/Forums/en-US/06f1b011-1fe2-4a3a-a0a3-88256f83a3ea/can-i-prevent-multiple-instances-of-the-same-runbook-from-running-concurrently?forum=azureautomation
  But for me this looks as an inefficient approach.
• I did read that it is recommended to use more than one Office 365 account to perform “automated data migration” (https://learn.microsoft.com/en-us/archive/blogs/sposupport/data-migration-and-spo-service-throttling). I can imagine that this also applies to site provisioning
• I did read “PnP PowerShell is recommended since it handles request throttling internally.”. But I have not seen my PnP cmdlets retry any action, but instead throwing an exception error 429.

Could anyone please help me provide some guidance how to apply throttling mechanisms? Especially for my PnP PowerShell script. As I could not really find PnP code examples and I have no clue how to start there.

The desired outcome is that the PnP PowerShell runbook manages the retries by itself, and Logic Apps as well.
Maybe I should start with the Logic app trigger concurrency? https://learn.microsoft.com/en-us/azure/logic-apps/logic-apps-workflow-actions-triggers#change-trigger-concurrency
But if I would let 5 jobs run concurrently, it could still hit the throttling limit, so I guess I need to apply some measures inside my PnP script as well.

I hope this is clear enough, if not I will be happy to elaborate.

Thanks a lot for all help provided, I really appreciate all the efforts!

Kind regards,
Joost

Answer 1

Thanks for reaching out @jvdlinden !! Here are few documents which I found and which might be useful for you in applying throttling mechanisms. Kindly check it out and revert if you have further questions.

• Avoid getting throttled or blocked in SharePoint Online
• Handle SharePoint Online throttling by using exponential back off

Hope this helps!!

Answer 2

PnP PowerShell has built in functionality to deal with the 429 and retry the execution. This will happen blindly on the background. It will retry it by default 10 times with a 500 ms delay the first time and this time doubled on each subsequent attempt. If it still fails after this, you should see a MaximumRetryAttemptedException being returned. If this is not the case and you still see a 429 being returned, something is wrong and you can reach out for help through https://github.com/pnp/PnP-PowerShell/issues.

Some things to try first yourself as PnP PowerShell is community effort based on spare time of volunteers, so we're unable to handle the volume of questions we get at the moment: try running the same PnP PowerShell scripts locally and see if you get the same results. If you do, install and run Fiddler while you execute your script and see what the network traffic behind the cmdlets shows you. Are there really no retries happening? If there aren't, detail in the issue you create on the beforementioned link which connect-pnponline method you're using and provide more info about what you're trying to do. Do not include the Fiddler capture in the issue, as it may contain sensitive tokens which you don't want disposed publicly. Keep the Fiddler trace stored somewhere safe though, as we may reach out to you to share it privately to aid us in troubleshooting.