Rate limiting for for file scanning

Question

As per this doc, https://learn.microsoft.com/en-us/sharepoint/dev/general-development/how-to-avoid-getting-throttled-or-blocked-in-sharepoint-online ...an app running in an org with 4K people gets a quota of 2.4M REST requests a day.

Apply this math to app that needs to scan files. If the app makes 1 REST call per file and the average file size is 0.1M , that means the fastest the app can scan is 240Gigs / day. This rate is laughably low. An organization with 4K people will typically have 100 TB of data in their O365. It will take them forever to complete their initial scan of data

What am I missing?

Answer 1

Hi vikram,
Thanks for posting on the Q&A forum. the chart you are referring to is below:

License count	0 – 1k	1k – 5k	5k - 15k	15k - 50k	50k+
App 1 minute	1,200	2,400	3,600	4,800	6,000
App daily	1,200,000	2,400,000	3,600,000	4,800,000	6,000,000

The assumption of 1 resource per cost unit can in fact be higher depending on the operation in question.

Resource units per request	Operations
1	Single item query, such as get item
	Delta with a token
2	Multi item query, such as list children, except delta with a token
	Create, update, delete and upload
5	All permission resource operations, including $expand=permissions

As mentioned below:

Since application limits are in resource units, the actual request rate, such as requests per minute, depends on application’s API choice and the corresponding API resource unit cost. In general, you can estimate the request rate using an average of 2 resource units per request and divide resource unit limits by 2 to get the estimated request rate.

Note that best practices for discovering files and detecting changes at scale are also noted.

The being said, these rates are so that if an overwhelming number of requests occurs, throttling helps maintain optimal performance and reliability of the Microsoft Graph service.

For consideration, is that solutions that need to extract a large volume of data from Microsoft Graph should use Microsoft Graph Data Connect instead of the Microsoft Graph REST APIs. Microsoft Graph Data Connect allows organizations to extract Microsoft 365 data in bulk without being subject to throttling limits.

--please don't forget to upvote and Accept as answer if the reply is helpful--

Answer 2

Yes I am aware that some API calls are more expensive than others. I was making the point that even in near-impossible case, where a single file costs only 1 unit, it still works out to a very slow performance.

Graph data connect seems interesting but looking at the supported datasets , it doesn't look like the file content from onedrive/sharepoint is available in there