Bingbot; backoff after throttling

Jesus P 0 Reputation points
2023-07-13T05:08:15.86+00:00

Hello,

We have a website which is crawled by Bingbot and other bots. Sometimes, when our service is under extreme load, we have no other choice than throttle bot traffic to preserve the availability for our customers.

When throttling, we provide a HTTP error with retry-after so that bots can recover their traffic gracefully. We realized Bingbot doesn’t honor the retry-after. It doesn’t back off even we return the HTTP error 429 or 503 with retry-after (we experimented with both). Other bots like Google do back off and honor the retry-after to gradually recover their robot traffic.

My question is, how can we make Bingbot back off and gradually recover its traffic after throttling?

Please point me to the right place if this isn't the right forum for this kind of question. Thank you.

Reference: First asked in https://answers.microsoft.com/en-us/bing/forum/bing_webmaster/bingbot-backoff-after-throttling/7b069668-e7b8-477a-a046-7a20cb083c1f?rtAction=1689224357759

Community Center | Not monitored
{count} votes

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.