Bingbot; backoff after throttling

Question

Hello,

We have a website which is crawled by Bingbot and other bots. Sometimes, when our service is under extreme load, we have no other choice than throttle bot traffic to preserve the availability for our customers.

When throttling, we provide a HTTP error with retry-after so that bots can recover their traffic gracefully. We realized Bingbot doesn’t honor the retry-after. It doesn’t back off even we return the HTTP error 429 or 503 with retry-after (we experimented with both). Other bots like Google do back off and honor the retry-after to gradually recover their robot traffic.

My question is, how can we make Bingbot back off and gradually recover its traffic after throttling?

Please point me to the right place if this isn't the right forum for this kind of question. Thank you.

Reference: First asked in https://answers.microsoft.com/en-us/bing/forum/bing_webmaster/bingbot-backoff-after-throttling/7b069668-e7b8-477a-a046-7a20cb083c1f?rtAction=1689224357759