Checks scalability

As customers shifted their release pipelines to YAML from classic release management, the first challenge we encountered was handling checks at high scale. Two of the most commonly used checks are the Azure functions and REST API checks. The default experience in Azure Pipelines results in creating sychronous calls from Azure Pipelines into Azure functions or your REST server. On top of that, these calls are retried frequently until the check passes. This design is simple and works well for small workloads. However, it does not scale when there are a large number of checks, when the function or REST server takes more time to respond, or when there are a large number of retries.

We will streamline our asynchrononous checks implementation, also known as the callback model, and change the experience in the product to recommend that over the synchronous model. Using the callback model, the Azure function or REST server can decide to fail or pass the check at any time before a timeout is reached, thereby eliminating the need for retries altogether. This model also allows the Azure function or REST server to take as much time as it needs to evaluate the check without consuming compute resources in Azure Pipelines.