Deploying Azure Static WebApps with Terraform giving "Failure sending request: StatusCode=429"

Question

I have a terraform code which deploys few resources: 1 Application Gateway, 30 Static WebApps, 1 Key Vault. It was working fine for few months but now it is giving below error while terraform plan. Sometimes plan works fine but then same error is given by terraform apply.

web.StaticSitesClient#ListStaticSiteAppSettings: Failure sending request: StatusCode=429 -- Original Error: context deadline exceeded

Searching shows that error 429 is encountered when Azure APIs are throttling. I stated a terraform plan on my machine and started a loop to access the Azure API using Azure CLI: az staticwebapp appsettings list --name. But the CLI request were executing successfully. My understanding was that if Azure API requests from terraform are being throttled, then they should be throttled from CLI too (using same user account).

I tried reducing the parallelism count in terraform, but it doesn't help.

Please help me troubleshoot this issue.

Answer 1

Hi@Rajiv Bansal We are sorry to hear you're facing this issue. You're correct, the error you’re encountering, StatusCode=429, is related to Azure API throttling.

Azure Resource Manager applies throttling at two levels: the subscription and tenant, and the resource provider. The requests are throttled per hour. The default throttling limits per hour are as follows:

• Subscription reads: 12000
• Subscription deletes: 15000
• Subscription writes: 1200
• Tenant reads: 12000
• Tenant writes: 1200

These limits are scoped to the security principal (user or application) making the requests and the subscription ID or tenant ID.

Microsoft recently started migrating Azure subscriptions to a new throttling architecture. The new throttling limits are applied per region rather than per instance of Azure Resource Manager. The new architecture uses a token bucket algorithm to manage API throttling.

As for the throttling window, it depends on the time window of the API throttling violation. If unsure about the nature of throttling violations, then waiting the full 30 minutes is the most conservative tactic to take.

Here are a few solutions to troubleshoot this issue:

• Terraform offers built-in mechanisms for retrying failed operations with a backoff delay. Configure Terraform to automatically retry failed requests with an exponential backoff approach. This allows time for the throttling window to reset and increases the chance of successful execution. You can achieve this using the retry and retry_delay options in your Terraform configuration.
• If your deployment involves creating a large number of Static Web Apps (30 in your case), consider splitting it into smaller Terraform configurations. This can potentially reduce the number of API calls made within a short timeframe and mitigate throttling.
• As an alternative approach, explore using Azure Resource Manager (ARM) templates for deploying your Static Web Apps. ARM templates offer more granular control over resource creation and might have different throttling behavior compared to the Static Web Apps API used by Terraform.

By implementing these recommendations and monitoring your deployment process, you should be able to mitigate the throttling issue and successfully execute your Terraform code for deploying Static Web Apps in Azure.

Hope that helps.

-Grace