Azure FunctionApp Syncing Triggers Fails During Deployment (BadRequest)

Question

I am reposting the comment I left here https://github.com/Azure/azure-functions-core-tools/issues/3522#issuecomment-1811608054

Yesterday (November 14th, 2023) I began experiencing the following issue during deployment to an Azure Function App. I am trying to deploy to a Python 3.10 function app in West US 2. I am deploying from cli with the command func azure functionapp publish GameIngestorPriority1-Dev. Deployment proceeds as normal until the "syncing triggers" step. At that point I see:

(quackers-jobs) C:\data\repos\game-ingestor [feature/monthly-game-watcher ≡]> func azure functionapp publish GameIngestorPriority1-Dev

...

Resetting all workers for gameingestorpriority1-dev.azurewebsites.net
Deployment successful. deployer = Push-Deployer deploymentPath = Functions App ZipDeploy. Extract zip. Remote build.
Remote build succeeded!
Syncing triggers...
Syncing triggers...
Syncing triggers...
Syncing triggers...
Syncing triggers...
Syncing triggers...
Error calling sync triggers (BadRequest). Request ID = '60d409bf-a14c-4cca-9914-0bd940e8bfde'.

I set $env:CLI_DEBUG=1 and this is the error the client is experiencing. This is the output when the error occurs:

Request:
Method: POST, RequestUri: 'https://management.azure.com/subscriptions/1d560c12-339c-4485-a73f-8bd95cd6f144/resourceGroups/ana_hockey_ops_staging/providers/Microsoft.Web/sites/GameIngestorPriority1-Dev/host/default/sync?api-version=2015-08-01', Version: 1.1, Content: System.Net.Http.StringContent, Headers:
{
  Authorization: Bearer {BEARER_TOKEN}
  User-Agent: functions-core-tools/4.0.5198
  Accept: application/json
  x-ms-request-id: 339d0001-a9e3-4ce2-bfc1-486134954117
  Content-Type: application/json; charset=utf-8
}
null

Response:
StatusCode: 400, ReasonPhrase: 'Bad Request', Version: 1.1, Content: System.Net.Http.HttpConnectionResponseContent, Headers:
{
  Cache-Control: no-cache
  Pragma: no-cache
  Strict-Transport-Security: max-age=31536000; includeSubDomains
  Server: Microsoft-IIS/10.0
  X-AspNet-Version: 4.0.30319
  X-Powered-By: ASP.NET
  x-ms-ratelimit-remaining-subscription-writes: 1199
  x-ms-request-id: 62293244-93ec-497b-8296-2e968baafe16
  x-ms-correlation-request-id: 62293244-93ec-497b-8296-2e968baafe16
  x-ms-routing-request-id: CANADACENTRAL:20231115T000210Z:62293244-93ec-497b-8296-2e968baafe16
  X-Content-Type-Options: nosniff
  Date: Wed, 15 Nov 2023 00:02:10 GMT
  Content-Length: 346
  Content-Type: application/json; charset=utf-8
  Expires: -1
}
{"Code":"BadRequest","Message":"Encountered an error (ServiceUnavailable) from host runtime.","Target":null,"Details":[{"Message":"Encountered an error (ServiceUnavailable) from host runtime."},{"Code":"BadRequest"},{"ErrorEntity":{"Code":"BadRequest","Message":"Encountered an error (ServiceUnavailable) from host runtime."}}],"Innererror":null}


Azure.Functions.Cli.Common.CliException: Error calling sync triggers (BadRequest). Request ID = '62293244-93ec-497b-8296-2e968baafe16'.
   at Azure.Functions.Cli.Helpers.RetryHelper.Retry(Func1 func, Int32 retryCount, TimeSpan retryDelay, Boolean displayError) in D:\a\_work\1\s\src\Azure.Functions.Cli\Helpers\RetryHelper.cs:line 27
   at Azure.Functions.Cli.Actions.AzureActions.PublishFunctionAppAction.SyncTriggers(Site functionApp) in D:\a\_work\1\s\src\Azure.Functions.Cli\Actions\AzureActions\PublishFunctionAppAction.cs:line 519
   at Azure.Functions.Cli.Actions.AzureActions.PublishFunctionAppAction.PublishFunctionApp(Site functionApp, GitIgnoreParser ignoreParser, IDictionary2 additionalAppSettings) in D:\a\_work\1\s\src\Azure.Functions.Cli\Actions\AzureActions\PublishFunctionAppAction.cs:line 503
   at Azure.Functions.Cli.Actions.AzureActions.PublishFunctionAppAction.RunAsync() in D:\a\_work\1\s\src\Azure.Functions.Cli\Actions\AzureActions\PublishFunctionAppAction.cs:line 209
   at Azure.Functions.Cli.ConsoleApp.RunAsync[T](String[] args, IContainer container) in D:\a\_work\1\s\src\Azure.Functions.Cli\ConsoleApp.cs:line 64

Further context I added on my comment on github:

My app has a single function that is triggered by Service Bus. I checked the queue this function listens to. It has 78 messages on it. So I checked the activity on the queue. The last time a message was processed by the function was 3:37PM EST on 2023-11-13. Then I checked the logs in Application Insights (not just the logs my code is writing, but all traces including the system ones). Selecting all traces reveals that the last message published to Application Insights (by any code) was also at 3:37 PM EST on 2023-11-13. Since then, no traces.

This leads me to believe the app is somehow dead? I tried stopping it and restarting it in the Portal. I didn't get any error messages for either action. But this had no effect. Any help would be greatly appreciated. This is occurring on the dev instance of this function app. The production instance is still operating normally. It would be soothing to know the cause, such that it won't affect my production environment.

At this point, the queue that triggers the only function in the function app is up to 434 messages (0 outgoing messages since November 13th). This morning (November 15th) I went on the Portal to my Function App, and then navigated GameIngestorPriority1-Dev -> Log Stream -> Open in Live Metrics. But once I click through to Live Metrics, I just see Not available: couldn't connect to your application. Additionally, I tried stopping and starting the application from the cli (yesterday I only tried from the portal). This had no effect.

Answer 1

Hello @Paul Piggott - As was discovered through the support case, the issue didn't originate from the configuration on your side, and the product team has resolved the underlying issue and your app is now restored. Thanks again for your patience and help in providing additional details while we investigated the root cause.

---

Please "Accept Answer" if the answer is helpful so that others in the community may benefit from your experience.