This browser is no longer supported.
Upgrade to Microsoft Edge to take advantage of the latest features, security updates, and technical support.
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(80, 45%, 17%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EXL%3C/text%3E%3C/svg%3E)
Hello,
We recently added System assigned Managed Identity to our Linux hosted App Service for containers.
This Managed Identity is used to connection to our Azure SQL database and Key Vault.
This works well, but every time a new container image is deployed (via the Portal or Webhook) the MSI TokenService fails for about a minute resulting in application failures.
Request for site my-webapp-for-containers failed secret validation. Expecting c2b90bxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx, received dd30d8xx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
Secret header validation failed for request on port 8081
Returning response for Site , Endpoint 172.16.0.3:8081, Port 8081, Path /msi/token, Method GET, Result = 403
As soon as the new container image is ready to serve requests, the token service is available again.
MSI logs:
2022-08-09T13:56:24.734748107Z mywebapp : [34185627-x] Incoming request on /msi/token?resource=https%3A%2F%2Fvault.azure.net&api-version=2019-08-01
2022-08-09T13:56:24.741092076Z mywebapp : [34185627-x] Request to TokenService: Endpoint 172.16.0.3:8081, Port 8081, Path /msi/token, Query ?resource=https%3A%2F%2Fvault.azure.net&api-version=2019-08-01, Method GET, UserAgent azsdk-js-identity/2.1.0 core-rest-pipeline/1.9.1 Node/v14.20.0 OS/(x64-Linux-5.4.0-1074-azure)
2022-08-09T13:56:24.751954596Z mywebapp : [34185627-x] Request to retrieve token received
2022-08-09T13:56:24.752790505Z mywebapp : [34185627-x] AadTokenCacheEntryRead. TenantId: xxxxx, ClientId: xxxxx, Resource: https://vault.azure.net, Expiration: 08/10/2022 13:45:13
2022-08-09T13:56:24.753402912Z mywebapp : [34185627-x] AAD Token Response Audit. Site mywebapp, ExpiresOn: 1660139113, Resource: https://vault.azure.net, TokenType: Bearer, ClientId: xxxxx
2022-08-09T13:56:24.753462612Z mywebapp : [34185627-x] Returning response for Site mywebapp, Endpoint 172.16.0.3:8081, Port 8081, Path /msi/token, Method GET, Result = 2002022-08-09T13:56:33.573391767Z mywebapp : [000000000] Configuration update started. Site: mywebapp
2022-08-09T13:56:33.583333083Z mywebapp : [000000000] [SystemAssigned] Added Identity with ClientId: xxxxx
2022-08-09T13:56:33.890462975Z mywebapp : [000000000] Configuration update started. Site: mywebapp
2022-08-09T13:56:33.891252892Z mywebapp : [000000000] [SystemAssigned] Added Identity with ClientId: xxxxx2022-08-09T13:56:40.443484502Z mywebapp : [b9b93d63-x] Incoming request on /msi/token?resource=https%3A%2F%2Fvault.azure.net&api-version=2019-08-01
2022-08-09T13:56:40.444627310Z mywebapp : [b9b93d63-x] Request to TokenService: Endpoint 172.16.0.3:8081, Port 8081, Path /msi/token, Query ?resource=https%3A%2F%2Fvault.azure.net&api-version=2019-08-01, Method GET, UserAgent azsdk-js-identity/2.1.0 core-rest-pipeline/1.9.1 Node/v14.20.0 OS/(x64-Linux-5.4.0-1074-azure)
2022-08-09T13:56:40.445751618Z mywebapp : [b9b93d63-x] Request for site mywebapp failed secret validation. Expecting c2b90bxx-xxxx-xxxx-xxxx-x, received dd30d8xx-xxxx-xxxx-xxxx-x
2022-08-09T13:56:40.472750010Z mywebapp : [b9b93d63-x] Secret header validation failed for request on port 8081
2022-08-09T13:56:40.649821473Z mywebapp : [b9b93d63-x] Returning response for Site , Endpoint 172.16.0.3:8081, Port 8081, Path /msi/token, Method GET, Result = 403
2022-08-09T13:57:19.811328657Z mywebapp : [5645b46d-x] Incoming request on /msi/token?resource=https%3A%2F%2Fvault.azure.net&api-version=2019-08-01
2022-08-09T13:57:19.812257967Z mywebapp : [5645b46d-x] Request to TokenService: Endpoint 172.16.0.3:8081, Port 8081, Path /msi/token, Query ?resource=https%3A%2F%2Fvault.azure.net&api-version=2019-08-01, Method GET, UserAgent azsdk-js-identity/2.1.0 core-rest-pipeline/1.9.1 Node/v14.20.0 OS/(x64-Linux-5.4.0-1074-azure)
2022-08-09T13:57:19.813176178Z mywebapp : [5645b46d-x] Request for site mywebapp failed secret validation. Expecting c2b90bxx-xxxx-xxxx-xxxx-x, received dd30d8xx-xxxx-xxxx-xxxx-x
2022-08-09T13:57:19.813196078Z mywebapp : [5645b46d-x] Secret header validation failed for request on port 8081
2022-08-09T13:57:19.813202278Z mywebapp : [5645b46d-x] Returning response for Site , Endpoint 172.16.0.3:8081, Port 8081, Path /msi/token, Method GET, Result = 4032022-08-09T13:56:33.130061275Z Warning! Cannot copy to bundle: /usr/local/share/ca-certificates/azure
2022-08-09T13:56:33.871796868Z WARNING: ca-certificates.crt does not contain exactly one certificate or CRL: skipping
2022-08-09T13:56:33.880435556Z WARNING: ca-cert-azure.pem does not contain exactly one certificate or CRL: skipping
2022-08-09T13:56:34.417600320Z Updated CA certificates
2022-08-09T13:56:47.943831735Z mywebapp : [000000000] Configuration update started. Site: mywebapp
2022-08-09T13:56:48.025757651Z mywebapp : [000000000] [SystemAssigned] Added Identity with ClientId: xxxxx
2022-08-09T13:56:48.046455356Z mywebapp : [000000000] Starting TokenService version 2.0.1.3 from /app with the LinuxDedicated SKU. Environment check: IsLinuxDedicated: True, IsLinuxConsumption: False, IsContainerApps: False, IsWindows: False.
2022-08-09T13:56:48.329133687Z Hosting environment: Production
2022-08-09T13:56:48.331515599Z Content root path: /app
2022-08-09T13:56:48.383934965Z Now listening on: http://[::]:8081
2022-08-09T13:56:48.383969865Z Application started. Press Ctrl+C to shut down.
2022-08-09T13:56:49.459989205Z mywebapp : [c3cb26e3-x] Incoming request on /healthcheck?api-version=2021-08-01
2022-08-09T13:56:49.463075020Z mywebapp : [c3cb26e3-x] Request to TokenService: Endpoint 172.16.0.6:8081, Port 8081, Path /healthcheck, Query ?api-version=2021-08-01, Method GET, UserAgent HealthCheck/1.0
2022-08-09T13:56:50.646766243Z mywebapp : [c3cb26e3-x] Returning response for Site , Endpoint 172.16.0.6:8081, Port 8081, Path /healthcheck, Method GET, Result = 200
2022-08-09T13:57:30.453751935Z mywebapp : [c13f55a5-x] Incoming request on /robots933456.txt
2022-08-09T13:57:30.499923900Z mywebapp : [c13f55a5-x] Request to TokenService: Endpoint 172.16.0.6:8081, Port 8081, Path /robots933456.txt, Query , Method GET, UserAgent HealthCheck/1.0
2022-08-09T13:57:30.500017001Z mywebapp : [c13f55a5-x] Returning response for Site , Endpoint 172.16.0.6:8081, Port 8081, Path /robots933456.txt, Method GET, Result = 404
2022-08-09T13:57:40.539461130Z mywebapp : [e4dc6085-x] Incoming request on /msi/token?resource=https%3A%2F%2Fvault.azure.net&api-version=2019-08-01
2022-08-09T13:57:40.540546737Z mywebapp : [e4dc6085-x] Request to TokenService: Endpoint 172.16.0.6:8081, Port 8081, Path /msi/token, Query ?resource=https%3A%2F%2Fvault.azure.net&api-version=2019-08-01, Method GET, UserAgent azsdk-js-identity/2.1.0 core-rest-pipeline/1.9.1 Node/v14.20.0 OS/(x64-Linux-5.4.0-1074-azure)
2022-08-09T13:57:40.760013925Z mywebapp : [e4dc6085-x] Request to retrieve token received
2022-08-09T13:57:42.936256562Z mywebapp : [e4dc6085-x] Requesting an AAD Token using ADAL for Authority: https://login.microsoftonline.com/xxxxx, Resource: https://vault.azure.net, ClientId: xxxxx
2022-08-09T13:57:43.754027812Z mywebapp : [e4dc6085-x] AadTokenCacheEntryAdded. TenantId: xxxxx, ClientId: xxxxx, Resource: https://vault.azure.net, Expiration: 08/10/2022 13:57:42
2022-08-09T13:57:43.764947481Z mywebapp : [e4dc6085-x] AAD Token Response Audit. Site mywebapp, ExpiresOn: 1660139862, Resource: https://vault.azure.net, TokenType: Bearer, ClientId: xxxxx
2022-08-09T13:57:43.932656137Z mywebapp : [e4dc6085-x] Returning response for Site mywebapp, Endpoint 172.16.0.6:8081, Port 8081, Path /msi/token, Method GET, Result = 200
2022-08-09T13:57:50.482848061Z mywebapp : [cd28e89e-x] Incoming request on /msi/token?resource=https%3A%2F%2Fdatabase.windows.net&api-version=2019-08-01
2022-08-09T13:57:50.536023393Z mywebapp : [cd28e89e-x] Request to TokenService: Endpoint 172.16.0.6:8081, Port 8081, Path /msi/token, Query ?resource=https%3A%2F%2Fdatabase.windows.net&api-version=2019-08-01, Method GET, UserAgent azsdk-js-identity/2.1.0 core-rest-pipeline/1.9.1 Node/v14.20.0 OS/(x64-Linux-5.4.0-1074-azure)
2022-08-09T13:57:50.536068993Z mywebapp : [cd28e89e-x] Request to retrieve token received
2022-08-09T13:57:50.536076493Z mywebapp : [cd28e89e-x] Requesting an AAD Token using ADAL for Authority: https://login.microsoftonline.com/xxxxx, Resource: https://database.windows.net, ClientId: xxxxx
2022-08-09T13:57:50.709470174Z mywebapp : [cd28e89e-x] AadTokenCacheEntryAdded. TenantId: xxxxx, ClientId: xxxxx, Resource: https://database.windows.net, Expiration: 08/10/2022 13:57:49
2022-08-09T13:57:50.726837882Z mywebapp : [cd28e89e-x] AAD Token Response Audit. Site mywebapp, ExpiresOn: 1660139869, Resource: https://database.windows.net, TokenType: Bearer, ClientId: xxxxx
2022-08-09T13:57:50.727656087Z mywebapp : [cd28e89e-x] Returning response for Site mywebapp, Endpoint 172.16.0.6:8081, Port 8081, Path /msi/token, Method GET, Result = 200
Docker logs:
2022-08-09T13:56:23.711Z INFO - Recycling container because of AppFrameworkVersionChange and appFrameworkVersion = xxxx.azurecr.io/mywebapp:main-v0.0.27
2022-08-09T13:56:50.921Z INFO - Pulling image: xxxx.azurecr.io/mywebapp:main-v0.0.27
2022-08-09T13:56:51.418Z INFO - main-v0.0.27 Pulling from mywebapp
.
.
.
2022-08-09T13:57:24.432Z INFO - xxxxxxxxxxx Pull complete
2022-08-09T13:57:24.681Z INFO - Digest: sha256:xxxxxxxxxxx
2022-08-09T13:57:24.912Z INFO - Status: Downloaded newer image for xxxx.azurecr.io/mywebapp:main-v0.0.27
2022-08-09T13:57:24.938Z INFO - Pull Image successful, Time taken: 0 Minutes and 33 Seconds
2022-08-09T13:57:25.259Z INFO - Starting container for site
2022-08-09T13:57:25.265Z INFO - docker run -d -p 6558:8080 --name mywebapp_1_f62859ee -e PORT=8080 -e WEBSITES_ENABLE_APP_SERVICE_STORAGE=false -e WEBSITE_SITE_NAME=mywebapp -e WEBSITE_AUTH_ENABLED=False -e WEBSITE_ROLE_INSTANCE_ID=0 -e WEBSITE_HOSTNAME=mywebapp.azurewebsites.net -e WEBSITE_INSTANCE_ID=xxxxxxxxxx -e HTTP_LOGGING_ENABLED=1 -e WEBSITE_USE_DIAGNOSTIC_SERVER=False xxxx.azurecr.io/mywebapp:main-v0.0.272022-08-09T13:57:30.383Z INFO - Initiating warmup request to container mywebapp_1_f62859ee_msiProxy for site mywebapp
2022-08-09T13:57:30.480Z INFO - Container mywebapp_1_f62859ee_msiProxy for site mywebapp initialized successfully and is ready to serve requests.
2022-08-09T13:57:30.482Z INFO - Initiating warmup request to container mywebapp_1_f62859ee for site mywebapp
2022-08-09T13:57:46.231Z INFO - Waiting for response to warmup request for container mywebapp_1_f62859ee. Elapsed time = 15.8582603 sec
2022-08-09T13:57:51.506Z INFO - Container mywebapp_1_f62859ee for site mywebapp initialized successfully and is ready to serve requests.
Any help is appreciated,
Kind regards
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(204.8, 2%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMB%3C/text%3E%3C/svg%3E)
We're seeing the same behaviour. We've been using managed identities with container app services for a while now and up until fairly recently everything has been working well. However, starting around the beginning of June we've started seeing frequent errors obtaining the msi token. Once everything has started up, everything works well, until the next time we deploy a new version of the container web app.
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(153.6, 92%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
XavierL-0557, Apologies you're facing this issue. I understand the issue occurs only when a new container image is deployed. While I check on this further internally --
Just to highlight, Managed identities use certificate-based authentication. Each managed identity’s credential has an expiration of 90 days and it's rolled after 45 days.
See, this doc. for more info.
If you haven’t done this already, try these steps:
1 . Navigate to the Access Policies Blade of KeyVault from the Azure Portal.
2. Click on "+ Add Access Policy"
3. Provide the necessary permissions. [For Key permission, method Get]
4. Choose the Service Principal (name of the App Service)
Kindly let us know how it goes, I’ll also follow-up privately to fetch additional info for a deeper investigation.
MarkBowler-7155, Just confirming, in your case what type of Identity Used (System Assigned \User Assigned )? I see that you started seeing this beginning of June/ frequent errors obtaining the msi token. Do you recall making any specific changes? what region have you provisioned the resources?
ajkuma-MSFT, The necessary permissions are correctly configured in the KeyVault Access Policy.
They same problem occurs when requesting a token for Azure SQL database.
The problem indeed only occurs when a new container image is deployed.
Hi @ajkuma ,
We are using user assigned managed identities and also are using the managed identity to connect to Azure SQL via JDBC.
We have not made any changes to the application related to the database connections.
To try to resolve the issue, we have just updated to the latest version of the JDBC driver (10.2.1) but unfortunately the issue still persists.
We are seeing the issue in multiple regions, UK West, West Europe and North Europe.
As per above the issue is only occurring when a new container image is deployed, or the application is restarted / scaled out, etc.
Regards,
Mark
XavierL-0557, MarkBowler-7155 , Thanks for the follow-up and sharing additional info. I'm checking on this further and will get back shortly.
To benefit the community members, posting our offline discussions.
Our product team has identified a bug and the fix has been deployed to all the regions.
We apologies for all the inconvenience with this. We appreciate your patience and cooperation!
Thanks for reporting this issue -XavierL-0557 and MarkBowler-7155.
Thanks for this. I can confirm that this issue is no longer occurring for us.
MarkBowler-7155, Glad to hear! thanks for the follow-up.