SCCM Cloud Management Gateway Deployment Failed after upgrade 2203->2207

Question

After upgrading from SCCM 2203 to 2207, the upgrade failed, we were getting an error, failed to update key vault, bad request.

Operation name
Update Key Vault
Time stamp
Mon Oct 31 2022 17:21:16 GMT-0500 (Central Daylight Time)
Event initiated by
ConfigManager_Server
Error code
BadRequest
Message
Invalid value found at accessPolicies[0].ObjectId:

so in an effort to work around the problem, we decided we could try to tear out the existing CMG and reimplement using all new names and to a new resource group, However, the new cmg fails to deploy with the same exact BadRequest error on the update key vault step.

cloudmgr.log

STATMSG: ID=9405 SEV=I LEV=M SOURCE="SMS Server" COMP="SMS_CLOUD_SERVICES_MANAGER" SYS=SYSCEN.myorganization.COM SITE=777 PID=3972 TID=8660 GMTDATE=Tue Nov 01 16:17:43.748 2022 ISTR0="myorgcmg" ISTR1="South Central US" ISTR2="" ISTR3="" ISTR4="" ISTR5="" ISTR6="" ISTR7="" ISTR8="" ISTR9="" NUMATTRS=1 LE=0X0 AID0=404 AVAL0="["Display=\myorgcmg.myorganization.com\"]MSWNET:["SMS_SITE=777"]\myorgcmg.myorganization.com\" SMS_CLOUD_SERVICES_MANAGER 11/1/2022 11:17:43 AM 8660 (0x21D4)
Resource Manager - Initializing... Acquiring access token to resource manager and accessing the subscription SMS_CLOUD_SERVICES_MANAGER 11/1/2022 11:17:43 AM 8660 (0x21D4)
Resource Manager - Initialized SMS_CLOUD_SERVICES_MANAGER 11/1/2022 11:17:44 AM 8660 (0x21D4)
Resource Manager - Creating resource group myorgcmg with location South Central US SMS_CLOUD_SERVICES_MANAGER 11/1/2022 11:17:44 AM 8660 (0x21D4)
Resource Manager - Resource group myorgcmg created SMS_CLOUD_SERVICES_MANAGER 11/1/2022 11:17:44 AM 8660 (0x21D4)
UpdateServiceInfo: Service 16777224 to ServiceState 1 ServiceInfoStateDetail 1002. SMS_CLOUD_SERVICES_MANAGER 11/1/2022 11:17:44 AM 8660 (0x21D4)
Acquiring access token to Microsoft graph endpoint ... SMS_CLOUD_SERVICES_MANAGER 11/1/2022 11:17:44 AM 8660 (0x21D4)
Start to poll cloud service tasks from DB SMS_CLOUD_SERVICES_MANAGER 11/1/2022 11:17:51 AM 15040 (0x3AC0)
TaskWorker: Starting task: [CloudServicesTaskBuilder] SMS_CLOUD_SERVICES_MANAGER 11/1/2022 11:17:51 AM 14584 (0x38F8)
TaskManager: 1 task(s) running, 0 task(s) waiting to start. SMS_CLOUD_SERVICES_MANAGER 11/1/2022 11:17:51 AM 10436 (0x28C4)
TaskManager: Task [CloudServicesTaskBuilder] status is Running SMS_CLOUD_SERVICES_MANAGER 11/1/2022 11:17:51 AM 10436 (0x28C4)
TaskManager: Task [CreateDeployment for service myorgcmg] status is Running SMS_CLOUD_SERVICES_MANAGER 11/1/2022 11:17:51 AM 10436 (0x28C4)
CloudServicesTaskBuilder: Starting. SMS_CLOUD_SERVICES_MANAGER 11/1/2022 11:17:51 AM 1956 (0x07A4)
CloudServicesTaskBuilder: Stopping. SMS_CLOUD_SERVICES_MANAGER 11/1/2022 11:17:51 AM 1956 (0x07A4)
TaskManager: 2 task(s) running, 0 task(s) waiting to start. SMS_CLOUD_SERVICES_MANAGER 11/1/2022 11:17:51 AM 10436 (0x28C4)
TaskManager: Task [CloudServicesTaskBuilder] status is RanToCompletion SMS_CLOUD_SERVICES_MANAGER 11/1/2022 11:17:51 AM 10436 (0x28C4)
TaskManager: Removing task [CloudServicesTaskBuilder] from running tasks. SMS_CLOUD_SERVICES_MANAGER 11/1/2022 11:17:51 AM 10436 (0x28C4)
TaskManager: Task [CreateDeployment for service myorgcmg] status is Running SMS_CLOUD_SERVICES_MANAGER 11/1/2022 11:17:51 AM 10436 (0x28C4)
Start to poll cloud service tasks from DB SMS_CLOUD_SERVICES_MANAGER 11/1/2022 11:18:51 AM 6620 (0x19DC)
TaskManager: 1 task(s) running, 0 task(s) waiting to start. SMS_CLOUD_SERVICES_MANAGER 11/1/2022 11:18:51 AM 10436 (0x28C4)
TaskManager: Task [CreateDeployment for service myorgcmg] status is Running SMS_CLOUD_SERVICES_MANAGER 11/1/2022 11:18:51 AM 10436 (0x28C4)
Current CMG deployment package hash is hirtSbcxrnei5mduwSaSj4fVSg2hYy1nlBG/QcD+Tt0= SMS_CLOUD_SERVICES_MANAGER 11/1/2022 11:18:51 AM 11624 (0x2D68)
Current VMSS CMG deployment package hash is pi7K4BUFcpL890hViW18NlIbqereDGZXRsAoi6Ryh74= SMS_CLOUD_SERVICES_MANAGER 11/1/2022 11:18:51 AM 11624 (0x2D68)
CreateUpgradeTaskForPackageChange: Triggered to create upgrade task for service type CloudProxyService with current package hash hirtSbcxrnei5mduwSaSj4fVSg2hYy1nlBG/QcD+Tt0=. SMS_CLOUD_SERVICES_MANAGER 11/1/2022 11:18:51 AM 11624 (0x2D68)
TaskWorker: Starting task: [CloudServicesTaskBuilder] SMS_CLOUD_SERVICES_MANAGER 11/1/2022 11:18:51 AM 14584 (0x38F8)
CloudServicesTaskBuilder: Starting. SMS_CLOUD_SERVICES_MANAGER 11/1/2022 11:18:51 AM 1956 (0x07A4)
TaskManager: 1 task(s) running, 0 task(s) waiting to start. SMS_CLOUD_SERVICES_MANAGER 11/1/2022 11:18:51 AM 10436 (0x28C4)
TaskManager: Task [CloudServicesTaskBuilder] status is Running SMS_CLOUD_SERVICES_MANAGER 11/1/2022 11:18:51 AM 10436 (0x28C4)
TaskManager: Task [CreateDeployment for service myorgcmg] status is Running SMS_CLOUD_SERVICES_MANAGER 11/1/2022 11:18:51 AM 10436 (0x28C4)
CloudServicesTaskBuilder: Stopping. SMS_CLOUD_SERVICES_MANAGER 11/1/2022 11:18:51 AM 1956 (0x07A4)
TaskManager: 2 task(s) running, 0 task(s) waiting to start. SMS_CLOUD_SERVICES_MANAGER 11/1/2022 11:18:51 AM 10436 (0x28C4)
TaskManager: Task [CloudServicesTaskBuilder] status is RanToCompletion SMS_CLOUD_SERVICES_MANAGER 11/1/2022 11:18:51 AM 10436 (0x28C4)
TaskManager: Removing task [CloudServicesTaskBuilder] from running tasks. SMS_CLOUD_SERVICES_MANAGER 11/1/2022 11:18:51 AM 10436 (0x28C4)
TaskManager: Task [CreateDeployment for service myorgcmg] status is Running SMS_CLOUD_SERVICES_MANAGER 11/1/2022 11:18:51 AM 10436 (0x28C4)
ERROR: Exception occurred when retrieving service principal Id of app 2023d6ca-020b-4731-bf1f-0304d5ee4de2. System.AggregateException: One or more errors occurred. ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.~~ --- End of inner exception stack trace ---~~ at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)~~ at Microsoft.ConfigurationManager.AzureManagement.ResourceManager.GetAppServicePrincipalId(String msftGraphEndpoint, String appId)~~---> (Inner Exception #0) System.Threading.Tasks.TaskCanceledException: A task was canceled.<--- SMS_CLOUD_SERVICES_MANAGER 11/1/2022 11:19:25 AM 8660 (0x21D4)
Resource Manager - Creating key vault myorgcmg with deployment CreateKeyVaultdccd4986-46fd-4acb-9e11-c82e17b2f781 SMS_CLOUD_SERVICES_MANAGER 11/1/2022 11:19:25 AM 8660 (0x21D4)
Resource Manager - Created deployment CreateKeyVaultdccd4986-46fd-4acb-9e11-c82e17b2f781 SMS_CLOUD_SERVICES_MANAGER 11/1/2022 11:19:25 AM 8660 (0x21D4)
Resource Manager - Waiting for deployment CreateKeyVaultdccd4986-46fd-4acb-9e11-c82e17b2f781 to finish. Will check again in 15 seconds. SMS_CLOUD_SERVICES_MANAGER 11/1/2022 11:19:25 AM 8660 (0x21D4)
ERROR: Resource Manager - Failed to finish deployment. Check [Monitor/Activity log] on Azure Portal for more information SMS_CLOUD_SERVICES_MANAGER 11/1/2022 11:19:40 AM 8660 (0x21D4)
Resource Manager - Getting deployment operation details for deployment CreateKeyVaultdccd4986-46fd-4acb-9e11-c82e17b2f781 in resource group myorgcmg SMS_CLOUD_SERVICES_MANAGER 11/1/2022 11:19:40 AM 8660 (0x21D4)
Resource Manager - Got deployment operation details for deployment CreateKeyVaultdccd4986-46fd-4acb-9e11-c82e17b2f781 in resource group myorgcmg SMS_CLOUD_SERVICES_MANAGER 11/1/2022 11:19:40 AM 8660 (0x21D4)
ERROR: Resource Manager - Deployment operation details: {"value":[{"id":"/subscriptions/0fcbf951-eb22-4250-9c53-5047743d4284/resourceGroups/myorgcmg/providers/Microsoft.Resources/deployments/CreateKeyVaultdccd4986-46fd-4acb-9e11-c82e17b2f781/operations/07DCD11DEB65B564","operationId":"07DCD11DEB65B564","properties":{"provisioningOperation":"Create","provisioningState":"Failed","timestamp":"2022-11-01T16:19:26.2245854Z","duration":"PT0.2902726S","trackingId":"c496492e-17ad-40e6-9a93-b699359ef3be","statusCode":"BadRequest","statusMessage":{"error":{"code":"BadRequest","message":"Invalid value found at accessPolicies[0].ObjectId: "}},"targetResource":{"id":"/subscriptions/0fcbf951-eb22-4250-9c53-5047743d4284/resourceGroups/myorgcmg/providers/Microsoft.KeyVault/vaults/myorgcmg","resourceType":"Microsoft.KeyVault/vaults","resourceName":"myorgcmg"}}}]} SMS_CLOUD_SERVICES_MANAGER 11/1/2022 11:19:40 AM 8660 (0x21D4)
Resource Manager - Deleting deployment CreateKeyVaultdccd4986-46fd-4acb-9e11-c82e17b2f781 SMS_CLOUD_SERVICES_MANAGER 11/1/2022 11:19:40 AM 8660 (0x21D4)
Resource Manager - Deployment CreateKeyVaultdccd4986-46fd-4acb-9e11-c82e17b2f781 deleted SMS_CLOUD_SERVICES_MANAGER 11/1/2022 11:19:41 AM 8660 (0x21D4)
ERROR: Exception occured for service myorgcmg : Hyak.Common.CloudException: Failed to finish deployment~~ at Microsoft.ConfigurationManager.AzureManagement.ResourceManager.StartAndMonitorDeployment(String resourceGroupName, String deploymentName, Deployment deploymentProp, Int32 secondsToWait, Int32 timeoutInMinutes)~~ at Microsoft.ConfigurationManager.AzureManagement.ResourceManager.CreateKeyVault(String resourceGroupName, String keyVaultName, String location, String vmssObjectId, Int32 timeoutInMinutes)~~ at Microsoft.ConfigurationManager.CloudServicesManager.CreateDeploymentTask.Start(Object taskState). SMS_CLOUD_SERVICES_MANAGER 11/1/2022 11:19:41 AM 8660 (0x21D4)
STATMSG: ID=9410 SEV=E LEV=M SOURCE="SMS Server" COMP="SMS_CLOUD_SERVICES_MANAGER" SYS=SYSCEN.myorganization.COM SITE=777 PID=3972 TID=8660 GMTDATE=Tue Nov 01 16:19:41.089 2022 ISTR0="myorgcmg" ISTR1="South Central US" ISTR2="" ISTR3="" ISTR4="" ISTR5="" ISTR6="" ISTR7="" ISTR8="" ISTR9="" NUMATTRS=1 LE=0X0 AID0=404 AVAL0="["Display=\myorgcmg.myorganization.com\"]MSWNET:["SMS_SITE=777"]\myorgcmg.myorganization.com\" SMS_CLOUD_SERVICES_MANAGER 11/1/2022 11:19:41 AM 8660 (0x21D4)
STATMSG: ID=9401 SEV=E LEV=M SOURCE="SMS Server" COMP="SMS_CLOUD_SERVICES_MANAGER" SYS=SYSCEN.myorganization.COM SITE=777 PID=3972 TID=8660 GMTDATE=Tue Nov 01 16:19:41.107 2022 ISTR0="CreateDeployment for service myorgcmg" ISTR1="Failed to finish deployment" ISTR2="" ISTR3="" ISTR4="" ISTR5="" ISTR6="" ISTR7="" ISTR8="" ISTR9="" NUMATTRS=1 LE=0X0 AID0=404 AVAL0="["Display=\myorgcmg.myorganization.com\"]MSWNET:["SMS_SITE=777"]\myorgcmg.myorganization.com\" SMS_CLOUD_SERVICES_MANAGER 11/1/2022 11:19:41 AM 8660 (0x21D4)
UpdateServiceInfo: Service 16777224 to ServiceState 2 ServiceInfoStateDetail 2000. SMS_CLOUD_SERVICES_MANAGER 11/1/2022 11:19:41 AM 8660 (0x21D4)
SetTaskState: Task 16779394 State Failed. SMS_CLOUD_SERVICES_MANAGER 11/1/2022 11:19:41 AM 8660 (0x21D4)
TaskManager: 1 task(s) running, 0 task(s) waiting to start. SMS_CLOUD_SERVICES_MANAGER 11/1/2022 11:19:42 AM 10436 (0x28C4)
TaskManager: Task [CreateDeployment for service myorgcmg] status is Faulted SMS_CLOUD_SERVICES_MANAGER 11/1/2022 11:19:42 AM 10436 (0x28C4)
TaskManager: Removing task [CreateDeployment for service myorgcmg] from running tasks. SMS_CLOUD_SERVICES_MANAGER 11/1/2022 11:19:42 AM 10436 (0x28C4)
ERROR: TaskManager: Task [CreateDeployment for service myorgcmg] has failed. Exception Hyak.Common.CloudException, Failed to finish deployment. SMS_CLOUD_SERVICES_MANAGER 11/1/2022 11:19:42 AM 10436 (0x28C4)
TaskManager: Task [CreateDeployment for service myorgcmg] has faulted and will not be retried. SMS_CLOUD_SERVICES_MANAGER 11/1/2022 11:19:42 AM 10436 (0x28C4)

Not sure what to do next.

Answer 1

Have you registered the keyvault resource provider?

Answer 2

Have you checked the Azure VM ops logs? Does the authenticating account have the necessary owner and contributor permissions to the subscription?

Answer 3

We deployed in april 2022 using virtual machine scale set. Yes, the all of the necessary bits are registered for virtual machine scale set.

Microsoft.Storage
Microsoft.Keyvault
Microsoft.compute
Microsoft.network

we even enable classiccompute on the off chance that this had some impact.

This has been working for 9 months until it stopped..

Answer 4

The authenticating account is mine and has the Owner Role on the subscription as well as global admin in azure.

Answer 5

Hi,

hope this issue is resolved, if not what you have to do is add a permission named Azure Key Vault for ServerApp and Grant admin permission for the directory for it.