Error in update script for Sharepoint hybrid search

Question

Microsoft sent out an email saying that we have to run the script they provide, Update-FederatedHybridSearchForM365.ps1.

I'm running this on our dev farm to begin with, and I did have to run it a few times to get past some errors. However, it is stuck on this error:

EVO Successfully Registered as Trusted Token Issuer
Certificate was successfully retrieved.
WARNING: The service principal 'c3959f3a-5ad4-4d2b-b1f0-bc70f9a5d0a1' credentials cannot be created right now, waiting
for proper Service Principal creation. Trying again (attempt 1/12)
WARNING: The service principal 'c3959f3a-5ad4-4d2b-b1f0-bc70f9a5d0a1' credentials cannot be created right now, waiting
for proper Service Principal creation. Trying again (attempt 2/12)
WARNING: The service principal 'c3959f3a-5ad4-4d2b-b1f0-bc70f9a5d0a1' credentials cannot be created right now, waiting
for proper Service Principal creation. Trying again (attempt 3/12)
WARNING: The service principal 'c3959f3a-5ad4-4d2b-b1f0-bc70f9a5d0a1' credentials cannot be created right now, waiting
for proper Service Principal creation. Trying again (attempt 4/12)
WARNING: The service principal 'c3959f3a-5ad4-4d2b-b1f0-bc70f9a5d0a1' credentials cannot be created right now, waiting
for proper Service Principal creation. Trying again (attempt 5/12)
WARNING: The service principal 'c3959f3a-5ad4-4d2b-b1f0-bc70f9a5d0a1' credentials cannot be created right now, waiting
for proper Service Principal creation. Trying again (attempt 6/12)
WARNING: The service principal 'c3959f3a-5ad4-4d2b-b1f0-bc70f9a5d0a1' credentials cannot be created right now, waiting
for proper Service Principal creation. Trying again (attempt 7/12)
WARNING: The service principal 'c3959f3a-5ad4-4d2b-b1f0-bc70f9a5d0a1' credentials cannot be created right now, waiting
for proper Service Principal creation. Trying again (attempt 8/12)
WARNING: The service principal 'c3959f3a-5ad4-4d2b-b1f0-bc70f9a5d0a1' credentials cannot be created right now, waiting
for proper Service Principal creation. Trying again (attempt 9/12)
WARNING: The service principal 'c3959f3a-5ad4-4d2b-b1f0-bc70f9a5d0a1' credentials cannot be created right now, waiting
for proper Service Principal creation. Trying again (attempt 10/12)
WARNING: The service principal 'c3959f3a-5ad4-4d2b-b1f0-bc70f9a5d0a1' credentials cannot be created right now, waiting
for proper Service Principal creation. Trying again (attempt 11/12)
WARNING: The service principal 'c3959f3a-5ad4-4d2b-b1f0-bc70f9a5d0a1' credentials cannot be created right now, waiting
for proper Service Principal creation. Trying again (attempt 12/12)
Add-ServicePrincipalCredentials : An error occurred:
Microsoft.Online.Administration.Automation.MicrosoftOnlineException: Unable to complete this action. Try again later.
at Add-ServicePrincipalCredentials, E:\scripts\Update-FederatedHybridSearchForM365.ps1: line 226
at Set-S2SCertificateForSkill, E:\scripts\Update-FederatedHybridSearchForM365.ps1: line 212
at <ScriptBlock>, E:\scripts\Update-FederatedHybridSearchForM365.ps1: line 549
at <ScriptBlock>, <No file>: line 1
At E:\scripts\Update-FederatedHybridSearchForM365.ps1:212 char:9

• Add-ServicePrincipalCredentials $SkillAppId $StsCertB64 12
• ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
• CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
• FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Add-ServicePrincipalCredentials

It looks like something is not being created properly.
I ran this on the app server, I plan on running it on all on prem servers in our dev farm once it runs successfully.
It looks like an error in the script, which was provided by MS, so I'm hoping you can help fix it?

Answer 1

Hi @Raul Gomez Rodriguez ,

We tried running the script and encountered the following error. Any insight on what is causing the error would be helpful.

Successfully authenticated with AzureAD Module.
Successfully authenticated into SharePoint Online.
SharePoint On-Premises version is 2016
Creating Federated Search skill Service Principal in SPO.
WARNING: The service principal 'c3959f3a-5ad4-4d2b-b1f0-bc70f9a5d0a1' cannot be retrieved right now, waiting for proper
Service Principal creation. Trying again (attempt 1/12)
WARNING: The service principal 'c3959f3a-5ad4-4d2b-b1f0-bc70f9a5d0a1' cannot be retrieved right now, waiting for proper
Service Principal creation. Trying again (attempt 2/12)
WARNING: The service principal 'c3959f3a-5ad4-4d2b-b1f0-bc70f9a5d0a1' cannot be retrieved right now, waiting for proper
Service Principal creation. Trying again (attempt 3/12)
WARNING: The service principal 'c3959f3a-5ad4-4d2b-b1f0-bc70f9a5d0a1' cannot be retrieved right now, waiting for proper
Service Principal creation. Trying again (attempt 4/12)
WARNING: The service principal 'c3959f3a-5ad4-4d2b-b1f0-bc70f9a5d0a1' cannot be retrieved right now, waiting for proper
Service Principal creation. Trying again (attempt 5/12)
WARNING: The service principal 'c3959f3a-5ad4-4d2b-b1f0-bc70f9a5d0a1' cannot be retrieved right now, waiting for proper
Service Principal creation. Trying again (attempt 6/12)
WARNING: The service principal 'c3959f3a-5ad4-4d2b-b1f0-bc70f9a5d0a1' cannot be retrieved right now, waiting for proper
Service Principal creation. Trying again (attempt 7/12)
WARNING: The service principal 'c3959f3a-5ad4-4d2b-b1f0-bc70f9a5d0a1' cannot be retrieved right now, waiting for proper
Service Principal creation. Trying again (attempt 8/12)
WARNING: The service principal 'c3959f3a-5ad4-4d2b-b1f0-bc70f9a5d0a1' cannot be retrieved right now, waiting for proper
Service Principal creation. Trying again (attempt 9/12)
Trusted For Delegation (TFD) feature is successfully enabled for Service Principal.
Service Principal was successfully created in SPO.

ExtensionData : System.Runtime.Serialization.ExtensionDataObject
AccountEnabled : True
Addresses : {}
AppPrincipalId : c3959f3a-5ad4-4d2b-b1f0-bc70f9a5d0a1
DisplayName : Greenland Federated Search Bot Skill
ObjectId : 4da6695b-f0eb-446d-9d66-db8279b7bdcf
ServicePrincipalNames : {c3959f3a-5ad4-4d2b-b1f0-bc70f9a5d0a1, https://hybridsearchskill.cortana.ai}
TrustedForDelegation : True

Registering Application Principal in Sharepoint On-Premises.
Something went wrong when registering new application principal in Sharepoint On-Premises.
Register-AppPrincipalOnPrem : App Management Shared Service Proxy is not installed.
At D:\Update-FederatedHybridSearchForM365.ps1:626 char:27

• ... Principal = Register-AppPrincipalOnPrem -Site $Site -SkillNameIdentif ...
• ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
• CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
• FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Register-AppPrincipalOnPrem

Answer 2

Hi @S, Prasanna ,

Can you please check if the Hybrid Search Scenario you have implemented is Hybrid Federated Inbound Search rather than Cloud Hybrid Search? The main difference between the 2 of them is that Cloud Hybrid Search has only 1 index and your On-Premises documents are indexed within M365. On the other hand, with Federated Hybrid Search there are 2 separate indexes one for SharePoint Online and 1 for SharePoint On-Premises.

Please take a look at this article:
https://learn.microsoft.com/en-us/sharepoint/hybrid/hybrid-search-in-sharepoint

One way to tell if you're using Cloud Hybrid Search is if you're getting your On-Prem and SharePoint Online results all in one block like the image below:

Federated Hybrid Search will use 2 different results blocks for presenting results back like the image below:

If you are using Federated Hybrid Search then I believe that App Management Service is not enabled / installed in your SharePoint On-Premises installation, can you please do the following steps to confirm?

1. Open SharePoint Central Administration as Administrator.
2. Below “Application Management”, click on “Manage Web Applications”.
   
3. Select your web application, and from the above ribbon, click on “Service Connections”.
   If the App Management Service is not checked, check it.

Let me know it the App Management Service actually shows up, in case it doesn't you should be able to install it by following these steps:

1 Open Central Administration as administrator > Application Management > Manage Service Application.

2. From the ribbon Click on New > App Management Service.
   
3. The below dialog will be shown > Type the name of the service > Modify the DBName as you need.
   
4. Type the name of the application pool > Select the Managed account or create a new one.
   
5. Click OK to create an app management service.
   
6. The app management service should be now created.
   

Start App Management Service Application Proxy

As above shown, the app management service application proxy is stopped, and to start it, you should do the following:

1. Navigating to System Settings > Manage Services on the server.
   
2. Find App management service > Below Action> Click on start to start the service.
   
3. The App Managment Service should be now started.
   
4. Go back to manage service applications.
   
5. The App Management Service Application Proxy should now be started.
   
6. Go to Site Settings > Below Site Actions > Select Manage site features.
   
7. Activate "Workflows can use app permissions" feature
   

After all of these steps, please try running the script one more time. Let me know if any issues appear this time.

Steps above taken from: https://spgeeks.devoworx.com/app-management-shared-service-proxy-is-not-installed/