1. First just detect orphans-

1. Browse to https://provisioningapi.microsoftonline.com/provisioningwebservice.svc and do view certificates and install the chain of certificates from here
   1. Note: Need to ensure its entire chain is trusted
2. Open up mmc console and in certificates->trusted root authorities ensure all the 3 certificates are installed. Root cert, *.microsoftonline.com and also the intermediate certificate issued to Microsoft IT SSL SHA2. Most commonly the intermediate certificate certificate is missing

3. Onboarding script issue- Getting 401 Unauthorized on PreparePushtenant

Cause

Resolution

I was troubleshooting an issue where the Prepare Query suggestion timer job was taking a long time to execute and we wanted to output the history of the job run from across all servers in the farm and see how long the timer job took to execute.

Below command will give you that

$sug=get-sptimerjob -type Microsoft.Office.Server.Search.Administration.PrepareQuerySuggestionsJobDefinition

$sug.historyentries

Below sample of just one enty- In my env I saw entries, as far back as 10 days.

JobDefinitionId    : e4807402-1e7d-4e8c-96bc-5c4969ff3404
JobDefinitionTitle : Prepare query suggestions
WebApplicationId   : 00000000-0000-0000-0000-000000000000
WebApplicationName :
ServiceId          : c1a08ff9-cffe-4534-82ef-5d140a22b736
ServerId           : 092e41d7-6d0e-4ad0-b8ca-20b6ff91b97f
ServerName         : SP
Status             : Succeeded
StartTime          : 2/24/2016 11:32:53 PM
EndTime            : 2/24/2016 11:32:56 PM
DatabaseName       :
ErrorMessage       :

SensitiveType:"Credit Card Number" OR SensitiveType:"U.S. Social Security Number (SSN)" OR SensitiveType:"U.S. Driver's License Number"

Criteria you need to fulfill for the sensitive type documents to be shown in ediscovery queries.

SSN:

<Entity id="a44669fe-0d48-453d-a9b1-2cc83f2cba77" patternsProximity="300" recommendedConfidence="75">

      <Pattern confidenceLevel="85">

        <IdMatch idRef="Func_ssn" />

        <Any minMatches="1">

          <Match idRef="Keyword_ssn" />

          <Match idRef="Func_us_date" />

          <Match idRef="Func_us_address" />

        </Any>

      </Pattern>

      <Pattern confidenceLevel="75">

        <IdMatch idRef="Func_unformatted_ssn" />

        <Match idRef="Keyword_ssn" />

        <Any minMatches="1">

          <Match idRef="Func_us_date" />

          <Match idRef="Func_us_address" />

        </Any>

      </Pattern>

    </Entity>

 Eg1:   Formatted SSN

685-20-1398

and then enter either one of the following below in document

a. Keyword SSN

b.      Valid date in format month/year eg: 09/15

c.      Valid US address: it should have street, apt no, city, State and zip code

For this one the query would look something like:

SensitiveType="U.S. Social Security Number (SSN) | * | 85.."

Eg2:  Unformatted SSN

 561 20 1309 or 561201309

Put Keyword SSN

 and then enter either one of the following below in document

 a.      Valid date in format month/year eg: 09/15

b.      Valid US address: it should have street, apt no, city, State and zip code.

Credit Card:

<Entity id="50842eb7-edc8-4019-85dd-5a5c1f2bb085"
       patternsProximity="300" recommendedConfidence="85">
      <Pattern confidenceLevel="85">
       <IdMatch idRef="Func_credit_card" />
        <Any minMatches="1">
          <Match idRef="Keyword_cc_verification" />
          <Match idRef="Keyword_cc_name" />
          <Match idRef="Func_expiration_date" />
        </Any>
      </Pattern>
    </Entity>

Eg:1

Note: If you put a fake 16 digit number of credit card the this is not a valid test. We have an algorithm for verifying whether a number is a valid credit card number.  This is why you cannot test with random or fake 16 digit numbers—you must use a properly formatted credit card number.

**** **** **** ****

CC verification number: 273

Expiration date: 04/30/16

Drivers License:

</Pattern>

      <Pattern confidenceLevel="75">

        <IdMatch idRef="Func_washington_drivers_license_number" />

        <Match idRef="Keyword_washington_drivers_license_name" />

        <Match idRef="Keyword_us_drivers_license" />

      </Pattern>

      <Pattern confidenceLevel="65">

        <IdMatch idRef="Func_washington_drivers_license_number" />

        <Match idRef="Keyword_washington_drivers_license_name" />

        <Match idRef="Keyword_us_drivers_license_abbreviations" />

        <Any minMatches="0" maxMatches="0">

          <Match idRef="Keyword_us_drivers_license" />

        </Any>

      </Pattern>

Eg:1 (* indicates the actual Driver License number)

************

Washington Drivers license number (Replace with your state name)

 Eg:2

************

State: WA (Replace with your state name or initials)

Solution:

In order to see “Most popular Items” option for list enable the Cross Site publishing feature by going to Site settings-> Site collection features. Once you enable this you will be able to see the option

Note: this happens for fie which have their metadata "title" with no value. i.e Within the file if we go to file->prepare->Properties. Keep the title value empty.

 Two workaround that I found are:

1. Manually add value to the title field of the file and then run a crawl

2. Put basic:10 at top of mapping to MP “title” and then run full crawl

Permanent solution:

 The fix is now a  part of the July CU also mentioned in the KB:

https://support.microsoft.com/en-us/kb/3054926

When you crawl a file (.xlsx or .xlsm) that is created by Microsoft Excel 2007 or a later version of Excel on a file share, the file name appears in blank on the search result page.

The link for downloading the CU is https://support.microsoft.com/en-us/kb/3054937

On the Search farm (APC server) Sharepoint logs showed below:

03/11/2015 13:50:55.90        OWSTIMER.EXE (******:0x28E0)        0x44E4        SharePoint Server Search        Analytics        agmik        High        AnalyticsDataFilter: SPRequestUsageEntry filtered out because tenant settings could not be loaded.       

On the Content farm using Sharepoint timer Account we queried the SSA proxy for the analytics tenant settings and we got– Access Denied

$ssaproxy.getRawAnayticsTenantSettings([GUID]::Empty)  ( Where $ssaproxy=get-spenterprisesearchserviceapplicationproxy)

Exception calling "getRawAnayticsTenantSettings"with "1" argument(s):"Access is denied"

(Also note above that GetRawAnayticsTenantSettings is not a typo)

Looking at the logs:

04/17/2015 15:37:37.75              PowerShell.exe (0x2FEC)             0x35F0              SharePoint Server Search            Query   dka5      High              SearchServiceApplicationProxy::GetRawAnayticsTenantSettings--Error occured: System.ServiceModel.Security.SecurityAccessDeniedException: Access is denied.    Server stack trace:      at System.ServiceModel.Channels.ServiceChannel.ThrowIfFaultUnderstood(Message reply, MessageFault fault, String action, MessageVersion version, FaultConverter faultConverter)     at System.ServiceModel.Channels.ServiceChannel.HandleReply(ProxyOperationRuntime operation, ProxyRpc& rpc)     at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)     at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)     at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)    Exception rethrown at [0]:      at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)     at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)     at Microsoft.Office.Server.Search.Analytics.IAnalyticsServiceApplication.GetRawAnayticsTenantSettings(Guid tenantId)     at Microsoft.Office.Server.Search.Administration.SearchServiceApplicationProxy.<>c__DisplayClass2ea.<GetRawAnayticsTenantSettings>b__2e9(ISearchServiceApplication serviceApplication)     at Microsoft.Office.Server.Search.Administration.SearchServiceApplicationProxy.DoSpLoadBalancedUriWsOp[T](WebServiceBackedOperation`1 webServiceCall, Int32 timeoutInMilliseconds, Int32 wcfTimeoutInMilliseconds, String operationName)         01f12c40-f800-0000-5d5f-d685055fd001

So it looked like some sort of permissions were missing.  

Turns out Missing claimID of the consuming farm was the issue here

https://technet.microsoft.com/en-us/library/ff700211.aspx

Next ran below powershell to register FarmID as a claimType.

 Ran below in publishing farm:

 $security = Get-SPServiceApplication SSAGUID| Get-SPServiceApplicationSecurity

$claimprovider=(Get-SPClaimProvider System).ClaimProvider

$principal=New-SPClaimsPrincipal -ClaimType "https://schemas.microsoft.com/sharepoint/2009/08/claims/farmid" -ClaimProvider $claimprovider -ClaimValue "consumingfarmid"

Grant-SPObjectSecurity -Identity $security -Principal $principal -Rights "Full Control"

Set-SPServiceApplicationSecurity SSAGUID  -ObjectSecurity $security

Consumingfarmid is the GUID value of the consuming farm which you can get by running Get-SPFarm | Select Id

Run below command on content farm using the SP timer account: No access denied anymore.

$ssaproxy.getRawAnayticsTenantSettings([GUID]::Empty)  ( Where $ssaproxy=get-spenterprisesearchserviceapplicationproxy)

Waited for a day and next day the Usage reports showed us data.

SP logs showed:

Errorinfo is "Error caused by exception: Microsoft.BusinessData.Infrastructure.BdcException The shim execution failed unexpectedly - Exception has been thrown by the target of an invocation..: System.InvalidOperationException An internal server error occurred. Try again later."  [fltrsink.cxx:566]  search\native\mssdmn\fltrsink.cxx       

Now the reason that the BDC Exception is seen here is because SharePoint 2013 used the BCS Connector Framework when crawling external data such as Exchange/ Lotus etc

The above error happened because the crawler account was getting throttled on Exchange server. I worked with our Exchange engineer and below solution resolved our issue and we were able to crawl the Exchange content source successfully.

Here is event ID 9646 log from Exchange server:

Log Name:      Application
Source:        MSExchange
Date:          2/13/2015 10:36:29 AM
Event ID:      9646
Task Category: General
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      *******
Description:
Mapi session "8bcd2046-e644-4403-ae56-7be1c62e1e07: /o=J*****/ou=Exchange Administrative Group (******)/cn=Recipients/cn=SPSvcs service23a" exceeded the maximum of 250 objects of type "objtMessage".

Resolution Steps:

New-ThrottlingPolicy SharePoint  //this creates a new Throttling Policy called “SharePoint”;
 
• Type : Set-ThrottlingPolicy SharePoint -RCAMaxConcurrency $NULL -RCAPercentTimeInCAS $NULL -RCAPercentTimeInMailboxRPC $NULL -RCAPercentTimeInAD $NULL -EWSMaxConcurrency $NULL -EWSPercentTimeInAD $NULL -EWSPercentTimeinCAS $NULL -EWSPercentTimeInMailboxRPC $NULL -EWSMaxSubscriptions $NULL -EWSFastSearchTimeoutInSeconds $NULL -EWSFindCountLimit $NULL
 
• To confirm that all required settings have been set correctly please type: Get-ThrottlingPolicy SharePoint
 
• These items should now be blank:
 
CPAMaxConcurrency (Exchange 2010 SP1)
CPAPercentTimeInCAS (Exchange 2010 SP1)
CPAPercentTimeInMailboxRPC (Exchange 2010 SP1)
EWSMaxConcurrency
EWSPercentTimeInAD
EWSPercentTimeInCAS
EWSPercentTimeInMailboxRPC
EWSMaxSubscriptions
EWSFastSearchTimeoutInSeconds
EWSFindCountLimit
RCAMaxConcurrency
RCAPercentTimeInAD
RCAPercentTimeInCAS
RCAPercentTimeInMailboxRPC
 
To assign the Throttling Policy to the Default Access Account of SharePoint:
 
Set-Mailbox <Domain>\<UserName-DefaultContentAccessAccount> -ThrottlingPolicy SharePoint

 
Increase the number of allowed concurrent objects per session on the Exchange Server(s)

• Browse to the following registry key:   HKLM\System\CurrentControlSet\Services\MSExchangeIS\ParametersSystem\
•  Create a New Key called "MaxObjsPerMapiSession" ;
• Create a new Dword (32-bit) value objtFolder and set the value to 1000 ;
• Create a new Dword (32-bit) value objtMessage and set the value to 500;
• Restart the Exchange server after failing over the mailbox databases.

Note: I would recommend that you run these changes with your Exchange server Admin too to monitor any performance impact.