Refiner Sort drop down specifics

.ms-srch-result #ResultHeader
{
display: none !important;
}

Horizontal Search Refinement Panel

• Upload the Group template "Control_Refinement_Horizontal.html" in the Display Templates / Filters directory
• Edit your Search Center result page
• Add a Refinement Web Part in the main search results zone (below the Search Box or Search Navigation)

• Choose the Refiners you want to include. I recommend to carefully select Refiners with a limited number of values or with a visual like Created or Modified date time.

• Save the page

SharePoint Add-ins

Azure Access Control Service (ACS)

SecurityTokenServiceHostNameSuffix : 
TypeName                           : Azure Access Control Service Application Proxy
DiscoveryConfiguration             : Microsoft.SharePoint.Administration.SPAzureAccessControlServiceDiscoveryConfiguration
MetadataEndpointUri                : https://accounts.accesscontrol.windows.net/e46a619f-8a46-4b91-a8c5-60904e04e135/metadata/json/1
ManageLink                         : 
PropertiesLink                     : 
CanUpgrade                         : True
IsBackwardsCompatible              : True
NeedsUpgradeIncludeChildren        : False
NeedsUpgrade                       : False
UpgradeContext                     : Microsoft.SharePoint.Upgrade.SPUpgradeContext
Name                               : ACS
DisplayName                        : ACS
Id                                 : 31089571-eb78-4d69-a7ff-ae8bc3345ed3
Status                             : Online
Parent                             : SPAzureAccessControlServiceProxy
Version                            : 4034005
Properties                         : {}
Farm                               : SPFarm Name=SP_Config
UpgradedPersistedProperties        : {}
CanSelectForBackup                 : True
DiskSizeRequired                   : 0
CanSelectForRestore                : True
CanRenameOnRestore                 : True

ACS STS - TrustedSecurityTokenIssuer

IsSelfIssuer                  : False
NameId                        : 
RegisteredIssuerName          : 00000001-0000-0000-c000-000000000000@e46a619f-8a46-4b91-a8c5-60904e04e135
IdentityClaimTypeInformation  : Microsoft.SharePoint.Administration.Claims.SPTrustedClaimTypeInformation
Description                   : 
SigningCertificate            : [Subject]
...
AdditionalSigningCertificates : {[Subject]
...
MetadataEndPoint              : https://accounts.accesscontrol.windows.net/e46a619f-8a46-4b91-a8c5-60904e04e135/metadata/json/1
IsAutomaticallyUpdated        : True
Name                          : ACS-STS
TypeName                      : Microsoft.SharePoint.Administration.Claims.SPTrustedSecurityTokenService
DisplayName                   : ACS-STS
Id                            : 6626e9a3-8f2a-4797-90a7-52858cb10655
Status                        : Online
Parent                        : SPSecurityTokenServiceManager Name=SecurityTokenServiceManager
Version                       : 4034018
Properties                    : {}
Farm                          : SPFarm Name=SP_Config
UpgradedPersistedProperties   : {}

SPO App Management Proxy

TypeName                    : SharePoint Online Application Principal Management Service Application Proxy
OnlineTenantUri             : https://msxxxxxx.sharepoint.com/
ManageLink                  : 
PropertiesLink              : 
CanUpgrade                  : True
IsBackwardsCompatible       : True
NeedsUpgradeIncludeChildren : False
NeedsUpgrade                : False
UpgradeContext              : Microsoft.SharePoint.Upgrade.SPUpgradeContext
Name                        : SPO App Management Proxy
DisplayName                 : SPO App Management Proxy
Id                          : d8f3ff14-6182-4c6b-a544-1597ff00e20a
Status                      : Online
Parent                      : SPOnlineApplicationPrincipalManagementServiceProxy
Version                     : 4034024
Properties                  : {}
Farm                        : SPFarm Name=SP_Config
UpgradedPersistedProperties : {}

PowerShell Script to remove those tenant-bound entities

# Remove ACS Proxy

# Remove the ACS Token Issuer

# Remove the SPO App Management

So how do I proceed ?

• Open LogParser Studio
• Create a new query.
• The below query is adding extract columns for analysis enrichment. It also excludes 401 requests by default.

select EXTRACT_FILENAME(LogFilename) as filename,LogRow,
date,time,s-ip,cs-method,cs-uri-stem,cs-uri-query,s-port,TO_UPPERCASE(cs-username) as cs-username,c-ip,cs(User-Agent) as UserAgent,cs(Referer) as Referer,sc-status,sc-substatus,sc-win32-status,time-taken,
to_int(to_string(to_timestamp(date,time),'yyyy')) as TYear,
to_int(to_string(to_timestamp(date,time),'MM')) as TMonth,
to_int(to_string(to_timestamp(date,time),'dd')) as TDay,
to_int(to_string(to_timestamp(date,time),'hh')) as THour,
to_int(to_string(to_timestamp(date,time),'mm')) as TMinutes,
to_int(to_string(to_timestamp(date,time),'ss')) as TSeconds,
to_lowercase(EXTRACT_FILENAME(cs-uri-stem)) as fname,
to_lowercase(EXTRACT_EXTENSION(cs-uri-stem)) as fext,
case div(time-taken, 1000) when 0 then '00-01' else case div(time-taken,3000) when 0 then '01-03' else case div(time-taken,5000) when 0 then '03-05' else case div(time-taken,10000) when 0 then '05-10' else case div(time-taken,20000) when 0 then '10-20' else case div(time-taken,30000) when 0 then '20-30' else '30+' end end end end end end as PerfBucket
FROM '[LOGFILEPATH]'
where sc-status<>401

• Select the input type as IISW3CLOG

• Now it is time to indicate to LogParser Studio where your IIS log files are located.

One very good thing of LogParser is its ability to query against multiple files as input and merge them all into one single output.

• Click on the Log File Manager and add all IIS logs you want to process.

• You're all set to execute your first IIS query!
• You would notice two extra columns : Filename and RowNumber. Those are LogParser specific, automatically generated upon processing. They will always appear when executing SELECT * queries. Filename can be used to maintain the origin of your IIS or differentiate IIS logs set.

• To output in a CSV or TSV, add the INTO clause BEFORE THE FROM clause

INTO '[OUTFILEPATH]\Output.TSV' 
FROM '[LOGFILEPATH]'
where sc-status<>401

• Output into a TSV or CSV will open Excel to visualize the output generated from your query.

1. Analyze SharePoint Products and Technologies Usage
2. Other blogs related to LogParser and IIS.

Merge-SPLogFile -Path "D:\temp\sitemastercrawlerror.log" -Correlation "3006F2D8-20F6-40F1-86A7-87FFC4C5238E".ToLowerInvariant()

$ssa = Get-SPEnterpriseSearchServiceApplication | Where-Object {$_.Name -eq "Search Service Application"}
$id = Get-SPEnterpriseSearchCrawlContentSource -SearchApplication $ssa | ? { $_.Name -eq "Intranet"}
$log = New-Object Microsoft.Office.Server.Search.Administration.CrawlLog $ssa
$dt = $log.GetCrawledUrls($false, 1000000, $null, $false, $id.Id, 2, -1, [System.DateTime]::MinValue, [System.DateTime]::MaxValue)
$dt | Export-CSV –Path “Crawllogs.csv” -NoTypeInformation

Search Performance

High Level Query Flow

Structure of a DK91 Event Message

• "Elapsed Time" : Round trip execution time at the Web Front End level (from the moment the query enters a WFE and returns).
• "IMSProxyTime" : Round trip execution time at the SQSS level (internally named IMS).
• "QP Time": Round trip execution time at the Query Component level. This includes the Index Components exec times.

• "WFEIMSLatency" : Latency between the Web Front End and SQSS.
• "IMSQPLatency": Latency between the SQSS and the Query Component.

Time for data extraction

• Collect ULS logs files of your SP Web Front Ends (WFE). We will only use those to extract Search Query Performance data.
• Create a new Query in LPS

/* WFE dk91 Extraction */
/* SearchServiceApplicationProxy::Execute--Id: Elapsed Time: 250 IMSProxyTime 229 QP Time: 214 Client Type CSOM TenantId 00000000-0000-0000-0000-000000000000    ab83b99d-f737-a0bf-1ff7-265cc0d362bc */

SELECT 
extract_token(EXTRACT_FILENAME(Filename),0,'-') as server,
TO_INT(extract_token(substr(message, ADD(INDEX_OF(message,'Elapsed Time: '),14)),0,' ')) as ElapsedTime,
TO_INT(extract_token(substr(message, ADD(INDEX_OF(message,'IMSProxyTime '),13)),0,' ')) as IMSProxyTime,
TO_INT(extract_token(substr(message, ADD(INDEX_OF(message,'QP Time: '),9)),0,' ')) as QPTime,
extract_token(substr(message, ADD(INDEX_OF(message,'Client Type '),12)),0,' ') as ClientType,
SUB(ElapsedTime,IMSProxyTime) as WFEIMSLatency,
SUB(IMSProxyTime,QPTime) as IMSQPLatency,
STRCAT(STRCAT(YearText,MonthText),DayText) as Date,
TRIM(SUBSTR(Timestamp,6,4)) AS Year, 
TRIM(SUBSTR(Timestamp,0,2)) AS Month, 
TRIM(SUBSTR(Timestamp,3,2)) AS Day,
TRIM(SUBSTR(Timestamp,11,8)) AS Time,
TRIM(SUBSTR(Timestamp,11,2)) AS Hour,
TRIM(SUBSTR(Timestamp,14,2)) AS Minute,
TRIM(SUBSTR(Timestamp,17,2)) AS Second,
Correlation 
USING 
STRCAT(SUBSTR(Timestamp,6,4), '-') as YearText,
STRCAT(SUBSTR(Timestamp,0,2), '-') as MonthText, 
STRCAT(SUBSTR(Timestamp,3,2), ' ') As DayText

INTO 'F:\Temp\mergedULS-DK91.tsv' 
FROM '[LOGFILEPATH]' 
where eventid like '%dk91%'

• This query assumes the ULS file are named following SP standard

SERVERNAME-DATE-TIME.log

• This query will produce one TSV as you see in the INTO clause.

• Associate the file extension .tsv with Excel for a quick review. The association will result in opening Excel whenever the query is executed and produced the .tsv output file.

Time for Analysis

• First convert your raw data into an Excel Table.
• Select the A1 cell and press CTRL+T command

• Once converted to a table, you can create charts, Pivot Table to highlight your Query performance.

Tip #1 : Visualize the response time better with Conditional Formatting.

• Select the entire ElapsedTime column, Click on Conditional Formatting, Choose Data Bars and select the esthetic and color of your choice.
• Repeat the same process for IMSProxyTime and QPTime with different colors.

Tip #2 : Create an histogram of Query Component response time distribution (QPTime)

• Select the QPTime column
• Click on the Insert ribbon
• Click on Recommended Charts
• Select the Histogram

• Press Enter, you will the Histogram
• Now we will modify the x-axis to create more meaningful performance buckets.
• On the histogram chart, right click on the X-Axis,
• In The Axis Options, set Bin width to 1000 (1 second bucket).

• Now set a Chart Title, change the design to your esthetic, set the Data Labels

• Download LogParser Studio

• Open LogParser Studio.
  • The tool comes with a fair list of predefined queries.
  • I would encourage you to review them (IIS especially)
  • All queries are organized in libraries.

• Create a new query

• Once selected, click on the Setting icon (next the TSVLOG type text).

• Create a .tsv file to store the ULS Header. Use the below content (tab separated)

  Timestamp    Process    TID    Area    Category    EventID    Level    Message    Correlation

• Save your file on disk.
• In the Settings iHeaderFile key put the path of the ULS Headers. In my case  D:\ULS-Headers.tsv.
• iHeaderFile is the only option we would modify at that point. All other settings should be maintained as is.
• Click OK

• You are almost there...

• Now it is time to indicate to LogParser Studio where your ULS log files are located.

One very good thing of LogParser is its ability to query against multiple files as input. In that sense, it could allow you to merge multiple ULS files into one.

• Click on the Log File Manager

• Add Files or folders containing your ULS logs. In my example I added two ULS log files. Click OK.
• After you're all set to execute our first ULS query in LPS !
• The default query shows "SELECT TOP 10 * FROM '[LOGFILEPATH]'". You can execute the query by hitting F5 or click on the red ! button.

• The Query results of your query in the upper panel as follows

• You would notice two extra columns : Filename and RowNumber. Those are LogParser specific, automatically generated upon processing. They will always appear when executing SELECT * queries. Filename can be used to maintain the origin of your ULS or differentiate ULS logs set.

• Save your query !

SELECT TOP 10 * 
FROM '[LOGFILEPATH]'
WHERE Message like '%Exception%'

SELECT * 
FROM '[LOGFILEPATH]'
WHERE EventID = 'b4ly'

SELECT *
INTO 'F:\Temp\mergedULS-test.tsv' 
FROM '[LOGFILEPATH]'
WHERE EventID = 'b4ly' and TID = '0x1764'

• By default (0), non modified folders are crawled during inc. crawl regardless of their modified datetime (legacy file shares mostly).
• Setting this to 1 would only crawl folders with a modified datetime superior to the last inc. crawl date. This provides stronger inc. crawl performance.

A software update or service pack was installed on servers in the farm. See the instructions for the software update or service pack for more information.