You can integrate SharePoint with Windows Azure Active Directory (https://technet.microsoft.com/en-us/library/dn635311(v=office.15).aspx) to grant access to users from AAD to SharePoint 2013 sites.

Now what happens if you want to grant access to the AAD users using AAD groups? …the security groups will not be resolved in SharePoint. In fact, in the people picker control, you get results for any search string entered, but SharePoint does not validate the user input. Any input is validated as is against each claim type configured (whatever the user types). This is not specific to AAD, or AAD groups.

AAD group:

People Picker Control:

Why can we not grant permissions in SharePoint 2013 sites to groups from Azure Active Directory (AAD)?

There is a good reason behind it. When a user authenticates, AAD does not include his group membership in the SAML token received by SharePoint, so SharePoint does not know to which groups the user belongs to, thus cannot make authorizations based on groups.

Ok ok.. so now that we know the reason behind it,

How do we make it work?

We have to implement a custom claims provider ( https://msdn.microsoft.com/en-us/library/office/ee535894(v=office.15).aspx ) that will get the AAD groups of the user and add them to his SAML token, which will allow SharePoint to process authorization on groups.

Another benefit of using a custom claims provider is that it adds search capabilities to the people picker, so when a user uses people picker to grant a permission, he gets actual users and groups from AAD.

Project https://azurecp.codeplex.com implements what I describe above. Credits to the developer of the project who supported me also in writing this blog article! 

Summary:

1. Create a new user in Active Directory.

2. Create a new Web Application in Central Administration that uses forms-based authentication.

3. Configure the web.config files.

4. Create a new site collection for the new created Web Application and add the FBA user as site collection administrator.

5. Browse to the new created site and test the functionality.

1. Create a new user in Active Directory

I named the user ‘fbaadmin’ and placed it in an organization unit called FBA’.

2. Create a new Web Application in Central Administration that uses forms-based authentication.

a. Browse to Central Administration > Application Management > Manage Web Applications and click ‘New’.

b. Type in a port number (e.g. 1100) and check the boxes next to “Enabled Windows Authentication | Integrated Windows authentication | NTLM” and “Enabled Forms Based Authentication (FBA)”.

c. Type in a name for the “ASP.NET Membership provider name” (I named it ‘membership’) and for the “ASP.NET Role manager name” (I named it ‘rolemanager’).

d. Click OK to create the Web Application. 

3. Configure the web.config files for:

Ø  Central Administration

Ø  Security Token Service

Ø  The new Web Application created

a. To configure the Central Administration web.config file

i. Open IIS Manager via Start > Run > inetmgr.

ii. In the console tree, open the server name, and then Sites.

iii. Right-click the SharePoint Central Administration v4 site, and then click Explore.

iv. In the folder window, double-click the web.config file.

v.  In the <Configuration> section, find the <system.web> section and add the following example entry 

    <membership defaultProvider="AspNetSqlMembershipProvider">
      <providers>
        <add name="membership"
             type="Microsoft.Office.Server.Security.LdapMembershipProvider, Microsoft.Office.Server, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c"
             server="contosodc.contoso.com"
             port="389"
             useSSL="false"
             userDNAttribute="distinguishedName"
             userNameAttribute="sAMAccountName"
             userContainer="OU=FBA,DC=contoso,DC=com"
             userObjectClass="person"
             userFilter="(ObjectClass=person)"
             scope="Subtree"
             otherRequiredUserAttributes="sn,givenname,cn" />
      </providers>
    </membership>
    <roleManager enabled="true" defaultProvider="AspNetWindowsTokenRoleProvider" >
      <providers>
        <add name="roleManager"
             type="Microsoft.Office.Server.Security.LdapRoleProvider, Microsoft.Office.Server, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c"
             server="contosodc.contoso.com"
             port="389"
             useSSL="false"
             groupContainer="DC=contoso,DC=com"
             groupNameAttribute="cn"
             groupNameAlternateSearchAttribute="samAccountName"
             groupMemberAttribute="member"
             userNameAttribute="sAMAccountName"
             dnAttribute="distinguishedName"
             groupFilter="((ObjectClass=group)"
             userFilter="((ObjectClass=person)"
             scope="Subtree" />
      </providers>
    </roleManager>

Note: you have to substitute the yellow highlighted parts with the information specific to your environment, as follows:

·        For server you need the FQDN of your domain controller (in my case it is ‘contosodc.contoso.com’)

·        For userContainer you need the distinguished name of your user container – you can find it on the Domain Controller > Active Directory Users and Computers > ‘your domain’ > right click on the corresponding OU and switch to the ‘Attribute Editor’ tab:

·        The name of your role manager in <add name="roleManager".

·        For groupContainer you need the distinguished name of your group container – you can find it on the Domain Controller > Active Directory Users and Computers > right click on ‘your domain’  and switch to the ‘Attribute Editor’ tab:

b. To configure the Security Token Service web.config file

i. In the console tree of Internet Information Services (IIS) Manager, open the SharePoint Web Services site.

ii. In the console tree, right-click SecurityTokenServiceApplication, and then click Explore.

iii. In the folder window, double-click the web.config file.

iv. In the <Configuration> section, create a new <system.web> section and add the following example entry:

    <membership>
      <providers>
        <add name="membership"
             type="Microsoft.Office.Server.Security.LdapMembershipProvider, Microsoft.Office.Server, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c"
             server="contosodc.contoso.com"
             port="389"
             useSSL="false"
             userDNAttribute="distinguishedName"
             userNameAttribute="sAMAccountName"
             userContainer="OU=FBA,DC=contoso,DC=com"
             userObjectClass="person"
             userFilter="(&amp;(ObjectClass=person))"
             scope="Subtree"
             otherRequiredUserAttributes="sn,givenname,cn" />
      </providers>
    </membership>
    <roleManager enabled="true" >
      <providers>
        <add name="rolemanager"
             type="Microsoft.Office.Server.Security.LdapRoleProvider, Microsoft.Office.Server, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c"
             server="contosodc.contoso.com"
             port="389"
             useSSL="false"
             groupContainer="DC=contoso,DC=com"
             groupNameAttribute="cn"
             groupNameAlternateSearchAttribute="samAccountName"
             groupMemberAttribute="member"
             userNameAttribute="sAMAccountName"
             dnAttribute="distinguishedName"
             groupFilter="(&amp;(ObjectClass=group))"
             userFilter="(&amp;(ObjectClass=person))"
             scope="Subtree" />
      </providers>
    </roleManager>

Note: you have to substitute the yellow highlighted parts with the information specific to your environment, as described above at point 3.a.v.

c. To configure the new web application’s web.config file

i. In the console tree of Internet Information Services (IIS) Manager, right-click the site that corresponds to the name of the web applications that you just created, and then click Explore.

ii. In the folder window, double-click the web.config file.

iii. In the <Configuration> section, find the <system.web> section.

iv. Find the <membership defaultProvider="i"> section and add the following example entry to the <Providers> section: 

    <add name="membership"
        type="Microsoft.Office.Server.Security.LdapMembershipProvider, Microsoft.Office.Server, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c"
        server="contosodc.contoso.com"
        port="389"
        useSSL="false"
        userDNAttribute="distinguishedName"
        userNameAttribute="sAMAccountName"
        userContainer="OU=FBA,DC=contoso,DC=com"
        userObjectClass="person"
        userFilter="(&amp;(ObjectClass=person))"
        scope="Subtree"
        otherRequiredUserAttributes="sn,givenname,cn" />

Note: you have to substitute the yellow highlighted parts with the information specific to your environment, as described above at point 3.a.v. 

v. Find the <roleManager defaultProvider="c" enabled="true" cacheRolesInCookie="false"> section and add the following example entry to the <Providers> section: 

    <add name="roleManager"
        type="Microsoft.Office.Server.Security.LdapRoleProvider, Microsoft.Office.Server, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c"
        server="contosodc.contoso.com"
        port="389"
        useSSL="false"
        groupContainer="DC=contoso,DC=com"
        groupNameAttribute="cn"
        groupNameAlternateSearchAttribute="samAccountName"
        groupMemberAttribute="member"
        userNameAttribute="sAMAccountName"
        dnAttribute="distinguishedName"
        groupFilter="(&amp;(ObjectClass=group))"
        userFilter="(&amp;(ObjectClass=person))"
        scope="Subtree" />

Note: you have to substitute the yellow highlighted parts with the information specific to your environment, as described above at point 3.a.v.

After you have done all changes in the web.config files, save and close the files.

4. Create a new site collection for the new created Web Application and add the FBA user as site collection administrator

a. Browse to Central Administration > Application Management > Create site collection.

b. Choose the Web Application which you have created at step 2.

c. Add a Title for the site, e.g. rootsite.

d. In the Primary Site Collection Administrator click on the  symbol and search for the user name created at step 1.

Select the user found under “Forms Auth” and click OK. 

The settings for the new site collection will look like this (I added also a secondary site collection administrator): 

5. Browse to the new created site and test the functionality.

If you have selected both Windows NTLM and FBA authentication, you will get a dropdown menu where you can choose between the 2 authentication mechanisms:

SharePoint 2013 was no exception, although we kept the same mechanisms for gathering information like we had in 2010 (SPUsageProviders / SPUsageDefinition / SPUsageReceivers) the processing of the data gathered was moved towards the Usage Analytics Component of the Search Service Application.

Given that in the last couple of months we had an increasing number of cases dealing with Usage Analytics I wanted to share with you a broad overview of how we get collect and process the data for the usage analytics reports.

Data collection

In SharePoint 2010 we introduced the concept of SPUsageProviders and SPUsageDefinitions. I will not go into many details, but you can see this as being the definition of a data set that can be programmatically logged to Usage Database (WSS_Usage) from within your SharePoint code.

You can see all the definitions currently installed in your farm by running:

 Get-SPUsageDefinition

Each definition can have the following properties (and many more):

 - Columns - will define  the columns in the data set that we log.
 - Table - the name of the partitions you will see in the WSS_Usage database.
 - RetentionPeriod - the time we keep the data in the WSS_Usage database.
 - UsageReceivers - will discuss about this a bit later

Each time you add a new usage definition we will create a set of 32 partitions (with the name specified in the table property of the definition) in the WSS_Usage database (also a view with the name of the definition, that will basically cumulate the data from all the partitions).

Programmatically in your custom code (or in the MS code) we can create new entries for each definitions and add them to an internal store (first 1000 entries in memory, then they get moved to .usage files on the disk).

Every 5 minutes a timer job (Microsoft SharePoint Foundation Usage Data Import) will get all the definitions and call the ImportEntries method of each definition. If this method was not overwritten it will move the data from the in-memory store or .usage files to the corresponding tables in the WSS_Usage database.

When this is done, it will execute the SPUsageReceivers that are attached to the usage definitions. You can look at the usage receivers as being an event handler triggered after each import of the usage data. You can develop your own receivers with the purpose of running tasks like data analysis, data archival… etc.

For the OOB usage analytics we use two different usage definitions: Page Requests and Usage Analytics. For Page Requests we will save all the logged data in the WSS_Usage  database, for the Usage Analytics definition the ImportEntries method was overwritten so we don’t log anything to the WSS_Usage database.

When we provision the first SearchService application we will automatically also create an Analytics Component. During the provision process of the SSA we will also add a usage receiver to both the Page Requests and the Usage Analytics definitions. These receivers have the role of "transporting" the data from the usage definition store to the analytics component of the SSA. This happens by means of a web service method  called SendAnalyticsUsageEvents. The web service will receive all the usage entries logged and write them to the EventStore (default location on the analytics component: C:\Program Files\Microsoft Office Servers\15.0\Data\Office Server\Analytics_GUID\EventStore (where the GUID is the ApplicationName property of the SSA).

One of the more common problems we noticed is that the usage receivers are no longer registered to the usage definitions. You can check for this by using:

 $ad = Get-SPUsageDefinition | where {$_.Name -like "Analytics*"}
 $pd = Get-SPUsageDefinition | where {$_.Name -like "Page Requests"}
 $ad.Receivers
 $pd.Receivers

If the receivers are not displayed you can register them by using:

 $ad = Get-SPUsageDefinition | where {$_.Name -like"Analytics*"}
 $ad.Receivers.Add("Microsoft.Office.Server.Search.Applications, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c","Microsoft.Office.Server.Search.Analytics.Internal.AnalyticsCustomRequestUsageReceiver")
 $ad.EnableReceivers = $true

 $pd = Get-SPUsageDefinition | where {$_.Name -like"Page Requests"}
 $pd.Receivers.Add("Microsoft.Office.Server.Search.Applications, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c", "Microsoft.Office.Server.Search.Analytics.Internal.ViewRequestUsageReceiver")
 $pd.EnableReceivers = $true 

This pretty much ends the data gathering. The short version of this process being: we use the Page Requests and Usage Analytics usage definitions to gather the data, and attached usage receivers to send the data to the SSA by means of the SendAnalyticsUsageEvents web service method, that will store the data in the event store to await processing.

Data Processing

The processing is done by a component called the analytics engine. This will run once every day and read the data from the Event Store and then perform a number of computations on this data to compile the needed reports. The whole process is opaque and really complex, besides reading the event store, we also read data from the search analytics database and from the search index. The computation themselves can be fairly resource intensive and take a long time to finish. We recommend having a good amount of free space on the drive where the store is located, as during the processing we will create some temporary files that can reach large dimensions.

There are not many ways of interacting with the analysis engine. I would recommend using the following:

 $tj = Get-SPTimerJob -Type Microsoft.Office.Server.Search.Analytics.UsageAnalyticsJobDefinition
 $tj.GetAnalysisConfiguration()
 $tj.GetAnalysisInfo()
 

This will give some basic informations, as to what the running schedule is, successful runs, last errors… You can also start an off-schedule run of the analysis engine:

 $tj = Get-SPTimerJob -Type Microsoft.Office.Server.Search.Analytics.UsageAnalyticsJobDefinition
 $tj.DisableTimerjobSchedule()
 $tj.StartAnalysis()
 

And then don’t forget to do the:  

$tj.EnableTimerjobSchedule()

There is not much troubleshooting that can be done at Analytics Engine level. If the correct data is in the event store, the reports should reflect this.

After the analysis engine finishes processing the data it will write the results to the Search_Service_Application_AnalyticsReportingStoreDB database, AnalyticsItemData.

Final notes

The whole platform behind the usage analytics reports is expandable. The out of the box code, is not meant to cover all usage reporting scenarios. For more special reporting needs I recommend developing a custom solution that includes SPUsageReceivers and event custom usage definitions.

Also, I would like to extend again the warning, that changing settings of the different analyses will cause the engine to fail. Do not change settings you don't understand.

Let's define 2 Site Columns: "Zuständig" and "Fachlich zuständig" of Type "Person or Group" on a SharePoint Site,

o   “Zustaendig” -- Type "Person or Group" (New Group -- "CustomSearch") , Enforce unique values- No, Allow multiple selections - No, Choose from : All Users

o   “FachlichZustaendig” -- Type "Person or Group" (Group -- "CustomSearch") ,Enforce unique values- No, Allow multiple selections - Yes, Choose from : All Users

add them on a Document Library and use them with adding several items.

Then perform a Full Crawl. Once the Crawl is run, the Crawled Properties for these 2 entries are visible in the Search Administration Page under "Crawled Properties" as "ows_Zustaendig(Text)" and "ows_Fachlich_zustaeding(Text)". Do not worry about the (Text) Type, it's how the System handles the Columns as Crawled Properties.

The Add of Managed Properties is afterwards required: [SearchAdministration | Metadata Properties | New Managed Prop.]

o   "Zustaendig"  is mapped to "ows_Zustaendig(Text)" , has single value, Includes values from all crawled properties mapped,  is being used in Scopes and has following option enabled : "Add managed property to custom results set retrieved on each query. Note: Only the first 2 kilobytes of data is available for display by default"

o   "FachlichZustaendig" is mapped to "ows_Fachlichzustaeding(Text)"  , has multiple values , Includes values from all crawled properties mapped, is being used in Scopes  and has following option enabled : "Add managed property to custom results set retrieved on each query. Note: Only the first 2 kilobytes of data is available for display by default"

On the Search page, perform a search according to the Items added in the Document Library. Once results are displayed, the Search Center Site needs to be in Edit mode so that the Refinement Panel can be edited [Filter CategoryDefinition]. You will need to add following section:

<Category LessLinkText="show fewer" MoreLinkText="show more" MappedProperty="zustaendig" ShowMoreLink="True" SortDirectionForMoreFilters="Ascending" SortByForMoreFilters="Name" SortDirection="Descending" SortBy="Frequency" MaxNumberOfFilters="20" NumberOfFiltersToDisplay="4" MetadataThreshold="5"  Type="Microsoft.Office.Server.Search.WebControls.ManagedPropertyFilterGenerator" Description="Use this filter to restrict results.by a specific zust."  Title="Zustaendig"/>

Be aware that the Lowercase restriction on the MappedProperty should not be there any more, as this limitation was removed with SP1.

If you encounter issues with the functionality though, it's worth trying it out on lowercase also. 

The "Use Default Configuration" needs to be unchecked. Then once Saving and Exiting the Edit Mode, the newly added property should be available for refinement :

If this still does not show-up on page, make sure that you have enough items to display (>5); as per:  https://msdn.microsoft.com/en-us/library/ff625183.aspx  Category [Refinement] 

In our settings, we used MetadataThreshold="5" , but the value can be accordingly changed, as per own choice.

But what about the Accuracy Index?

This Value is being directly set-up once for the entire Refinement Panel Web-Part and it stands for how many items are considered in the creation of refiners. By default this is set to 50 so that only the first 50 items in your search results will be used to generate refiners.

As per Technet, this has no upper maximum: https://technet.microsoft.com/en-us/library/gg549985.aspx  “Change what appears in the Refinement Panel Web Part (SharePoint Server 2010)”

Next, what if you want to add such Property as Refiner in "Advanced Search"?

For this action, the Search Center Site has to be again in Edit Mode, this time the Advanced Search box will be edited:  [Properties] and add the following:   

<PropertyDefs>     
<PropertyDef Name="Path" DataType="text"  DisplayName="URL"/>   
.     
.    

<PropertyDef  Name="FachlichZustaendig" DataType="text" DisplayName="FachlichZustaendig"/>

</PropertyDefs>

and in the  <AllResults>  Section   

<ResultType DisplayName="All Results" Name="default">     
<KeywordQuery/>     
<PropertyRef Name="Author" /> 
.     
.

<PropertyRef Name="FachlichZustaendig" />   
</ResultType>

As such, it will get in the refiners list for the Advanced Search.

For any of the above handling of the managed properties, the special "Person or Group" Type  or single-, multi-value does not determine any special setting or limitation in functionality.

References:

https://msdn.microsoft.com/en-us/library/ff625183.aspx :: "Category [Refinement]"

https://msdn.microsoft.com/en-us/library/gg685426.aspx  :: "Adding a Refiner to the Refinement Panel Web Part"

Processing this item failed because of an unknown error when trying to parse its contents. (Error parsing document 'https://sharepoint.contoso.com/Project/abcd/Q_M/ABX/SitePages/Homepage.aspx'. Sandbox worker pool is
closed.; ; SearchID = 820E86ZD-5252-4G5F-AK9U-2096ZV3L56PP )

In order to fix this you can try to perform the following action plan:

Open "Local Policies”

Click on “User rights assignment”

Make sure that the search service account has the following rights:

"Replace a process level token”

"Adjust memory quotas for a process"

"Impersonate a client after authentication"

Please make sure that the policies don't get changed afterwards.

After implementing the above changes please run a clear configuration cache as explained here: https://blogs.msdn.com/b/josrod/archive/2007/12/12/clear-the-sharepoint-configuration-cache-for-timer-job-and-psconfig-errors.aspx 

After clearing the cache, start a full crawl and the errors should be gone.  

Methods that  we tried to change the hidden attribute back to FALSE were PowerShell and also SharePoint Manager but when updating for example using PowerShell the below message was presented.

The reason for which the field could not be made visible again is the fact that the CanToggleHidden atribute was set to false as seen in the screenshot when creating the field as hidden using the above method. CanToggleHidden field property is true if the column can be hidden, otherwise, false. (https://msdn.microsoft.com/en-us/library/microsoft.sharepoint.spfield.cantogglehidden.aspx).

This behavior has been presented to the product group and is currently under investigation. I will update this post once the product group has provided more information.

To workaround the issue you can :

a) Change the field definition inside Visual Studio as shown below and then redeploy from Visual Studio the solution.

b) Create the field inside Visual Studio without specifying neither Hidden and neither CanToggleHidden. After the field is created use PowerShell to set the hidden property to TRUE

$field = $web.fields | ? { $_.InternalName –eq „PatientName1“ }

$field.hidden = $true

$field.update()

c) The CanToggleHidden value can be changed using Powershell only using reflection which is not supported and I saw that on the internet there are a couple of people that provide the PowerShell script but once again to be on the supprted safe side please open a service request to Microsoft support. I don't see thus any reason why to go for this method as the above methods cover each situation, when the field is allready present and when not.

Basically the challenge was: how to extract a specific timeframe from an 18GB log file.

Answer(as suggested by the great CristiG): using logparser. Initially meant for IIS logs, we can use logparser also for ULS with a little bit of adaptation. The main challenge was parsing the Timeframe column that is not in the standard format that logparser understands.

The resulting script looks like:

select [Timestamp],Process,TID,Area,Category,EventID,Level,Message,Correlation into test3.log from problem.log
where TO_TIME(TO_TIMESTAMP(STRCAT(STRCAT(STRCAT(STRCAT(STRCAT(STRCAT(TRIM(SUBSTR(Timestamp,6,4)),'-'),TRIM(SUBSTR(Timestamp,0,2))),'-'),TRIM(SUBSTR(Timestamp,3,2))),' '),TRIM(SUBSTR(Timestamp,11,8))), 'yyyy-MM-dd hh:mm:ss')) between timestamp('01:33:00', 'hh:mm:ss') and timestamp('01:43:00', 'hh:mm:ss')

After installing logparser and adding it to your path variable you can run the following command to filter the logs: LogParser -i:tsv -o:tsv file:test2.sql (given that you have the input file and the script file are in the same directory).

Logparser can be downloaded for free from: https://www.microsoft.com/en-us/download/details.aspx?id=24659

• Site usage confirmation and deletion (https://technet.microsoft.com/en-us/library/cc262420.aspx)
• Audit Log Reports https://sharepoint.microsoft.com/Blogs/GetThePoint/Lists/Posts/Post.aspx?ID=414

Site Usage confirmation and deletion allows to define a specified amount of "inactivity" time and after that the site collection owners are informed by mail and the audit log reports will show you the changes that have happened

So what can you do from a developer point of view. You can use SPWeb.LastItemModifiedDate/SPList.LastItemModifiedDate OR you can use the Change Log by calling the GetChanges method of an SPList, SPWeb inside a SPChangeQuery class. https://msdn.microsoft.com/en-us/library/bb447511.aspx

So the question is what to use and when. Here are some pointers I found out lately and want to share 

• When calling SPWeb.LastItemModifiedDate in the background we call proc_GetWebExtendedMetaData which does SELECT and then takes MAX(ALAux.Modified) from AllListsAux so actually SPWeb.LastItemModifiedDate simulates this by taking the max of all list timestamps in SQL. Changes can happen to a web outside of a list that would not increment the values processed by SPWeb.LastItemModifiedDate which makes SPWeb.LastItemModifiedDate not very reliable. The time returned is using DateTime.SpecifyKind method  https://msdn.microsoft.com/en-us/library/system.datetime.specifykind.aspx. There is no timer job involved when calling SPWeb.LastItemModifiedDate. SPWeb.LastItemModifiedDate and SPList.LastItemModifiedDate provide a way to check date/time of change for items that are old (I mean here older thant 15 days - you will see below what I mean).
• The Change Log which actually means calling GetChances which is based on the EventCache table. There is a Change Log Timer Job that clears the EventCache table of entries that are older than 15 days. This value can be increased of course witht the risk that the EventCache Table will grow. So as you see using the GetChanges method you will get entries that are never as the log retention period defined so it is a good method of tracking changed only for new changes not old changes.  The modified date/time can be also fetched using let's say using SPChange.Time https://msdn.microsoft.com/en-us/library/microsoft.sharepoint.spchange.time .

Job-change-log-expiration https://technet.microsoft.com/en-us/library/cc424964(v=office.12).aspx - Specifies the time schedule when the change log timer job occurs

Change Log - Removes expired entries from the change log of the Web application.https://technet.microsoft.com/en-us/library/cc678870.aspx

Change-log-retention-period https://technet.microsoft.com/en-us/library/cc261921(v=office.12).aspx

It seems the 'Title' managed property is always populated with the filename despite the title being mapped by default.

In order to receive the title of the zip file in the search results rather than the filename you have to implement the following changes:

When investigating the current crawl/index behavior togheter with Senior Support Escalation Engineer Holger Lutz we could see that out-of-the-box SharePoint choses for .zip files to store the filename in the index as title
instead of the SharePoint column title:

 
    
The filename as title gets returned in the search results:

Here is an excerpt of the debug-output of our troubleshooting simplepi.dll where we see that AppPoolAssociationForm.zip gets identified as title and put into the index:

PID=2 Title Value(LPWSTR)=AppPoolAssociationForm.zip Property:   gather,   Search= '0B63E350-9CCC-11D0-BCDB-00805FCCCE04', 'urn:schemas.microsoft.com:fulltextqueryinfo:displaytitle', 31,  crawled_propid=107 pri=300 Flags: Index Retriev ResPri Dups MD5

In the debug-output we see that the right title gets identified but assigned to a different crawled property called EmbeddedContent:

Text:plugin,  ++Gatherer= '0B63E343-9CCC-11D0-BCDB-00805FCCCE04', 'EmbeddedContent', 31, crawled_propid=101 pri=500 Flags: Index Retriev ResPri Dups MD5 PID=2 Title Lcid 1031 ZipTitle

As a solution to get the SharePoint title column content indexed and correctly returned in the search results for .zip files a change of the crawled property <-> managed property association is required:

This is how the Title managed property is configured out-of-the-box:

You have to add the EmbeddedContent(Text) property at the following position for the Title managed property to get the title correctly returned in the search results for .zip files:

The title gets correctly returned after a full crawl / incremental crawl which picks up the zip file(s) in the search results:

Our tests show that the EmbeddedContent(Text) crawled property will be empty for other files like doc / docx / txt / xls / xlsx / …but gets the title value for .zip files.

This behavior is expected (it dates back to the days of FrontPage 2000) and is also usually the reason why the folder was named with a leading underscore.

However, if the folder is then renamed and the underscore is removed, the folder will still remain hidden in "Explorer View". This is why:

Windows Explorer communicates with SharePoint over WebDAV. For each object stored in SharePoint and browsed from Windows Explorer, the Win32FileAttributes property is sent over the wire:

Request:

PROPFIND <DocLib_URL>

[...]

Depth:1

Response:

207 MULTI-STATUS

[...]

<D:prop><D:displayname>_hidden_folder</D:displayname>[...]<D:isFolder>t</D:isFolder>[...]<Z:Win32FileAttributes>00000016</Z:Win32FileAttributes></D:prop>

[...]

The value of Win32FileAttributes (specified in the Microsoft Extensions to the WebDAV protocol) informs Windows Explorer to treat this folder as hidden. In SharePoint, this property is stored in the SPFolder object's property bag:

$folder = (Get-SPWeb https://url).Folders["DocLib_Name"].SubFolders["_Folder_with_underscore"]

$folder.Properties["vti_winfileattribs"]

It is not automatically removed when the folder is renamed, but can be manually reset using:

$folder.Properties["vti_winfileattribs"]=""