FromTheField

Test-SPContentDatabase : Automating missing feature removal

Steve Jeffery — Sat, 26 Aug 2017 06:26:46 GMT

Recently I've been working with a customer to help them with a SharePoint migration project. One of the steps that we've worked on is to run the Test-SPContentDatabase cmdlet to check for missing files, features, web parts, assemblies, etc. One of the content databases that was tested had somewhere in the region of 200 missing features. The customer confirmed that they didn't need to keep the feature references so we looked at ways of automating the removal of features (shockingly we never wanted to remove these manually!). Test-SPContentDatabase return errors in a really nice format (Microsoft.SharePoint.Upgrade.SPContentDatabaseTestResult). As each error is returned as an object, we can check the 'Category' property in our script and do something if the category equals 'MissingFeature', which is what we do in the script. An example of the type of error that we want to check for:

[MissingFeature] Feature Id [fe5408f9-d9d4-a317-6c7f-122032c54475] is referenced [12] times in the database [WSS_Content_ContentDatabase1], but is not installed on the current farm.

We can check each error by storing all errors that are detected: $testResults = Test-SPContentDatabase -Name -WebApplication And then looping through each error individually:

foreach ($result in $testResults)
{
<# We want to check the type of error #>
}

Once we've matched the category:

if ($result.Category -eq "MissingFeature")
{
<# We've found a missing feature #>
}

We then extract the feature Guid from the 'message' property using a regular expression:

$regularExpression = "([a-f0-9]{8}[-][a-f0-9]{4}[-][a-f0-9]{4}[-][a-f0-9]{4}[-][a-f0-9]{12})"
$match = [regex]::Matches($result.Message, $regularExpression)
$guid = $match[0].Value

And then try to delete the feature reference by calling a function that we've created (see the entire script below):

Write-Host "Deleting feature id:" $guid
Delete-SPFeatureReference -contentDatabase  -guid $guid

Hopefully that all makes sense, the entire script is below:

$testResults = Test-SPContentDatabase -Name  -WebApplication 

function Delete-SPFeatureReference()
{
Param
(
[string]$contentDatabase,
[string]$guid
)

$db = Get-SPDatabase | Where-Object {$_.Name -eq $contentDatabase}

foreach ($site in $db.Sites)
{
Start-SPAssignment -Global
$siteFeature = $site.Features[$guid]
if ($siteFeature)
{
$site.Features.Remove($siteFeature.DefinitionId, $true)
}
else
{
foreach ($web in $site.AllWebs)
{
$webFeature = $web.Features[$guid]
if ($webFeature)
{
$web.Features.Remove($webFeature.DefinitionId, $true)
}
}
}
Stop-SPAssignment -Global
}
}

foreach ($result in $testResults)
{
if ($result.Category -eq "MissingFeature")
{
$regularExpression = "([a-f0-9]{8}[-][a-f0-9]{4}[-][a-f0-9]{4}[-][a-f0-9]{4}[-][a-f0-9]{12})"
$match = [regex]::Matches($result.Message, $regularExpression)
$guid = $match[0].Value
Write-Host "Deleting feature id:" $guid
Delete-SPFeatureReference -contentDatabase  -guid $guid
}
}

Hopefully this helps, you could extend the script to remove other kinds of errors based on the category. Cheers, Steve

SharePoint Saturday Cambridge

Brendan Griffin — Mon, 12 Sep 2016 10:47:40 GMT

I was fortunate to speak at SQL/SharePoint Saturday Cambridge last weekend, this was an absolutely fantastic event and kudos must go to Mark Broadbent for organising this event. Unlike other SharePoint Saturday's this is a joint event with SQL so plenty of opportunities for SharePoint folk to learn more about SQL! I presented the session "What's New for IT Pro's in SharePoint 2016" and here are the slides. There were two questions that I received during my session that I had to take a closer look into, here are my findings:

The Automatic Index Management capability targets lists that have at least 2500 items, this figure is derived from the List View Threshold. Basically it takes the current value for this setting and divides by 2. Therefore, if you change the LVT for a Web Application this will affect the execution of the Timer Job. For example if you increased the LVT to 10,000 this will cause the Timer Job to only process lists with greater than 5000 items, rather than the default of 2500.
When the Extensible App Launcher is in use, end users do not need direct access to the Internet in order for this to be populated with additional Apps that have been added (however the Office 365 ones would be useless in this scenario!). When the On-Premises SharePoint site is requested for the first time a request is sent to /_api/Microsoft.SharePoint.Portal.SuiteNavData.GetSuiteNavData. The On-Premises SharePoint Web Server then makes a request to the Office 365 MySite host (https://tenant-my.sharepoint.com) to obtain a list of apps to add to the navigation.

Brendan Griffin - @brendankarl

Removing Multiple Site Collections from Search Results in SharePoint Online

Brendan Griffin — Thu, 28 Jul 2016 12:45:10 GMT

I recently blogged about a scenario that my customer was having whereby they needed to temporarily prevent a Site Collection from appearing in search results within SharePoint Online and I documented a solution that used a search query rule - https://blogs.technet.microsoft.com/fromthefield/2016/07/20/2095 My customer has been playing around with this and ran into some unexpected behaviour - in short if you have multiple sites that you'd like to omit from search results, consolidate these into a single query rule rather than creating a separate query rule for each to prevent inconsistent search results. In my example below, I have omitted results for both the HR and Finance Site Collections using the "OR" operator. Brendan Griffin - @brendankarl

Removing a Site Collection from Search Results in SharePoint Online

Brendan Griffin — Wed, 20 Jul 2016 12:33:22 GMT

One of my customers presented me with an interesting scenario recently, their security team were looking to define a process to take a SharePoint Online Site Collection temporarily offline. This is easy enough to achieve using the PowerShell cmdlet Set-SPOSite (example below).

Import-Module Microsoft.Online.SharePoint.PowerShell -DisableNameChecking -ErrorAction SilentlyContinue
Connect-SPOService -Url https://tenant-admin.sharepoint.com -credential tenantadmin@tenant.onmicrosoft.com
Set-SPOSite -Identity https://tenant.sharepoint.com/sites/site -LockState NoAccess

However, there is still a chance that items contained within the Site Collection are returned in search results which could disclose some of the content (although the items themselves wouldn't be accessible as the site is locked). We took a look at the Search Results Removal feature – Admin Center > Admin Centers > SharePoint > Search > Remove Search Results. However, this feature is for removing specific items from the search index rather than an entire Site Collection. This would be perfect if needed to remove a specific document from the search index, unfortunately in this case we needed to remove an entire Site Collection. The approach that we eventually took was to use a Query Rule. This doesn’t actually remove the Site Collection from the search index, it merely adapts the search query to omit any results for content within the Site Collection so isn't 100% foolproof and can be worked around. If you do need to block a Site from being crawled you can do this via Search and Offline Availability in Site Settings. In order to create a Query Rule to omit the Site from search results, do the following:

Navigate to the Search Center Site Collection within your tenant (if you'd like to enforce this tenant-wide rather than this specific Search Center you will need to do this via SharePoint Admin Center > Search > Manage Query Rules)
Select Site Settings
Select Search Query Rules within the Site Collection Administration section
From the Select a Result Source… dropdown select Local SharePoint Results
Click New Query Rule
Give it an appropriate name
Within the Query Conditions section click Remove Condition
Within the Actions section select Change Ranked Results by Changing the Query
Append the following in the Query text box (updating the URL to that of the Site Collection to omit from Search Results) -path:https://tenant.sharepoint.com/sites/HR. In my example I have excluded the HR Site Collection.
Click OK and then Save

If you now submit a search query via the Search Center, you should notice that any results from this Site Collection will be omitted. If you then need to permit search results from the Site Collection you can either delete the Query Rule or make it inactive (via the rule dropdown). This approach will also work with SharePoint 2013 and 2016. Brendan Griffin - @brendankarl

Export-SPWeb - exporting lists

Steve Jeffery — Thu, 14 Jul 2016 14:52:05 GMT

Hi. Recently I've been working with the Export-SPWeb Cmdlet, helping out a customer who needs to export a specific list. I quickly realised that it is actually a little bit more confusing than what you would expect, so I thought I would share the following.
Example one - exporting a list that exists in the root site collection:

Export-SPWeb -Identity https://content.contoso.cloud -ItemUrl /lists/testlist -Path "c:\export\export1.cmp"

Example two - exporting a list from a site collection that is not root:

Export-SPWeb -Identity https://content.contoso.cloud/sites/site-collection2/ -ItemUrl lists/testlist2 -Path "c:\export\export2.cmp"

Example three - exporting a list that exists in a sub site, in a site collection that is not root:

Export-SPWeb -Identity https://content.contoso.cloud/sites/site-collection2/sub1/ -ItemUrl lists/testlist3 -Path "c:\export\export3.cmp"

Notice that the parameters change slightly depending on where the list exists?

Hope this saves you some time! :)

Cheers.

Site collection provisioning and customisation scenarios for SharePoint online

Steve Jeffery — Wed, 06 Jul 2016 16:40:48 GMT

Site collection provisioning and customisation scenarios for SharePoint Online

I’ve worked with a few customers recently, whom are either starting a deployment project in SharePoint Online, or some way into one, and one of the challenges they had to deal with is how best to provision customised site collections. This article covers site collection provisioning and customisation for the following scenarios: 1/ Bulk site collection provisioning and customisation 2/ Ongoing/ requested site collection provisioning and customisation.

Scenario 1 – Bulk provisioning of site collections.

Some customers must deploy large numbers of customised site collections, which means that provisioning site collections manually is not really an option. To provide some real world context, I worked with a customer a few months ago that was facing exactly this challenge. They had to provision ~250 site collections (one for each branch office); each site collection had several sub-sites and was relatively customised (custom master page, lists/libraries, content types etc). Manually it was going to take about 3-4 hours per site collection to apply the customisations. This, for obvious reasons, this option could not seriously be considered due to resource requirements. A solution that I worked with the customer to develop, was to automate the site collection provisioning process using PowerShell. We agreed that a CSV file would act as a work list containing the information required to provision site collections. The process would take each entry in the CSV file, provision a site collection, and apply customisations to it depending on the type of site collection required. The flow/ process looked something like this:

PowerShell…

PowerShell has been around for a long time now, and most customers that I work with are very experienced with PowerShell, but often haven’t had to use anything outside of the Cmdlets that are provided with the on-premise SharePoint Snap-in. A brief tangent from the story here, if you haven’t had to write any PowerShell for SharePoint online so far, then there are a few things that you should be aware of:

SharePoint Online management shell (introduction & download) – is a really useful tool if you are a SharePoint Online administrator. It contains approx. 42 Cmdlets that enable to you to perform basic administration tasks; however, you should understand a few things:
1. The smallest entity that you can work with is a Site Collection; therefore, you will not be able to access a sub-site, list/ library, etc.
2. To use the SharePoint Online management shell, you must be a SharePoint Online global administrator.

SharePoint Online objects – are different from their on-premise cousins. Try comparing the results from returning a SharePoint Online site collection (Get-SPOSite) and the On-Premise equivalent (Get-SPSite). You’ll notice if you run a Get-Member (e.g. Get-SPSite -Identity https://intranet.contoso.com | Get-Member) on both, that the SharePoint Online site collection object is actually a Microsoft.Online.SharePoint.PowerShell.SPOSite. It has no real methods of use, and some properties are read-only; it also has substantially less properties than the on-premise equivalent.

CSOM – If you have been scripting with PowerShell for SharePoint (on-premise) for a few years, then there is every chance that you haven’t had to deal with the CSOM. The Client Side Object Model offers an extensive number of SharePoint objects that you can work with, from your client PC/ any server that has CSOM available. It does however, require you to really think about how you are going to approach your script as you should understand ‘request batching’.

Back to the scenario! PowerShell was the preferred choice for this customer scenario; because using a CSV input file it meant that the customer was in control over the provisioning process; and PowerShell is (debatably) easier to read. There is no reason you couldn’t take another approach. However, as I will iterate several times, there are a few differences to the approach you must take developing a PowerShell script for SPO as opposed to On-premise installations of SharePoint.

Connections

– whether you are going to run Cmdlets from the SharePoint Online management shell, use the CSOM in your PowerShell script or the PnP PowerShell Module, you are not running your code directly on the SharePoint server. This means that you must establish a connection with your SPO tenant from your local system, or the system running the code, before you attempt to do anything else. This is achieved by running the Connect-SPOService Cmdlet, providing the Url of your admin tenant, and the username of your SharePoint Online global administrator account:

$adminCenterUrl = "https://-admin.sharepoint.com" $adminAcct = "admin@.onmicrosoft.com" Import-Module Microsoft.Online.SharePoint.PowerShell -DisableNameChecking Connect-SPOService -Url $adminCenterUrl -Credential $adminAcct

Also, please note that in the code above, I have imported the SharePoint Online management shell module prior to running the Connect-SPOService Cmdlet. A few things to note here… The script is going to prompt you for the password of the account (credential) that you use. If you want to run the script unattended (i.e. you won’t be around to type the password in), then I invite you to look into using Windows Credential Manager: Store credentials using Windows Credential Manager:

Open the control panel (from run: control panel).
Click on User Accounts
Click on Credential Manager
Click on Windows Credentials
Click Add a generic credential
Enter the following information:
Internet or network address – think of this as a reference or alias
User name
Password

You should end up with something like this: You can now make an adjustment to your script: Connect-SPOService -Url $adminCenterUrl -Credential SPOAdmin The script will now refer to the credential that you just created; meaning you can run the script with a scheduled task if needed (please ensure that you have logged on with the account that you intend on using to ensure that a local profile exists).

Input file…

The script that we developed took a CSV file (see below) as its input with the intention of iterating through each line and provisioning a site collection using the OOTB team site. We achieved this using the Import-Csv Cmdlet: $build = Import-Csv -Path "C:\temp\SiteCollectionBuild.csv"

Templating…

Back in the day, customers would often create a site, configure it for a purpose through the user interface, save it as a template, and then make it available to end users. This is fine, however, it does present a few administration challenges. Nowadays I always recommend to customers that they first define what a site should look like (e.g. document any sub-sites, lists, libraries, content types, web parts etc) and provision them with code; either CSOM or PowerShell. This does mean that you must have the skillset to write the code/ script but in my opinion provides a more controlled method for deploying site collections. We’ll be talking a bit more about this in the next scenario; however if this approach to site collection ‘templating’ makes you feel nervous, then perhaps have a watch of this channel 9 session where Vesa Juvonen explains it in far more detail than what I am going into here. How we managed the type of site in the customer scenario that I am walking you through was the addition of a ‘Site template’ column in the CSV file: Now, in the provisioning code, we can detect the type of site collection that should be provisioned (if $siteCollection.Type -eq “ProjectSite”):

foreach ($siteCollection in $buildList)
{
$site = $siteCollection.Url
Write-Host "Provisioning $site" -ForegroundColor Yellow
New-SPOSite -Url $siteCollection.Url -Owner $siteCollection. Owner -Title $siteCollection.Title -StorageQuota 1000 -Template STS#0 -LocaleId 1033
if ($siteCollection.SiteType -eq "ProjectSite")
{
New-SPOList -Title "Project Risks" -Template GenericList -Url Lists/ProjectRisks -OnQuickLaunch
New-SPOList -Title "Project Issues" -Template GenericList -Url Lists/ProjectIssues -OnQuickLaunch
New-SPOList -Title "Project Documents" -Template DocumentLibrary -Url Lists/ProjectDocuments -OnQuickLaunch
New-SPOList -Title "Project Design" -Template DocumentLibrary -Url Lists/ProjectDesign -OnQuickLaunch
}if ($siteCollection.SiteType -eq "TeamSite")
{
New-SPOList -Title "Team Social" -Template GenericList -Url Lists/TeamSocial -OnQuickLaunch
Add-SPOField -List "Team Social" -DisplayName "Event Name" -InternalName "EventName" -Type Text -AddToDefaultView
}
}

A quick walkthrough of the code above. We’ve imported the contents of the CSV file into a variable called $buildList. We then iterate through each line of the $buildList, and provision a new site collection using the OOTB team site template (STS#0). At this point, we check to see what type of customisations we must apply to the site collection. This is managed by the conditional ‘if’ statement:

If ($siteCollection.SiteType -eq “ProjectSite”)
{

}

The code between the curly braces is where we start to provision the relevant artefacts that make up the type of site; e.g. ‘ProjectSite’. In the example that I am using here, you can see that I am provisioning new custom lists and fields. In your scenario you might also be deploying branding, content types, etc. The SharePoint Online management shell doesn’t have any Cmdlets that provide this functionality, so how is this working?

Office Dev Patterns and Practices…

The script that the customer and I wrote heavily used an amazing community collaboration program called the Office Dev Patterns and Practices. This is a collection of source code projects that address common challenges/ requirements, and in addition to this, an extremely comprehensive collection of PowerShell Cmdlets designed to help customers be incredibly efficient. This PowerShell module is how we apply the customisations to our site collection. The PowerShell Cmdlets are available at GitHub. Here is how to install and start using the module (download it before proceeding J): Install PnP-PowerShell-master binaries

Download and extract PnP-PowerShell-master from the GitHub repository.
Navigate to the Binaries folder.
Run SharePointPnPPowerShellOnline.msi
Choose (or not) to accept the terms in the license agreement.
Click Install.

Alternatively you could use the Install-Module Cmdlet: Install-Module SharePointPnPPowerShellOnline Now you are good to start using the Cmdlets. If you head over to the GitHub repository you’ll be able to review the Cmdlet documentation, which is very thorough and get an idea of what you can now do with your SPO tenant! The complete code now looks like this:

$adminUrl = "https://-admin.sharepoint.com"
$spoAdmin = "admin@.onmicrosoft.com"
$buildList = Import-Csv -Path C:\temp\SPOSiteCollectionBuildList.csv
Connect-SPOnline -Url $adminUrl -Credentials $spoAdminforeach ($site in $buildList)
{
New-SPOSite -Url $site.Url -Owner $site.Owner -Title $site.Title -Template STS#0 -StorageQuota 2000 -LocaleId 1033If ($site.Type -eq 'ProjectSite')
{
<#Apply customisations, e.g. branding, content types, lists & libraries, etc

#>
}
}

That wraps up the first scenario; I hope that you find it useful.

Scenario two – ongoing provisioning and customisation of site collections

The second scenario that I want to talk about is managing the ongoing requests for site collections. In the first scenario we discussed a fantastic community program (Office Dev Patterns and Practices). We’re going right back to this now as there is a fantastic code sample that provides us with a solution to provisioning site collections. Please visit this URL. This is a great example of asynchronous site collection provisioning using a provider-hosted SharePoint add-in. Vesa Juvonen has a very detailed walk through of the code and flow in a channel 9 video. The flow is remarkably like what we covered in the bulk site collection provisioning scenario: The process starts off with the user completing a custom site collection request form, like this example (note: the screenshot was taken from a high-trust add-in implementation, so ignore the managed path selection box): Once the user has submitted the request, the user receives confirmation: And at this point a list item is created in a site provisioning request list. That is the first phase of the provisioning process completed. The second phase is to provision the site collection. Much like the bulk site collection creation scenario that we covered earlier, we have a couple of options available to us:

A console application
A PowerShell script

Either of the above would be able to build a site collection and apply the customisations depending on the type of site required. If you are new to provider-hosted add-ins, then here are some useful resources: Introduction to SharePoint add-ins: https://msdn.microsoft.com/en-us/library/office/fp179930.aspx Hands on lab: getting started with SharePoint Add-ins - https://dev.office.com/hands-on-labs/1518

Cloud Search Service Application - No Results Returned from On-Premises Search Center

Brendan Griffin — Tue, 21 Jun 2016 08:27:52 GMT

The Cloud Search Service Application is a brilliant new-ish feature of SharePoint 2013 and 2016 that allows you to store your On-Premises search index in Office 365. My colleagues Manas Biswas and Neil Hodgkinson have written a fantastic guide on how to set this up here. When I setup the Cloud SSA for the first time I run into an issue whereby I could retrieve search results from the SharePoint Online Search Center, which included items from both On-Premises and SharePoint Online, however when I submitted a search query via my On-Premises Search Center no results were returned - even from On-Premises! It turns out there is an additional step required, I'm documenting this here as as I know others that have run into this too. Here are the steps to rectify this:

Launch Central Administration
Select Manage Service Applications
Select the Cloud SSA that you have provisioned
Select Result Sources
Click New Result Source
Give it an appropriate name, select Remote SharePoint for the Protocol
Within the Remote Service URL enter the URL of the top-level Site Collection for your tenant (e.g. https://tenant.sharepoint.com)
Click Save
Right click the new result source that you have just created and select Set as Default

That should hopefully sort it out for you :) Brendan Griffin - @brendankarl

Outputting SharePoint Online Tenant Settings using CSOM

Brendan Griffin — Wed, 01 Jun 2016 12:54:56 GMT

Here is a quick script that outputs the current SharePoint Online tenant settings using CSOM with PowerShell, this is similar to using the Get-SPOTenant Cmdlet from the SharePoint Online Management Shell however could come in useful if you don't have this installed. Simply update the value of the $Site variable, replacing "tenant" with the name of your tenant. This script requires the CSOM DLLs, the easiest way to get these is via Nuget. #Tenant Admin URL $Site = "https://tenant-admin.sharepoint.com" #Add references to SharePoint client assemblies and authenticate to Office 365 Add-Type -Path "D:\CSOM\Microsoft.SharePoint.Client.dll" Add-Type -Path "D:\CSOM\Microsoft.Online.SharePoint.Client.Tenant.dll" $Username = Read-Host "Please enter your username" $Password = Read-Host -Prompt "Please enter your password" -AsSecureString $Context = New-Object Microsoft.SharePoint.Client.ClientContext($Site) $Creds = New-Object Microsoft.SharePoint.Client.SharePointOnlineCredentials($Username,$Password) $Context.Credentials = $Creds #Create tenant object and output current settings $Tenant = New-Object Microsoft.Online.SharePoint.TenantAdministration.Tenant($Context) $Context.Load($Tenant) $Context.ExecuteQuery() $Tenant Here is an example of the output: Brendan Griffin - @brendankarl

SharePoint Saturday Paris

Brendan Griffin — Sun, 29 May 2016 17:24:54 GMT

I had the pleasure of speaking at SharePoint Saturday Paris yesterday (28th May), this was an absolutely fantastic event with some great speakers and top quality sessions. Here are the slides from my presentation on SharePoint Hybrid Capabilities, where I covered Hybrid OneDrive for Business, Team Sites and the Cloud Search Service Application. Jeff Teper delivered the keynote and we had the pleasure of his company at the speaker dinner on the Friday night.

SharePoint Responsive UI Package - Issue with JQuery

Brendan Griffin — Wed, 04 May 2016 11:23:23 GMT

I've been having a play around with the fantastic SharePoint Responsive UI package that was recently released by the PnP team. This provides the ability to make SharePoint 2013 and 2016 responsive. I initially set this up in one of my test VMs that isn't connected to the Internet and had some issues getting it to work, I fired up Fiddler and found the culprit which was that it references JQuery. As my VM doesn't have Internet access, it wasn't able to download JQuery and it failed miserably! This was easy enough to fix, I headed over to https://jquery.com/download and downloaded jquery-1.12.0.min.js, uploaded to a Site in my SharePoint farm that was running on my VM and then updated the reference within PnP-Responsive-UI.js to point to the location I uploaded JQuery to. I then disabled/enabled using the Disable/Enable-SPResponsiveUI.ps1 PowerShell Cmdlets and it worked! Obviously this scenario is only really useful if you don't have Internet access! Brendan Griffin - @brendankarl