• Run the following PowerShell
  $outputFolder = Read-Host "Enter a location for the output file (For Example: C:\Temp)" $filename = "Check-InvalidSystemUserKey.csv" $outFile = $outputFolder + "\" + $filename $webAppUrl = Read-Host "Please enter the URL of the Web App you want to check ( Ex: http://sharepoint )" "SiteUrl~WebUrl~User~LoginName~InvalidSystemUserKey" | Out-File $outFile $wa = Get-SPWebApplication $webAppUrl foreach($site in $wa.Sites) { foreach($web in $site.AllWebs) { "" "Checking web: " + $web.Url $SPUsers = $web.SiteUsers | ?{$_.SystemUserKey -notmatch "c:" -and $_.SystemUserKey -notmatch "i:" -and $_.SystemUserKey -notmatch "s-1-"} if($SPUsers.Count -gt "0") { foreach($user in $SPUsers) { Write-Host ("Found user, " + $user.DisplayName + ", with an Invalid SystemUserKey ") -ForegroundColor Yellow $web.Site.Url + "~" + $web.Url + "~" + $user.DisplayName + "~" + $user.UserLogin + "~" + $user.SystemUserKey | Out-File $outFile -Append } } else { Write-host ("No Invalid SystemUserKey found within this set of Users... ") -ForegroundColor Green } } $web.Dispose() } $site.Dispose() "" "Finished Checking Web App..." "Please send the file created at '$outFile' to Microsoft Support"
• This will prompt you for the directory ( this path, like 'C:\Temp' should already exist ) you want to save your file called "Check-InvalidSystemUserKeys.csv"
• You will then get prompted for the URL of the Web App you want to check
• It will then iterate through each Site Collection and Web within the Web App and check for users that have a "SystemUserKey" that does not look conventional ( like starting with "c:", "i:" or "S-1-x" )
• It will output on the console as it checks each SPWeb and report if it finds any "Invalid" users..
  
• It will also put the info into this .csv file.. and that info will look like this:
  ** Note ** Often times, if a user is granted Perms at a Site Collection level ( Root Site )  and you have subsites that inherit permissions from the Root, then you will see this same user as "Invalid" throughout all the webs within that Site Collection..
• You can then import this file into Excel, using the " ~ " as the delimiter and then sort and filter based on like the "SiteUrl" column if you need to..
• If you see accounts with SystemUserKey info like above, this is likely your culprit to the errors you see in the Crawl Logs..
  • We have also seen where the "SystemUserKey" is actually "null" or "empty".
  • If this is the case, please refer to this article, because we suspect the errors in your Crawl Log and ULS logs will match this and it will fail on the mssdmn.exe process
• Coming back to this issue..

How do we fix this?

• You can run the following PowerShell to again iterate the Web Application, looking for "invalid" accounts and if found, it will remove them..
  $webAppUrl = Read-Host "Please enter the URL of the Web App you want to check ( Ex: http://sharepoint )" $wa = Get-SPWebApplication $webAppUrl foreach($site in $wa.Sites) { foreach($web in $site.AllWebs) { "" "Checking web: " + $web.Url $SPUsers = $web.SiteUsers | ?{$_.SystemUserKey -notmatch "c:" -and $_.SystemUserKey -notmatch "i:" -and $_.SystemUserKey -notmatch "s-1-"} if($SPUsers.Count -gt "0") { foreach($user in $SPUsers) { Write-Host ("Found user, " + $user.DisplayName + ", with an Invalid SystemUserKey ") -ForegroundColor Yellow Write-Host ("Removing User: " + $user.DisplayName) -ForegroundColor DarkYellow Remove-SPUser -Web $web -Identity $user -Confirm:$false } } else { "No Invalid SystemUserKey found within this set of Users... " } } $web.Dispose() } $site.Dispose() "" "Finished Checking Web App..."
• After this, try your crawl again.

• Let's say that you have 4 servers in your SharePoint Farm and of those 4, 2 of them belong to my search topology and my Topo looks something like this:
  
• For my example here, I am troubleshooting a query related issue so I want to collect ULS logs from "ajcnsv5app" and "ajcnsv5app1" from earlier in the day ( 11/14/2017 ) between like 12:30 PM and 1:30 PM
• I have put this script on one of my servers, which has ample space, in a path like "D:\PowerShell"
• I open the SharePoint Management Shell, or PowerShell ISE, and I change paths to "D:\PowerShell"
• The command will be ran like: ( a few comments below about usage )
  .\Get-UlsLogs.ps1 -Servers ajcnsv5app, ajcnsv5app1 -startTime "11/14/2017 12:30PM" -endTime "11/14/2018 13:30"
  • You can pass in any array of servers you have. If you leave the "-Servers" switch off, it will grab ULS from All your SharePoint servers in the farm.
  • If you do not append the "startTime\endTime" in the command, it will prompt you for each parameter as we consider these mandatory parameters
  • In the example above, the time can be AM\PM or 24-hour format and you can wrap it in single or double quotes.

• It will prompt you for a location you want to save these files:
  
• As you can see I chose "D:\Temp" since I have plenty of space on my server to store these..
• The next thing it will do is check to see if the D:\Temp exists, if not, it will tell you and you will be prompted again for a path to store the files.
• When you hit Enter, it will begin working through the servers you defined and do the following:
  • Create a temp directory @ "D:\Temp\ajcnsv5app"
  • Look for ULS logs that match the startTime\endTime criteria ( we don't actually crack the ULS logs open and pull those specific times. We utilize the "CreationTime" property of the file and manipulate the startTime a little by substracting some minutes from the time entered. We did this to ensure we grabbed the ULS log that would contain the timeframe we are looking for. )
  • It will copy the files that we matched on into this temp path @ "D:\Temp\ajcnsv5app"
  • It will then zip those Log files up for you
    
  • It will also clean up the Temp Paths it creates for each server so that all you have left is the Zip files created:
    
  • You can then upload these Zipped ULS logs to the Workspace that Microsoft provides for you.

• We can utilize PowerShell to emulate this sitedata.asmx call on the SiteCollection Object.
• I suggest doing this, so we can investigate the XML and\or use it for reference later.
• You can see in the ULS sample above that we called the "SiteCollection" object on https://foo/_vti_bin/sitedata.asmx ( if it were another SiteCollection, you would see the URL in this ULS entry )
• In order to get this XML, you can run this ( when prompted for creds, you want to use the Default Content Access Account credentials )

  ########################################################################################

  ## Set SiteData Web Service Object ### Use the Content Access Account for the $creds variable

  ########################################################################################

  $creds = Get-Credential $url = "https://foo" $wsUrl = $url + "/_vti_bin/sitedata.asmx?wsdl" $sitedataWS = New-WebServiceProxy -Uri $wsUrl -Credential $creds

  ########################################################################################

  ## SiteCollection Object - ObjectType = 2 ##

  ########################################################################################

  $int = [int]2 $xmlinput = "<GetContent><ObjectType>2</ObjectType><ObjectId /><FolderUrl /><ItemId /><RetrieveChildItems>False</RetrieveChildItems><SecurityOnly>False</SecurityOnly><LastItemIdOnPage /><AllowRichText>False</AllowRichText><RequestLoad>100</RequestLoad><RemoveInvalidXmlChars>True</RemoveInvalidXmlChars></GetContent>" 

  $xml = $sitedataWS.GetContentEx($int,$xmlinput)

  $xml | Out-File "C:\Temp\SiteCollection.xml"

• Once you collect the XML, we can parse it through additional PowerShell
• Some time back, we created some PowerShell that would try to mimic this "ParseGroupCollection" and try to identify duplicate "Groups" within the returned XML.

#the attributes that are in the group element/node

$attributes = "ID","Name","OwnerID","OwnerIsUser","Description"

#the object array of custom objects we will use to return data

$results = @()

#setup the xml reader and get the data

$xmlReader = [System.Xml.XmlReader]::Create('C:\Temp\SiteCollection.xml')

#read dat data

while($xmlReader.Read())

{

#Set things we want to check ahead of the compare operation

$isElementNodeType = $xmlReader.NodeType -eq [System.Xml.XmlNodeType]::Element 

$isGroupNode = $xmlReader.Name -eq 'Group' #Compare ALL THE THINGS! if($isElementNodeType -and $xmlReader.HasAttributes -and $isGroupNode)

{ #create object to store the current group element  $currentGroup = "" | Select "ID","Name","OwnerID","OwnerIsUser","Description"

#loop through attribute array set above. foreach($attribute in $attributes) { #start populating data in $currentGroup by reading that attribute from current point in reader

$currentGroup.$attribute = $xmlReader.GetAttribute($attribute) 

} #add current group to results $results += $currentGroup } } $resultcount = $results.Count Write-Host "There are $resultcount group elements" ##see the items easily #$results | Select Id, Name | Out-GridView #group items and then look for items when 

$results | group -Property name | ? {$_.count -gt1} | select count, name $results | group -Property Id | ? {$_.count -gt1} | select count, name

• If there are any Duplicates, on the Name or ID, we would dump out the Name or ID with a count and the Name of the duplicate Group
• If you get any returns on these, where count > 1, then you can go find those groups in the Site Permissions of your Site and remove one of the Groups.
• There are times that parsing through this XML doesn't yield any Duplicate Names\IDs so what do you do in that case?
• This is where a Process Dump would be handy
• We have a pre-created rule that will\should dump on the following:

• Download "Debug Diag 2.0" from here:

• Install it on your Crawl Server(s)
• Once you install it, open "DebugDiag 2.0 Collection"
• When the crash\hang wizard comes up, click on "Close" or Cancel
• Then in the Lower right corner, click on the "Import" button
• Select that "tag_dm1k_partial.ddconfig" file you saved to your Crawl Server(s)
• Click OK
• Once the rule loads up, right click the Rule > Activate Rule
• Trigger a crawl..

• This should monitor mssdmn.exe process and capture a DMP file when that EventId = dm1k is thrown, with that message in the ULS

• looking at the DMP file.

• i see this
  0:106> !threads ThreadCount: 143 UnstartedThread: 0 BackgroundThread: 143 PendingThread: 0 DeadThread: 0 Hosted Runtime: no Lock ID OSID ThreadOBJ State GC Mode GC Alloc Context Domain Count Apt Exception 0 1 1a60 000000338a9306f0 20220 Preemptive 0000000000000000:0000000000000000 000000338a923120 0 MTA 4 2 3138 000000338a93b3e0 2b220 Preemptive 0000000000000000:0000000000000000 000000338a923120 0 MTA (Finalizer) 9 3 3628 00000033a48469a0 20220 Preemptive 0000000000000000:0000000000000000 000000338a923120 0 MTA 13 4 24ec 00000033ad63a0e0 28220 Preemptive 0000000000000000:0000000000000000 000000338a923120 0 MTA 18 5 1be8 00000033ad63e890 28220 Preemptive 0000000000000000:0000000000000000 000000338a923120 0 MTA <truncated> 12 71 24a4 00000033b02fa880 20220 Preemptive 0000000000000000:0000000000000000 000000338a923120 0 MTA 104 72 8cc 00000033b7c2b8b0 20220 Preemptive 0000000000000000:0000000000000000 000000338a923120 0 MTA 106 73 188c 00000033b0336fd0 20220 Preemptive 000000338DE59980:000000338DE5AFC0 000000338a923120 0 MTA System.ArgumentException 000000338de50b08
• next i dumped the Stack out

0:106> ~k

# Child-SP RetAddr Call Site

00 00000033`ab3f5ef8 00007ff7`e637d4f7 Microsoft_Office_Server_Native!ULSSendFormattedTrace [n:\src\util\ulsapi\native\trace.cpp @ 404]

01 00000033`ab3f5f00 00007ff7`e637c265 ConnectorPH!DomainBoundILStubClass.IL_STUB_PInvoke(UInt32, UInt32, Microsoft.Office.Server.Diagnostics.ULSTraceLevel, System.String, Boolean)+0x157

02 00000033`ab3f6020 00007ff7`e637b095 Microsoft_Office_Server!Microsoft.Office.Server.Diagnostics.ULS.SendTraceImpl(UInt32, Microsoft.Office.Server.Diagnostics.ULSCatBase, Microsoft.Office.Server.Diagnostics.ULSTraceLevel, System.String, System.Object[])+0x125 [n:\src\otools\inc\osrv\Diagnostics\ULS.cs @ 2334]

03 00000033`ab3f6080 00007ff7`e7392bd1 Microsoft_Office_Server!Microsoft.Office.Server.Diagnostics.ULS.SendTraceTag(UInt32, Microsoft.Office.Server.Diagnostics.ULSCatBase, Microsoft.Office.Server.Diagnostics.ULSTraceLevel, System.String, System.Object[])+0x155 [n:\src\otools\inc\osrv\Diagnostics\ULS.cs @ 2303]

04 00000033`ab3f6120 00007ff7`e737627e Microsoft_Office_Server_Search!Microsoft.Office.Server.Search.Internal.Protocols.CProtocolHelper.HandleException(System.Exception, UInt32)+0xf1 [n:\src\search\managed\lib\OM\Protocols\util.cs @ 545]

05 00000033`ab3f61b0 00007ff8`45ac59c5 Microsoft_Office_Server_Search!Microsoft.Office.Server.Search.Internal.Protocols.SharePoint2006.CSTS3Helper.GetSite(System.String, Microsoft.Office.Server.Search.Internal.Protocols.SharePoint2006.sSite3 ByRef)+0x42e [n:\src\search\managed\lib\OM\Protocols\stsv3ph.cs @ 2732]

06 00000033`ab3f6220 00007ff8`45ac58c3 clr!ExceptionTracker::CallHandler+0xc5 [f:\dd\ndp\clr\src\vm\exceptionhandling.cpp @ 3341]

07 00000033`ab3f62c0 00007ff8`45ac42af clr!ExceptionTracker::CallCatchHandler+0x7f [f:\dd\ndp\clr\src\vm\exceptionhandling.cpp @ 535]

08 00000033`ab3f6350 00007ff8`4e53347d clr!ProcessCLRException+0x2e6 [f:\dd\ndp\clr\src\vm\exceptionhandling.cpp @ 1067]

09 00000033`ab3f6430 00007ff8`4e4f5405 ntdll!RtlpExecuteHandlerForUnwind+0xd [d:\blue\minkernel\ntos\rtl\amd64\xcptmisc.asm @ 252]

0a 00000033`ab3f6460 00007ff8`45ac4348 ntdll!RtlUnwindEx+0x385 [d:\blue\minkernel\ntos\rtl\amd64\exdsptch.c @ 1006]

0b 00000033`ab3f6b40 00007ff8`45ac4304 clr!ClrUnwindEx+0x40 [f:\dd\ndp\clr\src\vm\exceptionhandling.cpp @ 4328]

0c 00000033`ab3f7060 00007ff8`4e5333fd clr!ProcessCLRException+0x2b2 [f:\dd\ndp\clr\src\vm\exceptionhandling.cpp @ 1032]

0d 00000033`ab3f7140 00007ff8`4e4f4847 ntdll!RtlpExecuteHandlerForException+0xd [d:\blue\minkernel\ntos\rtl\amd64\xcptmisc.asm @ 131]

0e 00000033`ab3f7170 00007ff8`4e53258a ntdll!RtlDispatchException+0x197 [d:\blue\minkernel\ntos\rtl\amd64\exdsptch.c @ 535]

0f 00000033`ab3f7840 00007ff8`4b6b871c ntdll!KiUserExceptionDispatch+0x3a [d:\blue\minkernel\ntos\rtl\amd64\trampoln.asm @ 715]

10 00000033`ab3f7f50 00007ff8`45ac5294 KERNELBASE!RaiseException+0x68 [d:\blue\minkernel\kernelbase\xcpt.c @ 828]

11 00000033`ab3f8030 00007ff8`45ac508e clr!RaiseTheExceptionInternalOnly+0x2fe [f:\dd\ndp\clr\src\vm\excep.cpp @ 3134]

12 00000033`ab3f8130 00007ff8`453daacf clr!IL_Throw+0x11b [f:\dd\ndp\clr\src\vm\jithelpers.cpp @ 5017]

13 00000033`ab3f8300 00007ff7`e737cdc2 mscorlib_ni!System.Collections.Generic.Dictionary`2[[System.__Canon, mscorlib],[System.__Canon, mscorlib]].Insert(System.__Canon, System.__Canon, Boolean)+0xdd274f [f:\dd\ndp\clr\src\BCL\System\Collections\Generic\Dictionary.cs @ 314]

14 00000033`ab3f8390 00007ff7`e737660e Microsoft_Office_Server_Search!Microsoft.Office.Server.Search.Internal.Protocols.SharePoint2006.CSTS3Helper.ParseGroupCollection(System.Xml.XmlReader)+0x192 [n:\src\search\managed\lib\OM\Protocols\stsv3ph.cs @ 2642]

15 00000033`ab3f8410 00007ff7`e7375ef2 Microsoft_Office_Server_Search!Microsoft.Office.Server.Search.Internal.Protocols.SharePoint2006.CSTS3Helper.InitSite(System.String)+0x2ae [n:\src\search\managed\lib\OM\Protocols\stsv3ph.cs @ 2628]

16 00000033`ab3f84b0 00007ff7`e7375af9 Microsoft_Office_Server_Search!Microsoft.Office.Server.Search.Internal.Protocols.SharePoint2006.CSTS3Helper.GetSite(System.String, Microsoft.Office.Server.Search.Internal.Protocols.SharePoint2006.sSite3 ByRef)+0xa2 [n:\src\search\managed\lib\OM\Protocols\stsv3ph.cs @ 2704]

<truncated>

• We can see our function in here...

• the line above is the hashtable\collection
• so i dumped that Stack Object out:

• this group is of interest to me:

• so i went back to my SiteCollection.xml we had collected..
• i changed the PS a bit to dump the $results object from the Previous PS

              $results | Select Id, Name | Out-GridView

• and then I filtered where Name contains "Precision"
• i see these entries here:

• those with IDs = 9972, 9973, 9974 appear to be the Culprit here. ( somehow these entries made it into the Groups table with that funky encoded value )
• so we used the following PowerShell "SiteGroups" removed:

$ids = "9972", "9973", "9974" $w = Get-SPWeb https://foo

foreach($sg in $w.SiteGroups | ?{$_.Name -match "Precision breeding for piglet survival PhD Student"}) {  foreach($id in $ids) { if($sg.Id -eq $id) { "Removing Group " + $sg.Name + " With ID: " + $sg.Id

    $w.SiteGroups.RemoveByID($id) } } }

• this should get these groups out and allow the site to be crawled..

I have seen several inquiries out here on the internet, along with a few blogs where people have experienced issues when trying to slipstream SharePoint 2013 to the latest Cumulative Update, ex: October 2013 CU. As most of you may know, in order to get to the latest build, you must install March 2013 PU as a pre-requisite before you can get October 2013, December 2013 CU applied. In theory, the slipstreaming process consists of

1. Downloading the PU\CU
2. Extracting the contents of the PU\CU
3. Move them into the “updates” folder in your SP installation folder and GO

The problem we have seen with some of our customers in SP 2013, is they extract March PU, October 2013 CU along with the contents of the “ubersrv_1.cab”, and then copy these into the “updates” folder. They then install SharePoint with the expectation that when the farm builds out, it would reflect the build of the latest update ( ex: 4551.1000 for October ) but in reality it shows the farm build to be RTM still.

Note: I am going to stop right here for a minute and advise that if you want to install a base install with the latest builds, I would advise to go ahead and download a slipstreamed build of “SP 2013 with SP1” ( from MSDN ) OR download SP1, extract its contents ( using “officeserversp2013-kb2817429-fullfile-x64-en-us.exe /extract:C:\SP1_extract” ) and just build your slipstream this way. SP1 includes all of the PU\CU updates since the release of RTM up through December 2013 CU & January 2014 PU

If you decide you are not ready to do this and still want to slipstream March 2013 PU + “subsequent CU”, then I am going to outline how you can successfully do this. I would also like to preface this with the fact that this is primarily applicable if you are slipstreaming March PU + “subsequent CU” with the English language only.

Here we go! ( I am using October 2013 CU as my “subsequent CU” in my steps here )

I want to provide a little background on WHY just extracting and putting into “updates” folders will not work. Since March 2013 PU is a pre-requisite to installing CU’s, SP must know that March PU has been applied before it can apply the October 2013 CU. This is known as a new baseline called “SP0” How does it know its applied when this is a brand new install? It doesn’t, so it does not apply the updates at all and this is why you see the farm build out at RTM still. In order to get these updates to “apply” we have to do a little manipulation to the .msp files to “trick” SP into applying the March 2013 PU updates and then the later CU .msp will be applied.

Start By Downloading the following:

March 2013 Public Update: 

http://support.microsoft.com/kb/2767999

September 2013 Update:

 http://support.microsoft.com/kb/2810081

October 2013 Cumulative Update:

http://support.microsoft.com/hotfix/KBHotfix.aspx?kbnum=2825647

·         Once you have these downloaded and saved to C:\Hotfix\<UpdateDir>, you will want to extract each one..

·         Starting with March 2013 CU, I run the following command to extract the msp’s from the .exe

c:\Hotfix\March 2013 PU>ubersrvsp2013-kb2767999-fullfile-x64-glb.exe /extract:"C:\Hotfix\March 2013 PU\extracted"

·         Like I mentioned in a previous email, in order to get to the October 2013 CU or later builds, like October or December, the server needs to think it has a new baseline, what we refer to as SP0, before it will apply those binaries during the install.

·         In order to trick the server into thinking this, we have to do some manipulation to the March 2013 PU msp’s and rename them before putting them into the “updates” folder..

·         Once you have extracted them, there should be 16 of these files, we will want to rename them to have a “_” as a prefix and a “-SP0” as a suffix

·         I used a tool called “Bulk Rename” to do this, but this is what the file names should look like

_acsrvwfe-x-none-SP0.msp

_coreserver-x-none-SP0.msp

_eduwfe-x-none-SP0.msp

_ifswfe-x-none-SP0.msp

_lpsrvwfe-x-none-SP0.msp

_oserver-x-none-SP0.msp

_osfserver-x-none-SP0.msp

_ppsmawfe-x-none-SP0.msp

_pptserver-x-none-SP0.msp

_sms-x-none-SP0.msp

_sts-x-none-SP0.msp

_vsrvwfe-x-none-SP0.msp

_wasrvwfe-x-none-SP0.msp

_wdsrv-x-none-SP0.msp

_wss-x-none-SP0.msp

_xlsrvwfe-x-none-SP0.msp

·         Once you have renamed these, you can now place them into your “updates” folder ( C:\Apps\SP2013\October4551\Updates )

·         Next extract the September 2013 Update

c:\Hotfix\September 2013 Update>svrproofloc2013-kb2760250-fullfile-x64-glb.exe /extract:"C:\Hotfix\September 2013 Update\extracted”

·         Since this is English Only, we will copy the following file and place it into the “updates” folder”

svrproof-en-us.msp

·         Now lets move to the October 2013 CU

·         Keep in mind this patch has the “ubersrv2013-kb2825647-fullfile-x64-glb.exe” but it also has a “.com” file and the “ubersrv_1.cab” file…

·         We will want to extract the files from the “.exe” and the “.cab” file..

c:\Hotfix\October 2013-4551.1005>ubersrv2013-kb2825647-fullfile-x64-glb.exe /extract:"C:\Hotfix\October 2013-4551.1005\extracted"

·         To extract the .cab files, just go into the cab file, select All and then extract to "C:\Hotfix\October 2013-4551.1005\cab_extracted”

·         Once you have extracted all of these files, and since this is English Only, we want to go find the files ending in “en-us.msp” AND “x-none.msp”

From the “C:\Hotfix\October 2013-4551.1005\extracted”

acsrvmui-en-us.msp

coreservermui-en-us.msp

edumui-en-us.msp

ifsmui-en-us.msp

lpsrvmui-en-us.msp

osfservermui-en-us.msp

ppsmamui-en-us.msp

pptservermui-en-us.msp

smsmui-en-us.msp

acsrvwfe-x-none.msp

coreserver-x-none.msp

eduwfe-x-none.msp

ifswfe-x-none.msp

lpsrvwfe-x-none.msp

osfserver-x-none.msp

ppsmawfe-x-none.msp

pptserver-x-none.msp

sms-x-none.msp

sts-x-none.msp

vsrvwfe-x-none.msp

wasrvwfe-x-none.msp

wdsrv-x-none.msp

wss-x-none.msp

xlsrvwfe-x-none.msp

From the “C:\Hotfix\October 2013-4551.1005\cab_extracted”

acsrvmui-en-us.msp

wssmui-en-us.msp

pptserver-x-none.msp

sts-x-none.msp

wdsrv-x-none.msp

·         Put these into the “updates” folder

·         At this point, you should have 46 files in the folder.

You should be able to install SP at this point and it reflect the correct build when the farm is built. Setting this baseline by renaming the Marm 2013 PU msp’s is the key point here in getting this to work correctly. Our source code knows that the files with “_” should be installed first and then we can apply the later CU’s. This takes a bit of work to get this done, but it can be done. Again, if you want to just download SP1, you should be able to extract these files and put them into “updates” or download a pre-built SP1 install.

Hope this is helpful!

I worked with a customer a while back that ran into a scenario where they were using a standard “Search Service Application” within the SharePoint 2010 and they were crawling a non-SharePoint, Apache based wiki site, called “MediaWiki.” MediaWiki will allow you to create a new page with pretty much any name you would like.

Example:

http://www.contoso.com/mediawiki/wiki.test.page

When their content source would go out and crawl this site, it would not crawl and index this new page, so it was not showing up in search results.

By default, within the SSA (Search Service Application) Administration page we have a link called “File Types”, and this page controls what File Extensions we will and will not crawl.

By Default, within a SharePoint SSA, NOT FAST 4 SharePoint 2010, this “File Types” list is an “Include” list, so anything NOT in this list will not get crawled. Here in lies the problem for my customer. Being that end users had the capability to create a page with any name, the “extensions” could vary from anything. Trying to control what extensions will need to be in this list would be nearly impossible and require a lot of man hours to even try to maintain this!

Here is where my customer turned to Microsoft support, to look for a solution or a potential work around. The solution we provided the customer was to utilize PowerShell and manually “flip” this “Include” list to an “Exclude” list, so that the SSA will crawl any extension, except for what is now in the “File Types” list.

Here are the steps to accomplish this task:

Flip the List to an Exclude List

• Open PowerShell Management Shell for SharePoint with elevated rights, or if you have PowerShell ISE, you can use this also
• Execute the following:

$sa = Get-SPServiceApplication | where { $_.ApplicationClassId -eq "52547a3d-66ed-468e-b00a-8c4a3ec7d404" }

$sa.SetIsExtensionIncludeList($sa.GetVersion(),0);

• This will flip the SSA to make it an Exclude list
• Restart OSearch14 via services.msc console or with command line

net stop OSearch14

net start OSearch14

Keep in mind now that this list is now an “Exclude” so anything in here will no longer be crawled..

We then utilized additional PowerShell to clean out all the existing “File Types” and allowed the customer to start with a clean slate:

Remove the existing file types

• Again using PowerShell Management Shell or PowerShell ISE
• Execute the Following: (make sure you replace the “SSA” with the name of your Search Service Application)

$ssa = Get-SPEnterpriseSearchServiceApplication -Identity "SSA" $content = New-Object Microsoft.Office.Server.Search.Administration.Content($ssa)

$extList = $content.ExtensionList

$list = New-Object System.Collections.ArrayList

foreach ($ext in $extList)

{

$list.Add($ext);

}

for ($i = 0; $i -lt $list.Count; $i++)

{

$ext = $list[$i]

$ext.FileExtension

$ext.Delete()

}

Below is a list of Extensions that FAST includes OOB.. Feel free to add these in as needed.. I just wanted to give you a list of extensions that MS uses as “excludes”

I have tested this within our lab, using a clean slate, and it worked. Please let me know if you have any questions. Thanks!

I believe this should transfer over to SP 2013 also…

' THIS CODE AND INFORMATION IS PROVIDED "AS IS" WITHOUT WARRANTY OF

' ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING BUT NOT LIMITED TO

' THE IMPLIED WARRANTIES OF MERCHANTABILITY AND/OR FITNESS FOR A

' PARTICULAR PURPOSE.

' Copyright (C) Microsoft Corporation, 2013

The issue at hand is when directly linking Office Documents within your site, even though anonymous access is enabled, you are still prompted for credentials and if you cancel, an error is received:

401 Unauthorized

Ideally, as an anonymous user, you should be able to open this document, without being prompted for credentials and without getting a “401 Unauthorized”

Initially this was a common problem when direct links were added to .xlsx files.  Findings show when a Document library containing .xlsx and .xls files are directly linked to URL’s, the .xlsx fails and the .xls succeeds.

We are now seeing similar behavior in SharePoint 2010 where Office Web Apps have been installed on the SharePoint servers.

So… Why do we get this error in the first place…?

• In SharePoint 2007 and 2010, “.xlsx” extensions are mapped to the “xlviewer.aspx” page, via a file called, “serverfilesExcelServer.xml”
• This page actually requires an additional permission, “OpenItems”, in order for it to successfully render a “.xlsx” file.
• In SharePoint 2010, when you install Office Web Apps, we add additional file handlers that now map other extensions, like .doc and .docx, to pages, like “wordviewer.aspx”.
• These pages also require the “OpenItems” permission that anonymous users do NOT have.
• By default, Anonymous Users do NOT have this “OpenItems” permission and this is what causes the failure to open or render the files

Note: There have been numerous postings on the web advising simply remove the mapping within the above mentioned XML file, but this will put you into an unsupported state, so it is recommended NOT to do this.

If you were to look at the ULS logs while trying to access a directly linked document, you will see something like this:

09/19/2011 16:20:59.87   w3wp.exe (0x3164)             0x3DF4  SharePoint Foundation        General   ewn8      Verbose                OriginalPermissionMask check failed. asking for 0x00000021, have 0x1000031041             f22f656c-a188-4a4c-9b41-8deeb6b934bd

And this is what the Permission Mask(s) translates to:

OriginalPermissionMask check failed. asking for 0x00000021, have 0x1000031041

0x00000021 = ViewListItems, OpenItems

0x1000031041 =  ViewListItems, ViewVersions, ViewFormPages, Open, ViewPages, UseClientIntegration

As we can see, we are asking for “OpenItems” and we do not have it…

KB2498047 was published to address the “.xlsx” extension, but it will also address additional Office Doc extensions. There are 3 possible work arounds, but I am going to focus on the solution provided that utilizes PowerShell to add the “OpenItems” permission to our SPBasePermissions for anonymous users. This PowerShell will assist in adding “OpenItems” to a single web….

So what if I have a Site Collection and this site collection contains 2000 + webs?

What if some of these webs inherit permissions from the parent web, but a lot of them actually contain unique permissions? (They are still anonymous access webs, they just contain a Unique Permission Scope)

Do I need to execute this PowerShell for each web in my Site Collection…?

If your Site Collection had 2000+ webs and they all inherited from the parent web, then you could execute the PowerShell at the root Site Collection, or Top-Level Site and be done. But since you may have 2000+ webs that have unique permission scopes, we would need to iterate through the entire Web Application and set this “OpenItems” permission on our AnonymousPermMask64.

We put together a PowerShell script that would do this for you as well as the script to undo that permission add:

NOTE: Giving anonymous "OpenItems" permission allows anonymous users to have view source rights to certain files that could potentially contain sensitive info.

NOTE*: That you should only do this if you understand & accept the security implications.

To Set “OpenItems” across an entire Web Application:

·         Once the ps1 is saved, pass the URL as the $arg, Ex:

.\AnonPermMask64_OpenItems_Add.ps1 “http://www.somesite.com”

·         Code:

Add-PSSnapin Microsoft.SharePoint.PowerShell

 

[void][System.Reflection.Assembly]::LoadWithPartialName("Microsoft.SharePoint")

 

 

    $siteUrl = Get-SPWebApplication $args[0] | Get-SPSite -limit All;

 

    foreach($url in $siteUrl)

   

    {

        write-host $url.url;

        $site = New-Object Microsoft.SharePoint.SPSite($url.url);

       

        foreach($webs in $site.AllWebs)

        {

           

            $enumPerms = [Microsoft.SharePoint.SPBasePermissions];

 

           

            if ($webs.HasUniquePerm)

            {

           

            Write-Host 'Setting Attribute for '$webs.url;

           

            $webs.AnonymousPermMask64 = $webs.AnonymousPermMask64 -bor $enumPerms::OpenItems

            $webs.Update();

           

            Write-Host 'Attribute for '$webs.url' has been set';

           

            } else

           

            {

           

            Write-Host 'Setting Attribute for '$webs.url;

            Write-Host 'Site Inherits Permission from Parent'

           

            }

        }

       

        $site.Dispose();

     } 

       ##end of the for each loop

 

write-host 'Finished';

To Remove The “OpenItems” across the Web Application:

·         Code:

Add-PSSnapin Microsoft.SharePoint.PowerShell

[void][System.Reflection.Assembly]::LoadWithPartialName("Microsoft.SharePoint")

    $siteUrl = Get-SPWebApplication $args[0] | Get-SPSite -limit All;

    foreach($url in $siteUrl)

    {

        write-host $url.url;

        $site = New-Object Microsoft.SharePoint.SPSite($url.url);

        foreach($webs in $site.AllWebs)

        {

            $enumPerms = [Microsoft.SharePoint.SPBasePermissions];

            if ($webs.HasUniquePerm)

            {

            Write-Host 'Setting Attribute for '$webs.url;

            $webs.AnonymousPermMask64 = "ViewListItems, ViewVersions, ViewFormPages, Open, ViewPages, UseClientIntegration"

            $webs.Update();

            Write-Host 'Attribute for '$webs.url' has been set';

            } else

            {

            Write-Host 'Setting Attribute for '$webs.url 'back to default BasePermissions';

            Write-Host 'Site Inherits Permission from Parent'

            }

        }

        $site.Dispose();

     } 

       ##end of the for each loop

write-host 'Finished';

I hope you found this helpful. There is some information not covered here in order to keep it concise. If you have any questions, please feel free to post and I will address them as soon as possible. Thanks!

Consider the following Scenario:

• We have a user that has an account in Active Directory with account name = DOMAIN\jdoe
• This user has a SID = S-1-5-21-1461310-839809092-932994037-4144
• User, DOMAIN\jdoe, has been granted permissions to some 300 site collections
• Its determined that this user account is having some permission issues within the Domain so the AD account is deleted. :)
• The account is now re-created again within AD and given the same account name, DOMAIN\jdoe
• Except now a new SID is created within AD
• SID = S-1-5-21-1461310-839809092-932994037-4147
• The problem we now have is that because this user already had permissions on some 300 sites that are associated with the Old SID
• The account name may be the same but when this user tries to Access Any of the Sites, there is a SID mismatch and the user now gets an Access Denied:

• The UserInfo table is the table on Each Content Database that holds the users login information as well as the SID. These are stored in the following two columns within the UserInfo Table

LoginName

tp_systemID

So the question is….. How do we get the new SID from AD into the UserInfo table. While this can be done with creating a  temporary account in AD and doing some  "stsadm -o migrateuser" flipping from Temp Account to Valid Account and vise versa,  I have found that we can achieve this task by passing the same value for the "-oldlogin" & "-newlogin" and setting the "-ignoresidhistory" switch on our "stsadm -o migrateuser" command. So this is basically what you would do:

stsadm –o migrateuser –oldlogin DOMAIN\jdoe –newlogin DOMAIN\jdoe –ignoresidhistory

What this should do is flip the SID, or "tp_SystemId" in the UserInfo table to be the new account SID from AD and your user should now have access to all 300+ sites again.

Happy Migrating!!

I have seen a couple of cases where then end users where having an issue with the "Related Sites" search scope within a SharePoint Portal Site. By default, when you create a Collaboration Portal, the following 3 Search Scopes show up when you first load the page:

This Site: Home
All Sites
People  

These are the 3 main scopes that get pulled in from the SSP within a SharePoint farm. "This Site" would be considered a Contextual Scope and is specific to searching for content only within the context of that site. If you were to browse to the "Related Links scope settings" within a site and add an addtional search scope, based on another URL, like http://microsoft.com, and then re-compile your scopes manually or let the SSP scope compilation complete automatically (every 15 minutes by default). When you go back to the home page your site, you should see a "Related Sites" search scope in the drop down. This is all good if you intend to utilize this search scope, but the issue comes to surface when you decide that you no longer want to have any "Related Sites" search scopes within that site.  So lets say you perform the following actions to remove the related links:

• Browse to the site you want to remove the Related Links from via, http://fqdn.domain.com/_layouts/RelLinksScopeSettings.aspx
• Click on the Selected Links you want to remove ( all in this case) and for each one, you click on "Remove"
• You click on OK and suspect all is good.
• Wait for the Scopes to compile on their own or manually force the compilation via http://servername:6000/ssp/admin/_layouts/searchsspsettings.aspx and then click on "Update Scopes"
• reset IIS to clear the cache
• Browse back to the site and you notice the "Related Sites" is still in the drop down.

At this point you would expect the "Related Sites" to disappear, so you browse back to the /_layouts/RelLinksScopeSettings.aspx page and see the words "emptytext" in the "Available Links"section of the page. This gives the impression that a scope exists in the site so this is why the "Related Sites" scope is still present in the search drop down.

So you ask, how can I get rid of this at this time? It is not really causing any real problem other than maybe some confusion for the end users who are trying to search for information within this site. While there is not a direct way through the UI to remove this from the search scopes, I can provide you some sample Object Model code that you or a developer can compile and run against the site URL and get that to disappear from the drop down. Again this is Sample Code and has no warranty and is not supported by Microsoft. This code will basically identify the WebID within the content database for the site URL you want to remove the "related sites" link from and then correleate that GUID with the Scopes in the SSP DB and delete the item from the database. After this is completed, it is necessary to go ahead and update the scopes and then do an IISReset in order to flush the compiled pages from memory. Below is the Sample Code you can use to remove the "Related Sites" scope from a site"

 =======================================================================

Sample Code

·         We basically created a Form Box for the GUI 

·         Then compile the code below:

using System;

using System.Collections.Generic;

using System.ComponentModel;

using System.Data;

using System.Drawing;

using System.Linq;

using System.Text;

using System.Windows.Forms;

using System.Collections;

using Microsoft.SharePoint;

using Microsoft.Office.Server;

using Microsoft.Office.Server.Administration;

using Microsoft.Office.Server.Search;

using Microsoft.Office.Server.Search.Administration;

namespace scopedelete

{

     public partial class Form1 : Form

    {

        public Form1()

        {

            InitializeComponent();

        }

         private void button1_Click(object sender, EventArgs e)

        {

            if (textBox1.Text.Trim() == "")

            {

                MessageBox.Show("Please enter the root web URL.");

            }

            else

            {

                SPSite site = new SPSite(textBox1.Text);

                SPWeb myweb = site.OpenWeb();

                Guid mywebid = myweb.ID;

              //  MessageBox.Show("webid : " + mywebid.ToString());

                Uri siteuri = new Uri(site.Url);

                Microsoft.Office.Server.ServerContext myservercontext = Microsoft.Office.Server.ServerContext.GetContext("SSP");

                SearchContext scontext = SearchContext.GetContext(myservercontext);

                Scopes sspscopes = new Scopes(scontext);

                ScopeCollection allscopes = sspscopes.AllScopes;

                Scope myscope = null;

                Scope MydeleteScope = null;

                IEnumerator myscopeenum = sspscopes.AllScopes.GetEnumerator();              

                for (int i = 0; i < allscopes.Count; i++)

                {

                    while (myscopeenum.MoveNext())

                    {

                        myscope = (Scope)myscopeenum.Current;

                      //  MessageBox.Show(myscope.Name); 

                        if (myscope.Name == mywebid.ToString())

                        {

                            MydeleteScope = myscope;

                            break;

                        }                       

                    }

                }

                if (MydeleteScope == null)

                {

                    MessageBox.Show("Related sites scope for site " + myweb.Name + "(" + textBox1.Text + " ) does not exist.");

                }

                else

                {

                    MydeleteScope.Delete();

                    MessageBox.Show("Related sites scope for site " + myweb.Name + "( " + textBox1.Text + " ) deleted successfully");

                }

            }

        }

        private void Form1_Load(object sender, EventArgs e)

        { 

        } 

     }

   }

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

   ** Please note that where I have highlighted “SSP” you will need to replace that value with the name of your SSP 

' THIS CODE AND INFORMATION IS PROVIDED "AS IS" WITHOUT WARRANTY OF

' ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING BUT NOT LIMITED TO

' THE IMPLIED WARRANTIES OF MERCHANTABILITY AND/OR FITNESS FOR A

' PARTICULAR PURPOSE.

' Please stress test this code before deploying it in a production environment.
' This is intended as a sample of how code might be written for a
' similar purpose. This code has not been tested.  This code is
' also not to be considered best practices or prescriptive guidance. 
 
' It is highly recommended that you FULLY understand what this code is doing
' and use this code at your own risk.

' Copyright (C) Microsoft Corporation, 2009 

·         Once this is compiled and executed browse to your SSP Admin Site

·         Click on Search Settings:

·         Find the Scopes Section and click on “Start Update Now”

·         Once that has been updated go into your WFE server(s) and recycle the application pools for the Web App you just executed this against.

After the page has been rendered again, you should see that the “Related Sites” option is no longer present in the Search Drop Down Box

=======================================================================  

To confirm that the scope has been deleted, you can open up SQL Server Management Studio and then run the following queries to see that the GUID for that scope has been removed from the SSP Database.

• First you would need to get the WebId from the content DB that you ran the code against
• Open SQL Server Management Studio and create a New Query
• In the drop down, next to the "execute" function in the toolbar, make sure you have the Content Database select
• Then run the following:

• Identify the Id where root of that site exists.
• Copy this GUID and put it into notepad
• Then create a new query and this time make sure the SSP DB is selected, not the Search DB, but the SSP DB
• Then run the following:

          select * from MSSScopes

• If the GUID is still present in that table, then you have not successfully removed the scope. If it is no longer present, then the "Related Sites" scope should no longer be there when the Scopes are Updated and IIS has been reset.

 I hope this helps in the event you want to get rid of that "Related Sites" scope within your Search Drop Down!

I recently worked a case where my customer was reporting that at random, 4 columns were just disappearing from their Form Library. At first we were not able to indentify any particular event, rhyme or reason as to why these same four columns would just magically go away, until one day they were asked to create a list template of this Form Library and send it to us. After this task was done, they noticed the 4 columns were now gone. I now had the "ListTemplate.stp" file in my possession and thought that since the columns were present at the time of the list template creation that if I created a new form library based on this list template, the columns should be present in my newly created list, however; when the new library was created from this list template, these 4 columns were not there. If you are not familiar with a "listtemplate.stp" file, you can actually rename the file from a .stp to a .cab file and then extract the contents of the file. Within any .stp file, you will find a manifest.xml file that has all the definitions of the contents of that library. In my case, I decided to dig into that manifest.xml file. When I looked at the file, I saw all the attributes of the columns that should be in my library, but were not present when I created a list from this template. I thought this was rather bizarre until I continued digging into this manifest.xml file and found a reference to a "properties.xfp" file. I decided to look into this file and I opened my form library with SharePoint Designer. I opened this properties.xfp file and discovered that the exact columns that were appearing in my library after saving the list as a template were the columns being defined within this "xfp" file.

The "xfp" file is a file that gets created when you publish a form with InfoPath 2003 and promote fields. InfoPath makes a call to the "owssvr.dll" and pushes the file through commands via that dll. Come to find out that my customer did originally create and publish the form within InfoPath 2003 and then later down the line, upgraded to InfoPath 2007 and promoted addtional fields and then re-published the form from an InfoPath 2007 client. InfoPath 2007 handles promoted fields and column creation a bit different than InfoPath 2003 client does. It simply makes web service calls through an asmx page that checks to see if columns exist or not and then creates/deletes columns based on the web service calls to the sharepoint library. InfoPath 2007 never touches or updates this "properties.xfp" file. So new fields within InfoPath 2007 that had been created and published will go out and create the new columns. The problem you run into when you originally publish a form from InfoPath 2003 and and then you come behind that work and re-publish and promote addtional fields with an InfoPath 2007 client, is that sharepoint will still look at this "xfp" file as a field definition solution and update its list's columns based on these fields within this file. Evidently when you save a list as a template, this is an event that triggers sharepoint to read this file and redefine it's list columns per this "xfp" file.

So you ask... 

How Can I fix this? Do I have to keep track of every promoted field and update this "xfp" file...?

To answer that, we need to ask a couple questions as well....

Will InfoPath 2003 client continue to be used to publish forms or will it strictly be InfoPath 2007 client publishing forms?

- If InfoPath 2007 will the only client used from this point on, you can simply browse to your forms library and delete the "properties.xfp" file. This will have no negative ramifications on your current list. To delete this file, you can do it with Windows Explorer View so that you do not have to install SharePoint Designer.

Actions:

1.       Browse to the Library that contains this properites.xfp file.

2.       From the “Action” menu, select Open in Windows Explorer

3.       Once the window opens, browse into the “forms” folder

4.       Locate the properties.xfp file

5.       Delete it.

6.       Close window and then browse back to the list settings

7.       Now.. you can save the List as a Template

8.       Once this is saved you can browse back to the list and should see the columns still intact.

9.       Also create a new list based off this template to confirm that all columns are still intact.

- If InfoPath 2003 is still used in your environment then my recommendation would be to continue publishing the form with InfoPath 2003 until you are ready to make the change to InfoPath 2007

There is still a bit of investigation from a SharePoint Admin standpoint that I want to do but I wanted to share this for now in the event that any one experiences "columns" randomly disappearing from their sharepoint library and how you may want to look at the "properties.xfp" to see if this file exists in your library..

Chances are this is your culprit....

My name is Anthony Casillas and I am a Sr. Support Escalation Engineer at Microsoft supporting SharePoint Server.  I officially started in SharePoint on October 1, 2007.  I've been with Microsoft 5.5 years now and will continue immersing myself in this product and blogging on random SharePoint topics that I feel will benefit anyone outside MSFT. 

 My posts will be varied in technical depth and totally random.  My reasoning for doing this is to help any reader whether they are new to the product or have extensive experience...

Anthony