Limitations

Lets assume that based on some of the data that is being changed you’d like to call back to SharePoint to get some other data in another list to do some processing.  Or, lets assume you have added say… 20 columns to your list and you are comparing some of them to each other, if the user decided to use the Quick Edit in SharePoint you’d only receive the columns that were actually changed in AfterProperties, so now you’re a bit stuck because if you need to compare the changed data against other data in the list item that was not changed, you’d have to find some way to call back to SharePoint.

The call back to SharePoint would typically be done through a trusted connection in the context of the SharePoint user or an app… and for that you do indeed need an app.  Consider the following code that executes in a RER.

ClientContext clientContext = TokenHelper.CreateRemoteEventReceiverClientContext(properties);

clientContext, without an app in place will always return as null.

I’ll digress a   bit. This is the context that would typically be used to talk back to SharePoint.  If on prem and using high trust with a PHA, the code in the TokenHelper class grabs the path to your pfx file and the pfx password from your web.config, creates a windows identity X509 signing cred using claims passed from SharePoint, then calls back to SharePoint using the cred on behalf of the SharePoint user.   Whew!, and that’s paraphrased, there’s plenty of documentation out there about how this works.  If o365, an ACS access token is used to create the trusted connection, once again, beyond the scope of this post.  The bottom line is that if we don’t have an app then calling back to SharePoint would require creating a cred using through some other means like storing a user name and password, a service account, etc. etc..

Some other pretty cool things you can do without a PHA.

JavaScript Injection – this pattern is a great way for you to make changes to master pages, page layouts, any page that renders in your Site Collection for that matter, without the need for uploading custom versions.  You only need to run a bit of server side CSOM through a console app, windows forms app, web application, etc.  A custom action is added that tells SharePoint to add your js file as a script link, all you need is a web site to put your file.  That’s it, once added the js file will load each time a SharePoint page loads and “Inject” your mods dynamically.  No PHA required.  Check it out in GitHub.  There’s also a sample on how to use this pattern through JSOM here.

I was going to add more but really there are too many to list.  Check out all of the OfficePNP samples in GitHub here, you’ll notice that there are a plethora of things you can do to customize your SharePoint site that you may have been thought required a PHA.

1.      Create or use an existing SharePoint site, add a list, add some columns to the list, etc., etc.  We need a list in a SharePoint site for this example, use whatever one you want.

2.      In Visual Studio, create an “ASP.Net Empty Web Application” called RERWithoutPHA.

3.      Right click on the RERWithoutPHA project, then choose Manage “NuGet Packages…”.  In the search box enter “pnp”.  Choose on-prem or cloud and install.

(The PNP NuGet package is awesome, it scaffolds your project with all you’ll need to access SharePoint from your remote web server (or any application for that matter) and includes the Tokenhelper and SharePointContext classes, so if you were creating a PHA this is all you’d need to do to get rolling.  But… in all truth, for this example all we need is a reference to Microsoft.SharePoint.Client.dll and Microsoft.SharePoint.Client.Runtime.dll)

4.    Add a “WCF Service” to the RERWithoutPHA called RERSynchronous.  Edit the RERSynchronous service code behind then implement “IRemoteEventService”.

using Microsoft.SharePoint.Client.EventReceivers;

namespace RERWithoutPHA

{

    public class RERSynchronous : IRemoteEventService

    {
        public SPRemoteEventResult ProcessEvent(SPRemoteEventProperties properties)

        {
            SPRemoteEventResult result = new SPRemoteEventResult();

            // if this is a document library and this item is being updated, have a look at the

            //value of the fields before they were changed.

            // the BeforeProperties collection is only available for document libraries, it will be null for 
            // lists

            string myFieldBefore = properties.ItemEventProperties.BeforeProperties["myField"].ToString();

            // Have a look at the changed fields.  Of Course you'll have to change "myField" to the name of a
            // field in your list or document library

            string myFieldAfter = properties.ItemEventProperties.AfterProperties["myField"].ToString();

            // Maybe you'd like to change the value of a field in your list item

            result.ChangedItemProperties.Add("fieldIWantToChange", "My New value");

            //  maybe you don't like what's been done and want to tell SharePoint to cancel the save

            // comment in the lines below to see the save cancel

            //result.ErrorMessage = "No can do!";

            //result.Status = SPRemoteEventServiceStatus.CancelWithError;

            //YOU GET THE PICTURE
           
            return result;

        }
        public void ProcessOneWayEvent(SPRemoteEventProperties properties)

        {

            //this event will not fire unless Asynchronous methods have been registered

            //but must be here to satisfy the IRemoteEventService interface

        }

    }

}

5.       Add a console application to the solution called RERWithoutPHADeploy.

6.       Add the “OfficeDevPnP.Core” NuGet reference to the RERWithoutPHADeploy as well. (Once again, you really only need a reference to Microsoft.SharePoint.Client.dll and Microsoft.SharePoint.Client.Runtime.dll).

7.       Add the following code to your “Main” Method in the console app.  Be sure to add
using Microsoft.SharePoint.Client
above your namespace declaration.

Also, I wrote this on one of my VM’s so the logged on user had rights to run the code, if you’re deploying to o365 you’ll need to create an instance of SharePointOnlineCredentials and set it to the clientContext.Credentials, if on-prem using NTLM but not the user you’re logged in as then use NetworkCredentials. 

This code verbose for example purposes, it is more efficient in the supplied code sample.

string sharePointUrl = "<the url for your SharePoint site>";

string remoteWebUrl = "<the url for your web site where the RER is hosted>";

string listName = "<The name of the SharePoint list>";

// If you are authenticating onPrem as the user you’re in as then this will work fine, else
// you’ll need to pass an instantiation of a class that implements ICredentials to the clientContext.Credentials property

ClientContext clientContext = newClientContext(sharePointUrl);

List targetList = clientContext.Web.Lists.GetByTitle(listName);

clientContext.Load(targetList);

EventReceiverDefinitionCollection ec = targetList.EventReceivers;

clientContext.Load(ec);

EventReceiverDefinitionCreationInformation eventReceiver = newEventReceiverDefinitionCreationInformation();

eventReceiver.EventType = EventReceiverType.ItemUpdating;

// The ReceiverAssembly and ReceiverClass properties, as far as I can tell, are not needed, at
// least for this example, the RER gets deployed and functions properly with or without them, you

// may choose to use them for completeness

// they are, as far as I can tell holdovers from FTC Event Receivers, I could be wrong, don’t shoot!

//eventReceiver.ReceiverAssembly = "RERWithoutPHA";

//eventReceiver.ReceiverClass = "RERWithoutPHA.Services.RERSynchronous";

eventReceiver.ReceiverName = "RERSynchronous";

eventReceiver.ReceiverUrl = remoteWebUrl + "/RERSynchronous.svc";

eventReceiver.SequenceNumber = 1000;

eventReceiver.Synchronization = EventReceiverSynchronization.Synchronous;

targetList.EventReceivers.Add(eventReceiver);

eventReceiver = new EventReceiverDefinitionCreationInformation();

eventReceiver.EventType = EventReceiverType.ItemAdding;

eventReceiver.ReceiverName = "RERSynchronous";

eventReceiver.ReceiverUrl = remoteWebUrl + "/RERSynchronous.svc";

eventReceiver.SequenceNumber = 1000;

eventReceiver.Synchronization = EventReceiverSynchronization.Synchronous;

targetList.EventReceivers.Add(eventReceiver);

eventReceiver = newEventReceiverDefinitionCreationInformation();

eventReceiver.EventType = EventReceiverType.ItemDeleting;

eventReceiver.ReceiverName = "RERSynchronous";

eventReceiver.ReceiverUrl = remoteWebUrl + " /RERSynchronous.svc";

eventReceiver.SequenceNumber = 1000;

eventReceiver.Synchronization = EventReceiverSynchronization.Synchronous;
targetList.EventReceivers.Add(eventReceiver);

clientContext.Web.Context.ExecuteQuery();

8.   Run the console app to deploy the RER’s to SharePoint

9.    Ensure that your web site is up and running (preferably for testing in debug so you can see the event happening in code).

10.  Add/edit/delete items from your list and watch the sparks fly, no PHA required!

That’s it!  Please do have a look at the code sample in GitHub, the code’s a bit nicer.  But... provided as is.

If you disable SSL 3.0 on your remote web server and you are using remote event receivers then you must also disable SSL 3.0 on your SharePoint WFE’s.

The long answer…

This is a complicated one with a lot of moving parts and I’m still not sure about the root cause.  There are other reasons for this error that I’ve researched, but this is my tale.

There are some prerequisites that must be in place in order to see this behavior

1)       You're running on prem (or possibly a cloud VM that has not had it's SCHANNEL settings changed from the default) and the SharePoint WFE is running on a different machine than is your Provider Hosted App.

2)      You are using Remote Event Receivers

Recently, the project I’m working on underwent an information security review, one of the issues raised was that SSL 3.0 was enabled on our remote web server.  SSL 3.0 is prone to the POODLE attack, a pretty tough hack to throw but a vulnerability nonetheless. Now in the environment that we were using for the review we had no control over the SharePoint farm but we did have control over the remote web server where the Provider Hosted App runs.  Ok, no big deal, we’ll just shut off SSL 3.0 on our web server and be done with it.  There are plenty of posts detailing the process for enabling and disabling different SCHANNEL protocols, this is one.

Everything seemed to be working well, the app itself was fine, pages rendering and data through the high trust connection was retrieved with no problem.  But then I noticed that our Remote Event Receivers (RERs) weren't firing, we do some data entry validation in our RERs and I was not seeing the validation error messages that it sends back to SharePoint.  So a colleague started looking though the ULS and found this

App token requested from appredirect.aspx for site: 11111111-1111-1111-1111-111111111111 but there was an error in generating it.
This may be a case when we do not need a token or when the app principal was not properly set up.
LaunchUrl:https://app-myapp.mydomain.net/Pages/DataEntry.aspx?SPHostUrl=https://myapp.mydomain.net&SPLanguage=en-US&SPClientTag=0&SPProductNumber=15.0.4631.1000&SPAppWebUrl=https://App-11111111111111.mydomainpphost.net/myapp&SPListItemId=1&SPListId={11111111-1111-1111-1111-111111111111}
Exception Message:The Azure Access Control service is unavailable.
Stacktrace:    at Microsoft.SharePoint.ApplicationServices.SPApplicationContext.GetApplicationSecurityTokenServicesUri(SPServiceContext serviceContext)
at Microsoft.SharePoint.ApplicationServices.SPApplicationContext..ctor(SPServiceContext serviceContext, SPIdentityContext userIdentity, OAuth2EndpointIdentity applicationEndPoint)    
at Microsoft.SharePoint.SPSecurityContext.SecurityTokenForApplicationContext(SPIdentityContext userIdentityContext, String applicationId, Uri applicationRealm, SPApplicationContextAccessTokenType applicationTokenType, SPApplicationDelegationConsentType consentValue)    
at Microsoft.SharePoint.SPServerToAppServerAccessTokenManager.GetAccessTokenPrivate(SPServiceContext serviceContext, String appId, Uri appEndpointUrl, SPAppPrincipalInfo appPrincipal, SPApplicationContextAccessTokenType tokenType, Boolean useThreadIdentity, SPUserToken userToken)    
at Microsoft.SharePoint.SPServerToAppServerAccessTokenManager.GetAccessTokenFromThreadIdentityOrUserToken(SPServiceContext serviceContext, String appId, Uri appEndpointUrl, SPApplicationContextAccessTokenType tokenType, SPAppPrincipalInfo appPrincipal, Boolean useThreadIdentity, SPUserToken userToken)
at Microsoft.SharePoint.ApplicationPages.AppRedirectPage.ValidateAndProcessRequest().
Since this is a nonfatal error, it will be sanitized and posted to the app as part of the app launch.

We are on prem and not using ACS, the error message is misleading.  So I tested the scenario in a farm where I had full control of all machines and tried the same steps with the same result.  I then disabled SSL 3.0 on the SharePoint WFE as well and then everything started working.  All good but I’m still at a loss as to why the SCHANNEL connection would not start with or at least fail over to TLS 1.0, which was enabled on both servers and is more secure.  It may be that weak ciphers need to be disabled, etc., etc.  I just haven’t had the time to pursue it.

What I ended up with was TLS 1.0,1.1,1.2 enabled and SSL 2.0,3.0 disabled on SP WFE’s and remote web servers.

Before I close out I did test unsuccessfully with some other SCHANNEL settings as well and here are some of the results

1)      SSL 3.0, SSL 2.0, TLS 1.0 disabled and TLS 1.1, 1.2 enabled on WFE and remote web server.

SharePoint responds with a blank page, login is not requested.  Non starter here, not sure if this is possibly a cert issue.

2)      SSL 3.0, SSL 2.0, TLS 1.0 are disabled, TLS 1.1, 1.2 are enabled on remote web server,
SSL 2.0 is disabled, SSL 3.0 TLS 1.0, 1.1, 1.2 are enabled on SharePoint WFE.

ERROR: The client and server cannot communicate, because they do not possess a common algorithm

TLS 1.0 at least must be enabled, when TLS 1.0 is disabled on the remote web server the connection through the high trust cert to SharePoint is trying to find a common algorithm and I'm assuming the cert cannot do TLS 1.1 or 1.2 or the SharePoint server cannot handle TLS 1.1, 1.2.

I've gone so far as to decompile the SharePoint assemblies and begin jumping through the code but there is more research to be done here and I hope to have some time soon, but until then hopefully this information will help somebody.  Please chime in if you have any input.

<Where><Eq><FieldRef Name="MyUserField" /><Value Type="User">[Me]</Value></Eq></Where>
nor this
<Where><Eq><FieldRef Name="MyUserField" LookupId="True" /><Value Type="User">[Me]</Value></Eq></Where>
nor this
<Where><Eq><FieldRef Name="MyUserField" LookupId="True" /><Value Type="Lookup">[Me]</Value></Eq></Where>

this works

<Where><Eq><FieldRef Name="MyUserField" /><Value Type="Integer"><UserID Type="Integer" /></Value></Eq></Where>

The long answer...

I recently created a view through C# CSOM to return records where the logged on user was equal to the value of a “User” field.  Oddly, no records were returned when I looked at the view, even though my user account was the value of this field in several list items.  Even more puzzling, when I edited the view through the SharePoint UI, it looked correct.  Even more puzzling still, if I clicked the OK button to save the view it started working and all of the records where the “User” field was equal to “Me” were displayed.

So I used SharePoint Manager to do a comparison of what my query looked like before and after I clicked the OK button through the UI and noticed that my query had been changed from

<Where><Eq><FieldRef Name="MyUserField" /><Value Type="User">[Me]</Value></Eq></Where>

To

<Where><Eq><FieldRef Name=" MyUserField " /><Value Type="Integer"><UserID Type="Integer" /></Value></Eq></Where>

SharePoint Manager is an awesome tool for plumbing the depths and getting down to the nitty gritty of all of the objects in your farm, if you've not tried it, you should.

 
The final code comes out looking something like this

ClientContext cc = new ClientContext(https://my.SharePoint.url);
List l = cc.Web.GetListByTitle("My List");
string[] viewFieldsArray = null;
Systen.Collections.Generic.List  <String> viewFields = new List<string>();
viewFields.Add("LinkTitle");
viewFields.Add("Field 1");
viewFields.Add("Field 2");
viewFieldsArray = viewFields.ToArray();
string filter = string.Format("<Where><Eq><FieldRef Name=\"MyUserField\" /><Value Type=\"Integer\"><UserID Type=\"Integer\" /></Value></Eq></Where>");
l.Views.Add(new ViewCreationInformation() { Title = "My Records", ViewFields = viewFieldsArray, Query = filter, RowLimit = 30, Paged = true, SetAsDefaultView = false });
cc.ExecuteQuery();

No.

So I found this great post that gives some good instruction on how to modify Filter_Default.html to allow single valued refiners to appear collapsed when the page is initially rendered.  Awesome, 1 down.

But this didn't work for multi valued refiners because they use a different template, well... actually 2 templates, Filter_MultiValue.html and Filter_MultiValue_Body.html.  Ok, simple enough, we just have to make some mods to both files.  First, we edit Filter_MultiValue.html to add a line to set its defaulted state to collapsed.

var hasNoListData = ($isEmptyArray(listData));

var propertyName = ctx.RefinementControl.propertyName;

var displayTitle = Srch.Refinement.getRefinementTitle(ctx.RefinementControl);

var isExpanded = Srch.Refinement.getExpanded(ctx.RefinementControl.propertyName);

 becomes

var hasNoListData = ($isEmptyArray(listData));

var propertyName = ctx.RefinementControl.propertyName;

var displayTitle = Srch.Refinement.getRefinementTitle(ctx.RefinementControl);

Srch.Refinement.setExpanded(ctx.RefinementControl.propertyName, false);

var isExpanded = Srch.Refinement.getExpanded(ctx.RefinementControl.propertyName);

Now on to Filter_MultiValue_Body.html, and before I go much further, I’ve found what I consider a bug but generally, it would never be noticed  because the default state for any refiner is expanded.

Notice in Filter_MultiValue.html the line

var isExpanded = Srch.Refinement.getExpanded(ctx.RefinementControl.propertyName);

Now you would think that this function would return a boolean type variable, but it doesn’t, it’s a string. 

Srch.Refinement.getExpanded = function (f) {

    var e = "refinementExpandCookieName_" + f,

        a = document.cookie;

    if (!Srch.U.e(a)) {

        var b = a.indexOf(e + "=");

        if (b !== -1) {

            b = b + e.length + 1;

            var d = a.indexOf(";", b);

            if (d === -1) d = a.length;

            var c = a.substring(b, d);

            if (!Srch.U.n(c)) c = unescape(c);

            return c

        }

    }

    return"true"

};

Now take a look at the bottom of the file where our isExpanded string variable is saved for use in Filter_MultiValue_Body.html, the value of isExpanded is a string value of “false”.

ctx.RefinementControl["csr_isExpanded"] = isExpanded;

Now let’s take a look at Filter_MultiValue_Body.html, notice the line

var isExpanded = Boolean(ctx.RefinementControl["csr_isExpanded"]);

The JavaScript “Boolean” function when given a single string as an argument will return false only if the string is blank, this means that a string with a value of “false” will return true.  So, to be clear, any string with 1 or more characters will return true from the “Boolean” function.  So you can see the problem, upon rendering, the isExpanded variable in Filter_MultiValue_Body.html is always true.

I ran into another issue that was really just unexpected behavior, when you would select a few items to refine on, then click the "Apply" button, the entire refiner would hide itself again.  Duh, that’s what I told it to do.  I had to add some code to determine if there were some refiners selected and override the collapsed state.

Anyway, here’s the code in Filter_MultiValue_Body.html.

<!--#_

var propertyName = ctx.RefinementControl["csr_propertyName"];

var displayTitle = ctx.RefinementControl["csr_displayTitle"];

var filters = ctx.RefinementControl["csr_filters"];

var isExpanded = Boolean(ctx.RefinementControl["csr_isExpanded"]);

var renderEmptyContainer = Boolean(ctx.RefinementControl["csr_renderEmptyContainer"]);

var useContains = Boolean(ctx.RefinementControl["csr_useContains"]);

var useKQL = Boolean(ctx.RefinementControl["csr_useKQL"]);

var showCounts = Boolean(ctx.RefinementControl["csr_showCounts"]);

if($isEmptyString(propertyName) || (!$isNull(renderEmptyContainer) && renderEmptyContainer))

{

_#-->

        <divid="EmptyContainer"></div>

<!--#_

}

else if(!$isNull(filters) && Srch.U.isArray(filters) && !$isEmptyArray(filters))

{

    var expandedStatus = !$isNull(isExpanded) ? isExpanded : true;

    var iconClass = "ms-core-listMenu-item ";

    iconClass += expandedStatus ? "ms-ref-uparrow" : "ms-ref-downarrow";

_#-->

        <divid="Container">

            _#= Srch.U.collapsibleRefinerTitle(propertyName, ctx.ClientControl.get_id(), displayTitle, iconClass) =#_

            <divclass="ms-ref-unselSec" id="UnselectedSection">

                <divid="unselShortList" class="ms-ref-unsel-shortList">

Becomes

<!--#_

var propertyName = ctx.RefinementControl["csr_propertyName"];

var displayTitle = ctx.RefinementControl["csr_displayTitle"];

var filters = ctx.RefinementControl["csr_filters"];

var isExpanded = ctx.RefinementControl["csr_isExpanded"];

var renderEmptyContainer = Boolean(ctx.RefinementControl["csr_renderEmptyContainer"]);

var useContains = Boolean(ctx.RefinementControl["csr_useContains"]);

var useKQL = Boolean(ctx.RefinementControl["csr_useKQL"]);

var showCounts = Boolean(ctx.RefinementControl["csr_showCounts"]);

if($isEmptyString(propertyName) || (!$isNull(renderEmptyContainer) && renderEmptyContainer))

{

_#-->

        <divid="EmptyContainer"></div>

<!--#_

}

else if(!$isNull(filters) && Srch.U.isArray(filters) && !$isEmptyArray(filters))

{

    var expandedStatus = !$isNull(isExpanded) ? (isExpanded == "true" ? true : false) : true;

    if (!expandedStatus) {

        for (var i = 0; i < filters.length; i++) {

            var filter = filters[i];

            if(!$isNull(filter)) {

                if (Boolean(filter.IsSelected) == true)

                {

                    expandedStatus = true;

                    break;

                };

            }

        }

    }

    var iconClass = "ms-core-listMenu-item ";

    iconClass += expandedStatus ? "ms-ref-uparrow" : "ms-ref-downarrow";

    var displayStyle = expandedStatus ? "" : "none";

_#-->

        <divid="Container">

            _#= Srch.U.collapsibleRefinerTitle(propertyName, ctx.ClientControl.get_id(), displayTitle, iconClass) =#_

            <divclass="ms-ref-unselSec" id="UnselectedSection"style='display:_#=$htmlEncode(displayStyle)=#_'>

                <divid="unselShortList" class="ms-ref-unsel-shortList">

AutoCollapsedRefiners.zip

The longer explanation: In my continuing journey with Provider Hosted applications, I've run into a new issue that is documented in other areas of Microsoft technology, but not SharePoint.  I requested an enterprise cert that we would use for S2S high trust, typically certs that are generated for web environments are generated using a web site template, on Microsoft platforms the website templates typically use CNG (Cryptographic Next Generation) and IIS supports them just fine.  High trust certs, unlike standard website SSL certs, are not imported into IIS and used to encrypt browser transactions, they are broken up into public and private key files (cer and pfx), the public key is used on the SharePoint server and the private key used on the remote web server.  The files together provide secure encryption for the remote web and the SharePoint server to talk.  We built our app in Visual Studio 2013 and the standard Provider Hosted Application template in Visual Studio comes with some great base code that handles all of the nitty gritty of using the private key to talk back to SharePoint.  After setting up the key files and attempting to navigate to one of our I pages I received the error below.  At the point where the TokenHelper class attempts to access the private key in the pfx file the error is thrown, the reason is that the crypto api does not support certs generated with CNG.  See the short solution above.

Server Error in '/' Application.

An internal error occurred.
Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code. 
Exception Details: System.Security.Cryptography.CryptographicException: An internal error occurred.
Source Error: 

Stack Trace: 

[CryptographicException: An internal error occurred.
]
   System.Security.Cryptography.CryptographicException.ThrowCryptographicException(Int32 hr) +41
   System.Security.Cryptography.X509Certificates.X509Utils._LoadCertFromFile(String fileName, IntPtr password, UInt32 dwFlags, Boolean persistKeySet, SafeCertContextHandle& pCertCtx) +0
   System.Security.Cryptography.X509Certificates.X509Certificate.LoadCertificateFromFile(String fileName, Object password, X509KeyStorageFlags keyStorageFlags) +307
   System.Security.Cryptography.X509Certificates.X509Certificate2..ctor(String fileName, String password) +84
   Foo.Web.TokenHelper..cctor() in c:\Foo\FooWeb\TokenHelper.cs:674

[TypeInitializationException: The type initializer for 'Foo.Web.TokenHelper' threw an exception.]
   Foo.Web.TokenHelper.IsHighTrustApp() in c:\Foo\FooWeb\TokenHelper.cs:618
   Foo.Web.SharePointContextProvider..cctor() in c:\Foo\FooWeb\SharePointContext.cs:307

[TypeInitializationException: The type initializer for Foo.Web.SharePointContextProvider' threw an exception.]
   Foo.Web.SharePointContextProvider.CheckRedirectionStatus(HttpContext httpContext, Uri& redirectUrl) in c:\Foo\FooWeb\SharePointContext.cs:410
   Foo.Web.Default.Page_PreInit(Object sender, EventArgs e) in c:\Foo\FooWeb\Pages\Default.aspx.cs:15
   System.Web.UI.Page.PerformPreInit() +49
   System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) +1844

Version Information: Microsoft .NET Framework Version:4.0.30319; ASP.NET Version:4.0.30319.18446

I was thinking "well, it must be some issue with the high trust cert, maybe some oAuth problem, app manifest or web.config not right, etc.  So I went off and checked all kinds of things and then almost accidentally I was looking at the bin directory and noticed that the SharePoint client assemblies were all missing.  I then checked my Visual Studio assembly references for the project in question and noticed that "Copy Local" was false for all of the SharePoint client assemblies.

There are some very specific reasons for the "Copy Local" flag to be set to false and I don't think the SharePoint assemblies meet the criteria so I'm a bit in the dark about why this would happen in the first place.

1. If the reference is another project, called a project-to-project reference, then the value is true.
2. If the assembly is found in the global assembly cache, the value is false.
3. As a special case, the value for the mscorlib.dll reference is false.
4. If the assembly is found in the Framework SDK folder, then the value is false.
5. Otherwise, the value is true.

Anyway, lesson learned, always remember to check and verify the "Copy Local" flag is set to true on all assemblies that you've referenced unless you know they'll be in the GAC.

The Problem:

Let's assume you execute the following code against one of your SharePoint lists that contains taxonomy fields

List oList = cc.Web.Lists.GetByTitle(listName);
CamlQuery camlQuery = newCamlQuery();
camlQuery.ViewXml = caml;
ListItemCollection collListItem = oList.GetItems(camlQuery);
cc.Load(collListItem);
cc.ExecuteQuery();

Firstly, we'll assume (and you must),  you've added a reference to Microsoft.SharePoint.Client.Taxonomy.  Without a reference to the taxonomy assembly SharePoint has no idea what to do with the taxonomy fields in collListItem.

But yet, you look at what is returned and notice that taxonomy fields are returning as dictionaries while all other fields return correctly.

What's going on here?

Looking at the data you can see that the dictionary for the taxonomy fields contains 2 fields

[0]: {[_ObjectType_, SP.Taxonomy.TaxonomyFieldValueCollection]}
[1]: {[_Child_Items_, System.Object[]]}

The first one, _ObjectType_, dictates what class SharePoint will use to house the data.

But wait, we referenced the taxonomy assembly in our project, why is this still returning as a dictionary?

The problem is that just because you reference an assembly does not automatically mean that the assembly will be loaded, assemblies may load when first used in your code.
When the ListItemCollection above is populated, Microsoft.SharePoint.Client.Taxonomy has not yet been loaded, the code checks to see if the SP.Taxonomy.TaxonomyFieldValueCollection class is available and if not you get a dictionary.

The Solution

You must force the Microsoft.SharePoint.Client.Taxonomy assembly to be loaded.
There are other ways to accomplish this but here is what we did.
Just before we execute the code to get our list data, we added this line.

//*** dummy to force Microsoft.SharePoint.Client.Taxonomy.dll to be loaded
TaxonomyItem dummy = new TaxonomyItem(clientContext,null);

That did the trick.  We also tried creating the TaxonomyItem as null thinking that might do it but it still failed, it had to be an actual instantiation.

The final code

//*** dummy to force Microsoft.SharePoint.Client.Taxonomy.dll to be loaded
TaxonomyItem dummy = new TaxonomyItem(clientContext,null);
List oList = cc.Web.Lists.GetByTitle(listName);
CamlQuery camlQuery = newCamlQuery();
camlQuery.ViewXml = caml;
ListItemCollection collListItem = oList.GetItems(camlQuery);
cc.Load(collListItem);
cc.ExecuteQuery();

The result

I’ve been working on a SharePoint provider hosted app for a client and part of the overall solution is a deployment of artifacts to the Host web.  There are many valid and important reasons for having your app use the host web as its main data store opposed to the app web.  In short, if your app is considered the center piece of your site and is used to accomplish things you would have typically tried to do using full trust code in the past, the host web is probably your best bet.  If your app is a widget and you expect to install it in many sites then the app web is how you should probably go.  Just my 2 cents, probably worthy of another post to join the fray of opinions.  Just remember, if your app gets uninstalled, everything in the app web goes along with it.

Back to the issue, I’ll try to keep it short and sweet.  I opted to create a console application to do the deployment to the host web.  At some point in SharePoint development you usually run into the scenario where you either delete the site collection or just delete some lists, content types, etc. and try to re-deploy for testing.

After deleting some of my custom site columns of type “TaxonomyFieldType” and “TaxonomyFieldTypeMulti”, I tried to run my deployment project and ran into this issue.

When you create a Taxonomy field, SharePoint actually creates 2 fields.  The first is the field that you created.  The second is a hidden, cryptically named, “helper” field.  When you delete a taxonomy field the hidden field is left behind.

The naming logic goes like this.  Take the id of the Taxonomy field, convert to lower case, remove all hyphens, then replace the first character with another character.  I’m not sure how the first character is derived, I just know it replaces the original first character

So, for example...  {787F977C-EA60-47F0-88B8-B7651J21CECE} becomes l87f977cea6047f088b8b7651j21cece, and l87f977cea6047f088b8b7651j21cece is the name of the hidden field.  So in order to truly delete a taxonomy field I wrote the following code to delete both fields.  You could also attempt to create the field, catch the exception, extract the hidden field name from the exception message, delete the field again, then use the extracted hidden field name to delete the hidden field.

public static void DeleteField(ClientContext cc, string internalName,string id = "")
{
    FieldCollection fields = cc.Web.Fields;
    cc.Web.Context.Load(fields, fc => fc.Include(f => f.Id, f => f.InternalName));
    cc.Web.Context.ExecuteQuery();

    var field = fields.FirstOrDefault(f => f.InternalName == internalName);

    if (field != null)
    {
        id = field.Id.ToString();
        field.DeleteObject();
        cc.Web.Context.ExecuteQuery();
    }

    if (id != string.Empty)
    {
        string hiddenFieldName = id.ToLower().Replace("-", string.Empty).Replace("{", string.Empty).Replace("}", string.Empty).Substring(1);
        field = fields.FirstOrDefault(f => f.InternalName.EndsWith(hiddenFieldName));
        if (field != null)
        {
            field.DeleteObject();
            cc.Web.Context.ExecuteQuery();
        }

     }

     cc.Web.Update();

}