Would you call Microsoft to find out why the device wasn't working or would you call the vendor of the hardware piece you just installed?  You'd call the hardware vendor of course since it is up to them to provide the necessary testing needed to make sure it worked with your Microsoft product and make the correct drivers and software available for you to install.  Sound about right so far? Well this same scenario can be applied to your virtual environments.  All the virtual hardware is being provided by the vendor of the virtual software.  Microsoft's operating systems and applications, for the most part, are not cognizant of the existence of the virtual devices and treat them as if they are the real deal.  In some cases where the software or OS is aware of the virtual environment it will still treat it as physical.

So what does this have to do with my question regarding how SharePoint is supported in a virtual environment?  Well if you've seen the TechNet article on best practices you'll see part of the answer to that question. See https://support.microsoft.com/kb/2764086 as well as https://technet.microsoft.com/en-US/library/ff621103(v=office.15).aspx#ConfigMem

In that article it addresses how to setup the different virtual hardware devices such as your NIC, CPU, hard disk space and even memory.   For the most part it seems pretty standard until you get to features unique to the virtual software itself.  Which leads me to Dynamic Memory a feature you'll find in Hyper-V as well as VMware.  Even though both Hyper-V and VMware implement the use of Dynamic Memory in different ways neither of them is supported for SharePoint installations.  The reason is that this implementation of dynamic memory does not work with every SharePoint feature. For example, Distributed Cache and Search do not resize their caches when the allocated memory for a virtual machine is dynamically changed. This can cause performance degradation, especially when assigned memory is reduced. 

The official documentation on using a Microsoft product within a virtualized environment is this:https://www.windowsservercatalog.com/svvp.aspx?svvppage=svvpwizard.htm 

There are no industry standards to follow in implementing such features, and since 'by design' the operating system [whether that be Windows Server or some other OS] or application are not cognizant of such virtualization product functionality, there is no practical method of testing these virtualization product features using Microsoft-developed tests and tools. The SVVP program does not test these features or functions, and the virtualization product vendor is solely responsible for testing and supporting such features. However, unless otherwise stated in articles or documents for Windows Server operating systems or for Microsoft Server-focused applications, Microsoft does not preclude their use by customers. If a customer running a supported version of Windows Server on a validated virtualization solution experiences issues after using such third party virtualization features (such as live migration) that operate independently of Microsoft products, then the customer should contact the virtualization  vendor for assistance in resolving the issue. 

Long story short?  Microsoft does not support a SharePoint 2013 installation when any form of dynamic memory is enabled within the virtual product.   Read the Microsoft documentation on the topic found here and make note of page 17.   https://download.microsoft.com/download/0/0/1/001ADCCC-A45B-47E3-8DA4-ED51E3208021/Best_Practices_for_Virtualizing_and_Managing_SharePoint_2013.pdf

As a SharePoint Site administrator you've been given the task of finding a solution for the following request:

You've been dubbed the resident SharePoint guru of Contoso University's literature department.  Russell Johnson, one of the professors, has asked you to add a new sub site under the site https://ContosoUniversity.edu/Liter8r/101.  Your job is to evaluate Professor Johnson's expectations from the site and provide a solution.

The professor and the rest of his team are planning to host a 'Short Stories' competition for their young authors.  The students are expected to register on your new SharePoint site, upload a copy of their original short story and then each teacher will critique the students work and elect a winner.

After (not so much) deliberation you come up with the following sub site setting for Professor Johnson's upcoming competition:

• First you decide to create a Custom List site under the https://cu/Liter8r/101 master site.
• Second, you go to the new sites List Settings and you enable versioning on the site in order to keep track of any changes being made to the items throughout the competition. Just the minimum setting to allow a new version each time an item is edited.

• Then, while you're in the list settings, you add a new column so that all new items added can have their own comments field. You set it up for multiple lines of text and since you have versioning enabled the option to append changes to existing text is available so you add that in there too.

• Add each of the students from the Literature classes involved to the site permissions so they may create their own Items.

•  At this point you explain to the students and teachers involved with the competition how to use the site.

  • Student goes to the new SharePoint site and selects +New Item.
  • As part of the new item creation process the student gives it a name that includes the student name and the title of the short story then click OK to create.
  • The student can now click on their new item to open it up and add any comments to the new Multi Text column field that you added to the Custom List sites configuration.
  • From the tools ribbon at the top of the now open item the student can select to edit the item further or attach a file. In this case the student will select to attach a file and they will upload their short story in Word format to the item and click OK.

  As all students continue to follow these steps the professor and his team can open up the newly added items, open the attached Word document, read it and add their own annotations in edit mode. They can choose to add a comment to the item and move on to the next students listed item.

  Make sense so far? Does it sound like a good solution for this projects requirements? Well this is what actually happens…..

          Users, both student and teacher in this case, will observe what appears to be duplicate entries in all these items. Trouble is that they aren't duplicates. This behavior is actually by design. It's a normal behavior when both the versioning and append features fulfill their designed roles. When a custom column with the append option is added to the site design any data that is placed into this column will populate BOTH the items field itself as well as one of the metadata fields of the attached document.

          Since appending is not tracked like versioning is it will maintain whatever data is in its comment field until it is manually changed during an edit of the item or otherwise removed. So each iteration within the items modification list will appear to be duplicating the comments added by the last user. In fact it is just appending whatever the last update was to the comments field. Add to the confusion by editing the attachment and making changes or modifications of any kind and you'll see what appears to be multiple duplicates. Why? Because the item having been opened for editing now places a new modification entry in the list showing the users name, date and time of enabled edit (appending the latest comment to it…even if the comment was not made by the current user). It will then add another modification entry to the list of changes whenever the attachment (currently opened for editing) had modifications made to it and the attachment is saved. It doesn't have to be closed at this point for the new modification entry to appear. As long as a change is made and the document is saved EITHER MANUALLY or with AUTOSAVE. So depending on how long the document is open for editing and changes are made and saves of an auto or manual nature are occuring you will see what would otherwise look like duplicate modification entries in the modifications list. Now that was a mouthful! See why I thought it was best to describe a scenario with a complete implementation of this configuration? Try explaining this in two sentences or less to a support engineer and expect to get assistance that actually addresses your problem.

          One other thing to take note of in order to debunk the idea that these are bugged duplicate entries is to look closely at the modification date/time stamps for each entry. If you are trying to test your theory you are likely running through these steps quickly to see if it duplicates. If so….then yes it will look like a duplicate because the time it took you to run through the steps was likely within a minute. If you go through each step slowly….even waiting a full sixty seconds between steps you will see that each entry in your modifications list is taking place at very different times. Thus….not duplicated….but behavior completely natural for the versioning and append features to display under these circumstances.

            So now what? How do I get around this behavior? In my experience you have a couple options. You can remove the append option from the custom column you created to hold comments. Now you won't get a quick and easy modifications list to reference by simply opening the Custom List item but you can open the item and select Version History from the tools ribbon. This way you can still maintain a list of any changes to the document, including comments, but without the appearance of duplicate entries because the comments will no longer be required as an appended item. You will get a list of strictly changed/modified occurrences, by who and at what date/time.
  
  The other option? You can basically use any of the multiple SharePoint templates that suit this process….just don't use versioning AND append to track changes.

  I hope this proves helpful to some of you out there struggling with this scenario and tricked into thinking it's a bug or that something in your configuration is "broken". It's not broken. Just not a great combination of functions.

   

Now I'm not just talking about how to configure Outgoing or Incoming e-mail in SharePoint.  I'm talking about how an email is triggered, generated and then sent to a recipient(s) when someone clicks on (e.g) Follow when visiting a users profile or blog.  E-mail alerts is one thing...there are timer jobs and SQL tables and drop folders involved to make e-mail alerts work.  The simpler confirmation emails were what was troubling me.  After much digging it appeared to be much simpler than I imaged.

First off...a lot of what you will read below was gathered from TechNet articles and other MS resources after much digging.

This TechNet article helped better explain the Alert portion of e-mail generation quite well.     https://msdn.microsoft.com/en-us/library/office/hh161512(v=office.14).aspx

Confirmation Emails

These emails are actually sent by the SharePoint Web Front End server that is serving up the SharePoint page at the time it is generated. These emails are sent under the identity that the IIS worker process is using (w3wp.exe)

CONFIRMATION EMAIL PROCESSING
When outgoing email is enabled through the Central Admin dashboard the ability to set confirmation emails for web applications becomes available. All that this configuration does is provide the needed properties for generating an .eml file.  The process could almost be compared to clicking an email link in a web page and being taken to your mail client where it automatically populates the address of the recipient.  Only in this case the contents of the Outgoing Email configuration are used as a reference and then whatever profile you trigger during email generation provides the needed field for the rest of the SMTP packet.  Namely the Work email property in your user properties for UPA configuration.  As long as the Work email is linked to a valid AD account and the property configured contains an email address then it will send out a packet to whatever SMTP relay you setup .  <- And this is all I really learned about confirmation e-mail generation.  I'm sure there are steps I'm missing but this was the gist of it and it helped resolve an ongoing issue for me.

Alert Emails

Require the existence of a Timer Job <Immediate Alerts> This Timer Job runs under the credentials of an account that needs to be a member of the SharePoint Administrators Group and that it also has access to the Content Database in SQL. When an alert is triggered the email is not immediately sent (as one might expect) but rather is queue for delivery when the Time Job runs (typically every 5 minutes) and is sent under the credentials of the job account

ALERT EMAIL GENERATION
The following is the procedure for processing alerts:

1. The timer job calls the SharePoint Foundation alert handler every five minutes.

2. The SharePoint Foundation alert handler scans the SchedSubscriptions table to identify which alerts to process.

3. The SharePoint Foundation alert handler creates a query object from the information and calls the Execute() method on the SharePoint Server Query object model.

4. The alerts handler returns the results from the query to the SharePoint Foundation alert handler, which sends an alert with the results to the user.

These scheduled subscriptions will fire and execute the next time the SharePoint Foundation Timer service (owstime.exe) processes the default and custom timer jobs.
This Timer service executes at a predetermined time interval. Timer jobs are the SharePoint-specific way to perform a specific task on a scheduled basis.
Windows SharePoint Services enables support for users to create alerts for other users. The user with these privileges can customize the trigger for an alert, and customize the alert notification. This functionality is exposed in the SharePoint Foundation object model, enabling programmatic manipulation of alerts.

The SharePoint Foundation alert framework has multiple tables for storing information. The alert framework manipulates the following key database tables:

· EventCache
· EventLog · ImmedSubscriptions
· SchedSubscriptions · EventsSubsMatches

A timer job [inappropriately named] Job- immediate-alerts runs (because it processes both immediate and scheduled subscriptions) at intervals of every five minutes (the default). All events that occurred after the last run are picked up (in batches of 10,000 events), and both immediate and scheduled subscriptions are picked up.

Each event is compared with each subscription to find a match (same siteID, same listID and, if itemID is specified, same itemID). If there is a match, the subscription filter is run on the event data to find a match if there are any filters specified on the fields such as "tasks assigned to me filter". If there is a match, security trimming is done for the user who is the owner of the alert. For example, if the user did not have permissions to read the list items, he will not get the alert. If the match between event a nd subscription passes security trimming, what happens next depends on whether it matched to an immediate subscription or to a scheduled subscription.

If the event matched an immediate subscription, the event and subscription are added to an in - memory junction list consisting of subID -eventID, which is processed later to build the alert email or message and send it out. Otherwise, if the event matched a scheduled subscription, the event and subscription it matched is added to EventsSubMatches table to be picked up again when that subscription is actually due.
After all the events are processed, the matches between events and immediate subscriptions are built into alert messages and sent. The matches between events and scheduled subscriptions are added to the EventsSubMatches table. Then, the scheduled subscriptions that are due are picked up with event-subscription junction entries for each subscription that is due. All the events for each subscription are rolled up and sent as an alert.

Three Steps of SharePoint Alert Subscription Processing

The processing of changes to send out alerts occurs mainly in three steps:

1. Perform filtering and security trimming.

2. Process immediate subscriptions.

3. Process scheduled subscriptions.

Filtering and Security Trimming for SharePoint Alerts

The proc_GetEventDataAndSubscriptionFilters procedure is first called to get the latest top 10,000 events that have not been processed yet, that are visible to alerts (EventType & 0x0FFF != 0), and that have at least one subscription with the same siteId and listId in the subscription tables. This is an event batch.

The proc_GetEventDataAndSubscriptionFilters procedure also selects all the subscriptions from both the Immedsubscriptions and Scheduledsubscriptions tables that have at least one event with the same SiteId and ListId. Both the event batch and the list of subscriptions are sorted by SiteId and ListId.

The procedure also marks the ID and event time of the last event so that it can start from there the next time(EventBatch table). Finally, it moves all the events that were picked up in this batch to the EventLog table and sets the EventData and Acl columns of the events that were picked up to NULL in the EventCache table. This is done for performance reasons because the EventCache table is highly used. The entries in the EventLog table will be used in the processing of scheduled subscriptions.

For each subscription in the subscriptions list, all the matching events are found. Matching implies having the same site ID and list ID. Because the list of subscriptions and the list of events are sorted by site ID, list ID, the matching consists of just wa lking over the ordered lists.
When there is a match, further checking is performed to determine whether itemID is the same (if itemID is not NULL in the subscription, that is, it is an item - level alert). If it is, the filter as specified in the binaryfilter column (which is a binary form of filter column) is run on the event data. If the event passes the filter, security trimming is performed on the user of that alert.

NOTE
If it is a dynamic alert, the user to whom the alert will be sent will be picked up from the field specified by DynamicRecipient property in the alerts property bag. If the alert is assigned to a SharePoint group, the group is expanded for the individual u sers and security trimming is performed for each user.

After security trimming is performed, all the matches between events and scheduled subscription are entered in the EventsSubMatches table as junction entries by using the proc_InsertEventSubscriptionJunctionEntries procedure. Each junction entry consists of an eventid, a subscription ID and lookupfieldpermissionresults, which are the results of security trimming on lookup fields for the event.

Processing SharePoint Alerts That Are Scheduled for Immediate Execution

The proc_GetEventDataAndSubscriptionFilters procedure does not get all the columns in the subscriptions tables. As a result, now proc_GetMatchingSubscriptionsData is called in batches of 256 subscriptions for all subscriptions in the memory junction of event -subscription. This gets all the data for each subscription. Then, for each subscription, the alert email or message is built by using the template that is specified by the AlertTemplateName column.

Processing Scheduled SharePoint Alert Subscriptions

Scheduled subscriptions can be either daily or weekly regular alerts or daily or weekly always- notify alerts. Always notify alerts are alerts that fire without any matching event.
Each of the following steps happen serially for each type of scheduled subscription. Processing of scheduled subscriptions starts by getting a list of site collections that have at least one daily or weekly alert.

Scheduled SharePoint Alert Subscription Process

1. The proc_EnumSubscribedSites procedure is called to get a list of all site collections that have at least one daily regular alert in the schedsubscriptions table; actually, all site collections that have at least one daily regular alert and at least one event in the event log table, to eliminate sites that have a very low volume of events.

2. The proc_MatchSchedSubscriptions procedure is called for each SiteId in the list of site collections. It gets the list of daily alerts for that site collection, all the events-subscriptions junction entries for that site collection, and all the events from the EventLog table. It also checks the NotifyTime and gets only those subscriptions that are due, meaning that NotifyTime is earlier than the current time.

3. For each subscription, all the events are rolled up into one digest and the alert is sent out. The lookup permissions results are useful in building the alert because those lookup fields that the user is not authorized to see should not be rendered in the alert.

4. The proc_DeleteEventLog procedure is called to delete all the entries in EventLog that are older than one week because they will not be used in any subscription.

The previous steps are performed for each type of scheduled subscription: daily or weekly regular alerts and daily or weekly always notify alerts. Most of the time, always notify alerts do not have to have an email built and sent. Next, the proc_DeleteSubscriptionJunctionEntries procedure is called to delete all the junction entries for each subscription that was processed. This is done in batches of 256 subscriptions so that the database server is not overloaded. The proc_UpdateSchedSubscriptionTimes procedure is then called to update all the subscriptions whose NotifyTimeUtc is earlier than the current time. It adds a day to all the daily subscriptions and a week to all the scheduled subscriptions.

I hope that the above gathered information provided some help to those looking for resources on this topic!

The incoming e-mail feature in SharePoint enables sites to receive and store e-mail messages and attachments within SharePoint lists and libraries.  Making it possible for teams to store the e-mail that they send to other team members without needing to go to the SharePoint site and upload the content that was sent in e-mail. What makes this possible is the ability for lists and libraries to be assigned a unique e-mail address. 

Before configuring incoming e-mail, you must perform the following tasks (depending on the type of implementation you choose)

Incoming e-mail service timer job

Back in SharePoint 2010, the Incoming e-mail service timer job had an SPJobLockType of None, which meant that the job ran on all servers in the farm, as long as the Incoming E-mail service was deployed.

TechNet Article:  SharePoint Administration SPJobLock Types Defined

When SharePoint 2013 was released that SPJobLockType had changed to Job, which meant it could only run on a single member server in the farm.  This presented a problem for SharePoint administrators upgrading from SharePoint 2010 that intended to use round-robin MX records.  Administrators had to disable the Incoming E-mail service on all SharePoint 2013 servers except the one designated by the MX record.  By doing this it prevented the Incoming e-mail service timer job from running on a server that didn’t actually contain e-mail.

SharePoint 2013 Incoming e-mail service changed!

In December of 2013 Microsoft released a cumulative update for SharePoint 2013 Foundations that included a change to the Incoming E-mail service timer job.  Essentially it set the SPJobLockType back to None allowing administrators to take advantage of the multi-server Incoming E-mail service deployment option.

But the December 2013 CU didn’t help?

In some cases when an administrator applied the December 2013 CU they did not see the expected change in the SPJobLockType.  In these cases the problem occurred because the Internal E-mail service was already in place prior to the application of the December 2013 CU.  So the SPJobLockType remained the same (in Job state).  What the administrators in this case needed to do was to remove the existing Internal E-mail service feature and it’s timer job and then add them back again after the December 2013 CU was applied.  When these steps were followed the problem went away.

1. Delete the Job:
   
   $job = Get-Sptimerjob | ? {$_.Name -eq "job-email-delivery"}
   
   $job.Delete()
   
   
2. Re-enable the “Incoming E-mail”, this will recreate the job
   
   - Central Admin->System Settings
   
   - Configure incoming e-mail settings
   
   - Configure all option as necessary
   
   
   
3. Confirm the Locktype : None for job-email-delivery
   
   get-sptimerjob job-email-delivery | fl
   
   
4. Restart the timer job across all servers.

TechNet Articles:

• Plan incoming e-mail for a SharePoint farm in SharePoint 2013
• Configure incoming e-mail for a SharePoint 2013 farm

There are several authentication options when configuring an External Content Type to point to a SQL server list.

Connect with User's Identity

Connect with Impersonated Windows Identity

Connect with Impersonated Custom Identity

The steps for each method above are outlined here: 

Connect with User's Identity

Use this for authenticating with the credentials from SQL itself when a user visits SharePoint to view the SQL list content

I.SharePoint Designer

1. Create new External Content Type
2. Create new database connection. Use Connect with User's Identity
3. Create Read Item and Read List operation at the very least
4. Save External Content Type
5. Right-click newly created ECT and create External List
   

Connect with Impersonated Windows Identity

Use this option for SQL authentication when you want your user accounts hidden behind a single Windows alias. The alias account credentials are kept in the Secure Store Service Application.

I.  SharePoint Central Admin

1. Central Admin > Manage Service Application. Create a new Secure Store Service Application and edit it.
2. Create a new target application using the Group type.
3. Enter all the accounts that require access on the next screen.
4. Once the Target Application has been created open the associated drop down list and select Set Credentials and enter the Username/Password of the AD alias account.

II. SharePoint Designer

1. Create new External Content Type
2. Create new connection and use the Connect with Impersonated Windows Identity option and enter the Secure Store Application ID you just created in Central Admin above
3. Create a Read List and Read Item operation at the very least
4. Save the External Content Type
5. Open up the options menu of the newly created ECT to create an External List

III. SharePoint Central Admin

1. Once you are finished with the previous step in SharePoint designer it automatically creates your External Content Type
2. From the Business Data Connectivity Service Application find the new ECT, select the box next to it, and go to Set Object Permissions located on the command ribbon.
3. Select a user or group with access to the ECT and grant at least Execute permissions to it.
4. Select Set Metadata Store Permissions located on the command ribbon

Connect with Impersonated Custom Identity

This option is very much like the Impersonate Windows Identity selection but the configuration uses an account created in SQL as opposed to an AD account as was used In the Windows Identity section above.

I.   SharePoint Central Admin

1. Central Admin > Manage Service Application and select or create a new Secure Store Service Application and edit it. Create a new target application using the Group type.
2. After the Target Application has been created select Set Credentials from the dropdown and enter the Username/Password of a SQL account

II.  SharePoint Designer

1. Create a new External Content Type
2. Create a NEW connection, or use one that you know was setup w/ a SQL login. Use the Connect with Impersonated Custom Identity option and enter the Secure Store Application ID created in the previous steps.
3. Create a Read Item and Read List operation at the very least
4. Save the External Content Type
5. Open up the options menu of the newly created ECT to create an External List

III. SharePoint Central Admin

1. Once you are finished with the previous step in SharePoint designer it automatically creates your External Content Type
2. From the Business Data Connectivity Service Application find the new ECT, select the box next to it, and go to Set Object Permissions located on the command ribbon.
3. Select a user or group with access to the ECT and grant at least Execute permissions to it.
4. Select Set Metadata Store Permissions located on the command ribbon

Hope you find this useful.

I've seen several blogs addressing the need for SharePoint blog sites to require an approval process
before posts are published and made visible to the rest of the blog site members.  Unfortunately
the majority of them answer this need with a less than easy set of instructions for creating a
custom workflow.  Workflows are great solutions for so many function driven needs in SharePoint
but this particular topic can be handled quite easily with an OOB solution.

If you look at a SharePoint 2013 blog site it’s essentially just another way of presenting the
contents of a separate set of list objects.  One list that contains each post that is created and a
second list that is used to store the various comments created for each post. 

If you go to either of the links under your Blog tools for Manage posts or Manage comments
these links will bring you to the List site that holds each post or comment respectfully.

Let’s focus on the approval process for blog posts.  So from your main blog site you click your
Manage posts link and see the posts in list format.  A freshly created OOB blog site includes the initial
sample post “Welcome to my blog!” as you can see.

This is where you need to configure your versioning settings in order to prevent new posts from being
published and hence visible to the rest of the blog members.  Open up the LIST tab at the top of your
list site and under the Settings section select List Settings.

Under the General Settings you will see the Versioning settings link. Click the Versioning settings link
and select Yes under the Content Approval option [Require content approval for submitted items?]
and select “Only users who can approve items (and the author of the item)” under the Draft Item Security
option [Who should see the draft items in this list?].  Click OK.

Now if you go back to your blog sites home page select ‘Create a post’ under your Blog tools section.

As you can see an Information ribbon is presented to users creating the new post explaining that
approval is required and that their post submission will not be visible in public views until someone with the
proper rights approves it.

In the case where a member of an administrative group creates a new post they are automatically given the option to publish the post
upon completion.

Here is an example of the post creation console when opened by a user in the members group.

This next part is where many people get confused.  The newly created post will be visible on the blog site
home page to the person that posted it as well as to the members of the approval group.  The post will NOT
be visible to the rest of the blog site community until it has been approved.

1. Post Creators View

1. Approvers View

1. Public View

Once a member of the approvers group sees the new post they can go to the Manage posts link
under the Blog tools and approve or reject the new post.  Email notification is not part of the
versioning content approval process but you can easily set this up by using either SharePoint
alerting or by creating a simple workflow to notify members of the approval group whenever a
new list item is created with Pending as the approval status.  Of course if you’re taking the route
listed in this post then you’re probably trying to avoid workflows all together so the alerting feature may best suit
your needs when it comes to generating an email alert to the members of your approval group.

Here's a good Technet article on configuring Alerting: https://technet.microsoft.com/en-us/library/cc287751(v=office.14).aspx

Hopefully this will at least provide admins with the elusive fix for SharePoint 2013's inability to start up it's User Profile Synchronization service after installing or migrating to SQL Server 2014.

What's new in SQL Server 2014 [https://msdn.microsoft.com/en-us/library/bb500435(v=sql.120).aspx]

SQL Server 2014 Home Page [https://www.microsoft.com/en-us/server-cloud/products/sql-server/]