We found a way to reproduce the error with the following:

1.)   Create a new list

2.)   Create a new workflow with SharePoint Designer

             a.)   Step 1: Email user X

3.)   Create a new list item

4.)   Start the workflow

5.)   Watch the failure

Note: Every other attempt after this work would fine  (restarting the workflow)

Like I said – the ULS logs were clean, tracking the correlation ID we came up with nothing.    Right when we took a break to review the final 3 sets of logs, my friend Wes had attempted a fix he used in the past with another workflow problem.

Disabling the OffWfCommon feature. 

disable-spfeature -identity  "OFFWfCommon" –url https://yoursitehere/sites/site

enable-spfeature –identify “OffWfCommon”    –url  https://yoursitehere/sites/site

After doing this the workflow consistently worked - but had me questions as to why, so I did some checking and I have found this to be a fix for other similar workflow issues such as a modified “Workflow Task” content type version that contains this error:  

The requested workflow task content type was not found on the SPWeb

If you have a modified "Workflow Task" version greater then 0 and are seeing this same issue, the steps above may be relevant.  I would always recommend testing this in a test environment fully before doing it.   Here is some sample PowerShell code to help find versions that are not 0.  Again, to be used in a test environment.

$site = get-spsite https://yoursitehere/sites/site  
$site = $site.RootWeb
foreach ($sites in $site)
{
   foreach ($ctype in $site.ContentTypes)
  {
       if ($ctype.name -eq "workflow Task" -and $ctype.version -gt "0")
           {

              $ctype.Scope,$ctype.name,$ctype.version }
            }

  }

}

Please only use these steps if you are experience the exact scenarios listed above, and I hope this helps you!

Remember - The August 2012 CU is the first patch to require Service Pack 1 to be installed.  This means you must have SharePoint 2010 SP1 AND any Language Packs SP1 installed also.   If you are using multiple languages, you will have multiple Language Pack Service Packs to install.

Hope this helps you out!

If you ever have trouble explaining a problem in SharePoint, capture a video of it.  One of my favorite things to see customers do is send me screen shots of the issue, but even better a video!

Get to know this software!   https://blogs.msdn.com/b/devschool/archive/2011/03/07/community-clips-one-of-the-best-and-easiest-free-video-screen-capture-available.aspx

Before I start - this blog is what saved us, but it wasn't dedicated to our specific issue -  It was 'Scenario 4' that hurt us.

One of my customers is implementing a new SharePoint Farm using SharePoint Server 2010 and UAG so they can protect outside access on the permiter network, versus allowing authentication to happen on the internal SharePoint Server / network.    Everything was working great and was very seamless until we started to create different Alternate Access Mappings with SharePoint.  

As I said, our requirement was to have authentication done on the UAG Server and not SharePoint.  Scenario 4 (listed above) was what we needed as far as Host Header/AAM configuration.  In order to do scenario 4 you need to make sure IIS is configured to use HTTP and AAMs are set up to handle both HTTP and HTTPS traffic.  There is 1 minor catch to this when using UAG. 

First let me show you how this is typically done.

AAMs Standard Example - SSL Offloading

Now - let me show you how this is needed to be done while using UAG.

AAMs UAG Example - SSL Offloading

 Notice that the HTTP Internal URL has a 'dummy URL' that can be anything - It just cannot be the same as the Public Zones.  

You also have to change this inside of the UAG configuration (Remember:  Modify URLs to match your setup)

Hope this helps!

By now everyone should know about ULSViewer that is available for download from MSDN, and you also should be familiar with the data filter that comes built into the tool.   If you are not familiar with the ULSViewer you may be using some other log reader that is out there, but I would highly recommend getting use to ULSViewer as it is a powerful, user friendly,  tool.   This blog is about making your life easier to troubleshoot SharePoint issues using the ULS logs and ULSViewer filter.  

One of the  best things I have found with the ULSViewer is the ability to create, save, and reuse a filter.   When you do this, you can bring it to any other ULSViewer and import the filter.  Why is this handy?  This is handy because you may have a common web application, or site that you are trying to troubleshoot and you know exactly what you are looking for but don't like entering in all of the filter fields over and over again.

The following filter is one that I have used many times in performance related issues.   It has a the common stack of performance EventIDs and has saved me many times during troubleshooting performance problem.

_______Tag Description (EventID)________
nask             SPRequestMemory leak
c8hq             ECM Object Cache Full Flush - too many changes
8gsc             WSS Template Cache Trim - memory usage exceeds target
avjr              WSS Template Cache Trim - low memory condition
fa43             slow query with duration
fa44             call stack for slow query
tzkv             query text (useful for slow queries and sql exceptions)
tzku             connection string (useful for slow queries and sql exceptions)
8pbe            Watson bucketing parameters
fa45             exception encountered during a sql query
btq8             exception encountered during a sql query
d0d6            exception encountered during a sql query
880i             exception encountered during a sql query

So how to do this?

1.) Open ULSViewer and load a ULS log - this gives you the ability to create a filter

Notice:     The EventID are IDs inserted in the ULS directly from the SharePoint source.   In most scenarios this will be a pretty unique ID, however some generic catch-all's will be used and may not be as helpful, but great to Bing!

2.) Open the Filter Tool with the icon directly below the Edit Menu (shown above)

3.) Start creating your filter!

Note:  This will be the hardest part.  Everything will be OK in the end, so start clicking and typing!

Notice:  One thing to note here is the 'Save As', 'Save', and 'Load' button.  This is obviously where you would save, or import your filter for reuse.   Start using this TODAY!

Notice:  Another very cool feature is the ability to Group your AND/OR logic so your filter works as expected.   If you have a grouping set the bar will show on the left side of the filter.

4.)  That's It!  You now have created a very useful filter that will help you troubleshoot a lot of performance related problems.

On a final note, I really love when our customers are proactive in troubleshooting new features or applications in SharePoint.   You could create a AND filter like I have done above to watch for specific sites and see how they are performing when using your new tool, and if they trigger any of the performance EventIDs above when you test them.

If anyone has a useful filter combination please post it in the comments below and explain how it helped you troubleshoot SharePoint!