Historical Monitoring Using WCF Call Metrics
The WCF Call History section displays historical metrics for WCF calls coming into .NET Framework 4 WCF and WF services managed by AppFabric. The Completed and Errors summary metrics provide a cumulative total of successes and failures as a result of received WCF calls. You can expand the WCF Call History section (by clicking the down arrow or anywhere except the summary metric links within the WCF Call History title bar) to display the top five services with Completed Calls or Service Exceptions. This action also displays the number of Errors - Grouped by common types. You can use these top-level summary values, and their descending correlated detailed views, to monitor the history of WCF calls for your services.
WCF Call History Metrics
The WCF Call History section on the AppFabric Dashboard provides a summary view of WCF service calls in the shaded header box where the title WCF Call History exists. The following metrics give you a quick outline view of WCF calls that completed successfully and those that did not:
The Completed summary metric is a cumulative total of the received WCF calls that completed without generating an error or an exception, producing the OperationCompleted WCF event type.
The Errors summary metric is a cumulative total that represents all WCF-related exceptions and user-defined errors that occur throughout the entire WCF stack. This count includes, but is not limited to, errors resulting from the number of failed and faulted calls.
The Throttle Hits metric provides a count of the number of WCF throttle periods experienced by any of the services deployed within the applicable scope. A single throttle period begins when a WCF throttle threshold is exceeded. It ends when throughput decreases below 70% of the throttle threshold.
Below the summary metrics are the following service metrics that summarize the number of services within specific categories of calls in descending order:
Completed Calls - Grouped by Service (top 5) - The top five services with the highest total number of completed WCF calls within the specified time period.
Service Exceptions - Grouped by Service (top 5) - The top five services with the highest total number of exceptions within the specified time period.
Errors - Grouped by common types – Grouping of the total number of service exceptions or user-defined errors. The service exceptions count is also broken down further by two of the most common causes: failed and faulted calls. Service exceptions can also be caused by issues other than Failed or Faulted calls, such as activation errors.
Not all tracked WCF events are included in the Completed Calls - Grouped by Service (top 5) or Service Exceptions - Grouped by Service (top 5) columns within the Dashboard. Events displayed in these “top 5” columns do not include those emitted from the internal AppFabric service that manages endpoints and events. Only events that have a complete service virtual path will be displayed in these two columns. Events generated higher in the WCF channel stack, such as callbacks and service exceptions, don't contain a full service virtual path. Events that do not contain a complete service virtual path are excluded from these counters. This ensures that only user services, and not user applications, appear in those counter values.
Tracked Events Page
You can use the Tracked Events page to get a historical view of the events that occurred during a particular sequence of WCF calls for a service instance. Clicking any of the summary metrics (say Completed), or one of the service links under a column in the WCF Call History section (say under the Completed Calls column), takes you to the Tracked Events page.
.gif "Tracked Events Page")
The link that you click to take you from the AppFabric Dashboard page to the Tracked Events page is used to filter the WCF call and event data so that what is enumerated on the Tracked Events page is specific to that originating link. For example, clicking a service entry link under the Service Exceptions Grouped by Service (top 5) column takes you to the Tracked Events page. It enumerates events filtered by the service event type (WCF Exceptions in this case) and the emit time that corresponds to the metric and time period selected from the AppFabric Dashboard.
You can, however, change the value of one or more fields (say Events) within the Query Summary frame to change the initial output and do further troubleshooting on a specific event type. For example, if the original metric that took you to the Tracked Events page was WCF Exceptions, you could change the value of the Events field to All WCF Events, and then click Run Query to see different results.
You can use the following options to filter the type of events that are displayed:
All WCF events - Displays all WCF events available in the monitoring store including, but not limited to, WCF Completed Calls, WCF Exceptions, WCF Failed Calls, and WCF Faulted Calls.
WCF completed calls - Displays all completed WCF calls.
All WCF errors - Displays all events emitted at the Error level: service exception and user-defined error events.
WCF exceptions - Displays all WCF service exceptions.
WCF user defined errors - Displays all user-defined events that are emitted at the Error level.
WCF failed calls - Displays all failed WCF calls.
WCF faulted calls - Displays all faulted WCF calls.
WCF throttle hits - Displays WCF throttle events. Each event represents a throttling period and is emitted when a WCF throttling threshold is first exceeded.
All WF events - Displays all WF events.
For more information, see Tracked Events Page.
Troubleshooting by Monitoring WCF Call History Metrics
You can assemble the preceding information into a troubleshooting approach by using the WCF Call History section to monitor WCF calls into .NET Framework services. When you initially view the WCF Call History section, you get a high-level summary view of the status of WCF calls. You can quickly see if there is a problem at the WCF call level by any exceptions, failed, or faulted calls that have occurred. If either of the Service Exceptions - Grouped by Service (top 5) or Errors - Grouped by common types summary metrics contains a non-zero value, it indicates where a problem may have occurred. Each summary metric is linked to the Tracked Events page, where you can see explicit detailed metrics of WCF call data that the initial AppFabric Dashboard page summarized at the higher level. This raw data gives you additional information to isolate a problem surrounding WCF calls.
Let’s take a scenario where you are using the WCF Call History section to monitor WCF calls for your WCF or WF services at a given scope for any problems. If you see the Exceptions summary metric as non-zero, then expanding the widget will allow you to see a breakdown of the exceptions by the top five services. This allows you to focus on the services with the greatest number of potential issues. You can then touch on a specific problem service and specify details by going to the enumeration page and changing the query values.
Suppose the Errors summary header on the WCF Call History widget contains a non-zero value signifying that some errors have occurred. You can expand the WCF Call History widget and look under the Service Exceptions - Grouped by Service (top 5) column to see the top five services that have encountered the most exceptions during the selected time period. You can also look under the Errors - Grouped by common types column to see a breakdown of the types of errors: Service Exceptions (caused mostly by failed or faulted calls) or User-defined Errors. To gain more details on the errors or exceptions, you can click one of the service links to take you to the Tracked Events page.
Note
If your Service X is one of the top five services, it will be displayed here. In that case, you can click its service link to take you to the Tracked Events page. If Service X is not one of the top five services you can click any of the metrics within the section to go to the Tracked Events page. After you are on that page, you can modify the existing query accordingly (that is, add the appropriate service name condition).
The Tracked Events page is populated with a historical view of the exceptions that occurred at the specific scope in the IIS hierarchy X. You can click one of the WCF Exception events from the list (still within the Tracked Events page) to display specifics of that exception in the Details pane at the bottom of the page. Within the Details pane you can view information about the exception within the Overview, Tracked Variables, or Errors tabs. The Overview tab contains general information about the event, such as the E2EActivityID (used for correlation in end-to-end tracking), EventType, ExceptionTypeName, EventSourceId, and other event-related information. The Tracked Variables tab shows the values of any explicitly tracked variables that are stored in the event. The Errors tab gives you exception information about the failure when available. You can use this information to better understand and troubleshoot WCF exceptions.
If you need additional context about the WCF exception you can right-click the event in the list and then click View Tracked WF Instance. This functionality is only available for WCF Service Exceptions emitted from Workflow Services, whereby the Tracked WF Instances page will display all available events for the parent instance. You can also select from the context menu to View All Related Events for a WCF call. This refreshes the Tracked Events page and populates it with all events related to the initial event.
Note
To leverage the View All Related Events option, the Monitoring level for the application must be set to End-to-End Monitoring or higher. This level tells the Monitoring infrastructure to collect transfer events that associate one end-to-end activity ID (E2EActivityId) to another.
Alternatively you can look at the Service Exceptions - Grouped by Service (top 5) column and view the number of either Calls Failed or Calls Faulted. The total number of Calls Failed or Calls Faulted contributes to the total number of WCF Exceptions as represented in the section summary pane.Clicking the link for either of these also takes you to the Tracked Events page. Depending on the link you clicked, you will see an enumerated list of events for WCF Failed Calls or WCF Faulted Calls, respectively. As described above, selecting one of the call events in the enumerated list yields additional information in the Details pane. You can also right-click and select either the View All Related Events or View Tracked WF Instance option if applicable.
For information about how to obtain more specific information about a tracked event to help you solve a problem, see Tracked Events Page.
Displaying WCF Error Information
AppFabric displays WCF-based error information only when the monitoring level of an application is configured to at least the End-to-End Monitoring level. This level captures WCF message flow between services, and the relationship between WF and WCF events. For monitoring levels below and including Health Monitoring, any WCF-based error information will not be displayed within the following sections:
The Errors or Service Exceptions columns within the WCF Call History section of the AppFabric Dashboard page.
The Errors tab in the Details pane on the Tracked Events page.
Any sections on the Tracked Events page displaying WCF events as a result of query criteria.
For instance, the Errors tab on the Tracked Events page uses WCF transfer events to gather exception information from the Service Model exception event that corresponds to a WCF failed or faulted call event. If the application being monitored is configured to use a monitoring level below the End-to-End Monitoring level, the WCF Transfer events will not be captured. This leads to no values being displayed on the Errors tab inside the Details pane.
If you use the End-to-End Monitoring level to capture transfer events, and don't want to collect additional WF events, here are the steps to ensure that this occurs:
Open the server root configuration file at C:\Windows\Microsoft.NET\Framework(64)\v4.0.xxx\CONFIG\Web.config.
Copy all the code between, and including, the
<trackingProfile name="HealthMonitoring Tracking Profile">and</trackingProfile>entries, to the clipboard. An example of what you will copy is as follows.<trackingProfile name="HealthMonitoring Tracking Profile"> <workflow activityDefinitionId="*"> <workflowInstanceQueries> <workflowInstanceQuery> <states> <state name="Started" /> <state name="Completed" /> <state name="Terminated" /> <state name="Canceled" /> <state name="Unsuspended" /> <state name="Persisted" /> <state name="Aborted" /> <state name="UnhandledException" /> </states> </workflowInstanceQuery> </workflowInstanceQueries> <activityStateQueries> <activityStateQuery activityName="*"> <states> <state name="Closed" /> </states> </activityStateQuery> </activityStateQueries> <faultPropagationQueries> <faultPropagationQuery faultSourceActivityName="*" faultHandlerActivityName="*" /> </faultPropagationQueries> <customTrackingQueries> <customTrackingQuery name="*" activityName="*" /> </customTrackingQueries> </workflow> </trackingProfile>Edit the application’s Web.config file. Ensure there is a complete and valid element nesting of
<system.serviceModel><tracking><profiles><trackingProfile>within which the contents of the<trackingProfile>element will be pasted in upcoming steps.At the start of the
<system.serviceModel><tracking><profiles>section, add<remove name=”EndToEndMonitoring Tracking Profile” />.In the same section, right below the
removetag just added, paste the code snippet copied to the clipboard in step 2.Replace the text
HealthMonitoring Tracking ProfilewithEndToEndMonitoring Tracking Profilein the application’s Web.config file. Save the file and exit the editor.Using the slider control within the Configure WCF and WF For Application dialog box, change the Application Monitoring Level to End-To-End. AppFabric will use the local End-to-End tracking profile entry from the application’s Web.config file, which is nothing more than the tracking profile for the Health Monitoring level disguised as an End-to-End tracking profile entry.