Application Analytics: What Every Developer Should Know

Sebastian Holst is responsible for product and market strategy at PreEmptive Solutions focusing on application analytics and security. In addition to his work building software, Sebastian has been active in computing and industry standards bodies as a member of the W3C Advisory Committee, the MMA committees on Privacy and on Analytics and as a co-founder the Compliance Consortium. Sebastian also hosts a blogspot, Applications are people too, where he writes about the parallels and foibles of people, programs and culture. In his spare time he develops mobile applications and works with high school students helping them to do the same.

July 2012

Sebastian Holst discusses the objectives and advantages of application analytics.

Applies To

Application Lifecycle Management, Visual Studio 2012, Team Foundation Server

Objectives

Requirements

Restrictions

Visual Studio 2012 and Application Analytics

Imagine how much more efficient your developers would be if they didn’t have to guess which features were being used and which could safely be deprecated? What would the impact on user satisfaction be if exception data replete with usage context were delivered before your users had a chance to complain? How would the quality of your software improve if your test plans were aligned with actual usage patterns and user preferences in production? Application analytics is the branch of analytics purpose-built to make these scenarios a reality; to satisfy the "selfish interests" application stakeholders, e.g. development, test, product owners, operations, etc.

Figure 1: Application analytics enhances both development and operations by providing deep insight into application adoption and user behavior within established development and operations platforms.

Application analytics integrates application usage data, app-centric analytics software and heuristics integrated into development and operations.

The diversity of today’s analytics solutions is (as it should be) customer driven. For example, web analytics’ principle customer is marketing and sales resulting in a focus on page views, clicks and conversions. Web analytics solutions share common:

• Objectives: monetization of web properties,

• Requirements: analysis of visitors, impressions, clicks and conversions, and

• Restrictions: meeting privacy and performance obligations.

In agile parlance, application analytics encompass analytics solutions where the primary customer is one or more application development "personas" who share common objectives, requirements and restrictions.

Objectives

Feedback-driven development

The Agile Manifesto states: that development's "highest priority is to satisfy the customer through early and continuous delivery of valuable software." In that context, development’s success can only be accurately measured where users and their applications meet – at the "point of work" (or play). Application analytics provides empirical evidence of application usage and end-user behavior that, when properly integrated into a development process, provides:

• Insight into user requirements,

• Validation of development priorities and an

• Objective measure of test plan accuracy and completeness

Examples include:

• The Microsoft customer experience improvement program (CEIP) was "created to give all Microsoft customers the ability to contribute to the design and development of Microsoft products." CEIP collects information about how Microsoft programs are being used "in the wild".

DevOps quality

The Agile Manifesto also states that "working software is the primary measure of progress." Operations’ mission is to get the most out of today’s applications – future application iterations cannot address immediate stability, performance, user experience, or security concerns. Application analytics, when properly integrated into operations and support, provides:

1. Application adoption and usage metrics within a specific operations framework,

2. Production incident alerts from application exceptions, and

3. Organizational adoption and productivity analysis connecting application investment to enterprise ROI.

Examples include:

• PreEmptive Analytics Community Edition that gives developers using Microsoft Visual Studio 2012 Professional the ability to create their own CEIP by allowing development and operations to identify and quickly respond to application exceptions that occur in production.

Requirements

Given these objectives, the value of application analytics seem obvious, but the details can make it difficult. Collecting, analyzing and acting on application runtime data poses unique challenges both in terms of the kinds of data that need to be gathered and the metrics that measure success.

Effective application analytics implementations must accommodate the diversity of today’s applications and the emergence of cloud, mobile and distributed computing platforms. The following application analytics requirements make plain why narrower analytics technologies should never be expected to fully satisfy development objectives.

Runtime telemetry

The runtime data that streams from an application is typically far more complex and heterogeneous than what is typically streamed from a web page or portal.

Data types	Runtime telemetry: the variety, semantics and location of application runtime data
Feature	An application feature is not a click. A feature can span one or more methods, incorporate multiple components, run across runtime surfaces and even be implemented multiple times in different languages, e.g. Windows Presentation Foundation (WPF), Microsoft Silverlight and HTML5. Measuring usage and performance of an arbitrarily defined scope is required for monitoring across devices and platforms.
Application Data	Many of today’s applications are data-driven where the actual behavior itself is encoded in the data. Knowing what templates, workflows and other "modern" content is being processed can be more valuable than knowing which workflow or rendering engine processed that data.
Session	Session information can be defined differently within an app server, a mobile session, within a browser or distributed across all of the above serviced by a cloud-based service.
Event	Unhandled exceptions, caught and thrown exceptions, unexpected performance or suspicious user behavior can all constitute a "production event."
Application	Applications are often comprised of multiple components – some on-premises and some service-based. These applications (and their components) are versioned at unpredictable cadences. Calculating the workflow across distributed applications and then reconciling this activity over time and across versions is an applications analytics requirement.
Stack	While many applications are run inside a "sandbox", e.g. mobile, Windows Runtime, Windows Azure – many other applications have full access to the underlying OS and computing metal. Tracking screen resolution, chip manufacturers and hardware availability is often essential to understanding user experience and application behavior.
Identity	A users’ identity can be defined and tracked by device ID, IP address, user credentials, software license and more. Application analytics solutions must have the capacity to enforce privacy and security policies at both client and aggregate levels. Ensuring effective data governance is a precursor to effectively analyzing the resulting runtime data.

Runtime architectures and technologies

Given the complexity, diversity and distribution of today’s production platforms, it is no longer possible to simulate production. Application analytics can fill this gap only if there is comprehensive support for today’s computing platforms.

Categories	Runtime architectures and technologies: existing and emerging languages and platforms
Architectures and surfaces	Applications are much more than a simple presentation layer and a sequence of user actions. Instrumentation must span client-server, cloud services (public and private), web servlet and mobile platforms, architectures and surfaces.
Languages and runtimes	Today’s applications will incorporate managed, native and scripted components including the Microsoft .NET Framework, C++, Java and JavaScript.

IDE and ALM integration

For application analytics to have impact, the right information has to be delivered to the right roles at the right time and in the right context. This includes integrating the instrumentation task within the development and build process and surfacing application analytics inside the development, testing, deployment, and management phases.

Figure 2: Five functional phases of application analytics implementation.

DevOps phase	IDE & application lifecycle management (ALM) integration: role and use case driven scenarios
1. Instrumentation	Instrumentation is the logic inside an application that creates the runtime data to be analyzed. Instrumentation can be coded via an API (required for native and scripted apps) or injected post-compile into managed assemblies.
2. Build and deploy	Apps can be built manually, as part of a continuous build process and automated to support cloud platforms. Support for the various manufacturing processes and payload formats is required to ensure efficient and scalable deployments.
3. Runtime data management	Managing runtime data will require scale, governance, and security controls. Application runtime data management requirements shift dramatically across industry, use case and jurisdictional boundaries.
4. Runtime data publishing	Different stakeholders require distinct presentations and analysis; developers, architects, product owners and line of business management bring different perspectives and priorities to the same underlying runtime data. Reports, dashboards, export and programmatic access are all typically required when use cases span sprint planning, customer support and business performance monitoring.
5. Integration	Integration of application analytics into development platforms, e.g. Visual Studio and TFS, operations, e.g. operations manager, and customer relationship management, e.g. Microsoft Dynamics through reports and event scheduling delivers the "last mile" of application analytics’ productivity.

Restrictions

The cure cannot be worse that the disease. For application analytics, this means that the inclusion of application analytics into development and operations cannot result in productivity, performance, security or user experience risks greater than those that it is designed to mitigate. Given the many forms and roles that today’s applications can take, this is no small task.

Risk management	Restrictions: performance, stability, privacy, and complexity
Performance and stability	Collecting, caching, and transmitting runtime data efficiently across devices without performance penalties (when working properly) while not impacting application stability or user experience if/when one or more aspects of the analytics solution should fail. This can be especially challenging when considering special dependencies such as battery life, data plans, network characteristics…
Security and privacy	Consumer, business-to-business, and line-of-business applications each come with their own security and privacy obligations. These obligations are further fragmented by industry and by jurisdiction. Application analytics instrumentation, transport and content management must be extensible and able to enforce these requirements on an application-by-application basis.
Complexity	Complexity introduces waste and risk – and ultimately resistance to adoption. Integration into existing platforms, processes and methodologies is a requisite to effective application analytics implementations.

Visual Studio 2012 and Application Analytics

Exception analytics with Visual Studio 2012 PreEmptive Analytics CE

Visual Studio 2012 includes PreEmptive Analytics for TFS Community Edition (PA for TFS CE), an application analytics solution that monitors exceptions and creates or updates work items inside Team Foundation Server (TFS) based upon user-defined thresholds.

PA for TFS CE is designed to track unhandled exceptions on applications running on the .NET Framework and Java runtimes. Support for the five phases of application analytics as follows:

DevOps phase	PreEmptive Analytics for TFS Community Edition
1. Instrumentation	Instrumentation is accomplished with Dotfuscator Community Edition. Unhandled exception monitoring with optional opt-in and user-feedback is supported for the .NET Framework, Silverlight, Microsoft Windows Phone and XNA applications. An API for Java applications is also available as a free download that includes support for Android. Support for native code, JavaScript and Java is provided by PreEmptive Solutions.
2. Build and deploy	Dotfuscator Community Edition is interactive. The command line interface and MSBuild support is available with Dotfuscator Professional from PreEmptive Solutions.
3. Runtime data management	A server-side data collector is included withVisual Studio Team Foundation Server 2012. The collector endpoint is referenced via a URL embedded into the application being monitored as part of the instrumentation phase above. The collector can sit next to the TFS server, on an entirely different server, and even inside Microsoft Windows Azure.
4. Runtime data publishing	An aggregator service is also included with TFS in Visual Studio 2012 that polls the collector endpoint. When user-defined thresholds are met, the aggregator will create (or update) a Production Incident work item inside TFS in Visual Studio 2012.
5. Integration	In Visual Studio 2012, TFS work items created by PA for TFS CE are tracked, assigned, prioritized and reported against as any other first class work item type.

Figure 3: Finding PreEmptive Analytics inside Visual Studio 2012 off of the tools menu

Figure 4: Instrumentation. Inside Dotfuscator CE, adding the setup attribute identifies the collector endpoint for runtime data. The endpoint can be on-premises next to a TFS server or hosted on Windows Azure remotely.

Figure 5: Visual Studio 2012 integration. Production Incident work items are surfaced inside Visual Studio automatically when volume thresholds are met. In this "All Items" query, you can see the type of exception, how many exceptions of this type have been detected and on how many machines. You can see more detail on this work item below including a stack track as well as the work item’s assignment, prioritization and classification.

Figure 6: Reporting. One example of summary charts included with PA for TFS CE shows the status of all open incidents.

PreEmptive Analytics Professional Edition

In addition to enhanced TFS integration and instrumentation options, the Professional Edition of PreEmptive Analytics includes feature, session and user analytics designed to measure trends, usage patterns and user preferences throughout the lifetime of a production application.

Primary use cases for PreEmptive Analytics Community Edition and Professional

Use case	Community Edition	Professional
Track unhandled on .NET Framework and Java applications	Yes	Yes
Automatically create and update TFS work items in Visual Studio 2012	Yes	Yes
Provide opt-in and user-feedback options at runtime	Yes	Yes
Support Visual Studio 2010		Yes
Track caught and thrown exceptions		Yes
Support custom data and extensible rule and work item definitions		Yes
Support JavaScript and native application monitoring		Yes
Measure feature and session usage		Yes

• Development has unique requirements that are not being met by web, BI or other non-development centered analytics solutions.

• Application analytics offers specific capabilities designed to meet development and operation’s needs.

• Visual Studio 2012 offers integrated application analytics "out of the box" with opportunities to extend these capabilities through integration and partner options.

You can visit these sites to learn more:

1. http://www.preemptive.com/pa

2. https://www.microsoft.com/visualstudio/11/en-us/products/alm

3. https://blogs.msdn.com/b/bharry/archive/2012/04/11/preemptive-analytics-in-visual-studio-and-tfs-11.aspx