IaaS Usage and Service Reporting using System Center 2012 R2 and Windows Azure Pack

Following up Brad Anderson & Srikanth’s blog posts here & here, I would like to create a list of blog posts that describe how to deploy usage metering & reporting in System Center 2012 R2 & Windows Azure Pack for IaaS.

Some would ask why should I care about usage & service reporting for IaaS?

There are a couple of reasons why Service Reporting is important for the Cloud:

• Pay as you go & Pay only for what you use.

This supports the cloud business model where metering is one of the fundamentals of Cloud Computing.

• Elastic consumption.

This requires analytics at the macro and micro levels in order to meter at the tenant level as well as drill down to the single instance under a tenant.

To achieve this it requires significant effort and skills to gather, aggregate and summarize the right set of data. Microsoft is providing this capability as part of System Center 2012 R2.

In this series of blog posts I will focus on Usage and Metering for VM Clouds only, as this service seems to be area that most people start with. The process works very similar for other Windows Azure Pack services, once VM Cloud metering is up and running the, other service areas like Web Sites, MSSQL, MYSQL, ServiceBus could be a natural next step in the usage & metering process.

IaaS Service Reporting & Usage Metering

The overall solution contains the following:

Collecting the data:

• Collecting IaaS usage (CPU, Memory, Storage, Network).
• IaaS data comes from the Operations Manager DW through Service Provider Foundation (SPF).
• Windows Azure Pack provides consistent usage collection for every service offered all service providers such as Web Site, Virtual Machine, Service Bus, SQL and MYSQL servers.

Analyzing and Reporting on the usage data:

• Service Reporting analysis and Report on the collected data by using already existing Microsoft BI stack (SQL Server, Excel Power Pivot & SharePoint Performance Point).

Component overview and data flow.

In System Center 2012 R2 usage metering & reporting for IaaS (VM Cloud Service in Windows Azure Pack) involves the following list of components:

• Virtual Machine Manager 2012 R2 (VMM).
• Operations Manager 2012 R2 (OM).
• Service Provider Foundation (part of Orchestrator installation media) 2012 R2 (SPF).
• Windows Azure Pack (WAP).
• Service Reporting (part of Orchestrator installation media) (SR).
• SQL Server (2008 or 2012).

To dive a little deeper into the process flow, already described by Srikanth (here), you can divide it into the following three high level steps:

• Data collected by System Center (VMM + OM).
• Data collected by Windows Azure Pack (SPF + WAP).
• Data analyzed by Service Reporting (New component in System Center R2).

The illustration above shows how data flows from the different components in the process of usage, metering & reporting.

The following sections will explain each step in the process:

Data collected by System Center

System Center has valuable information about VM clouds. This data is collected by Virtual Machine Manager, in order to get this data aggregated and summarized for later usage by Windows Azure Pack the following needs to happen:

Process Flow

1. Virtual Machine Manager collects data for all VM and Hosts managed by Virtual Machine Manager. This includes metrics like CPU, Memory, Storage, Network, Started and Stopped. This data is only stored for one month by default in the VMM Database.

2. Virtual Machine Manager sends its performance data to Operations Manager by using the VMM to OM connector.

3. Operations Manager saves the raw data in the Operations Manager Data Warehouse.

4. Every hour Operations Manager Data Warehouse moves the raw data to the hourly dataset where it’s stored for one year by default. All the raw data is aggregated into min, max, avg & std deviation for each performance metric.

This process is described in the following picture:

Data collected by Windows Azure Pack (SPF + WAP)

Service Provider Foundation (SPF)

Service Provider Foundation exposes an extensible OData web service that interacts with Virtual Machine Manager (VMM) & Operations Manager (OM). This enables service providers and hosters to design and implement multi-tenant self-service & usage that integrate IaaS capabilities available in System Center 2012 R2. Service Provider Foundation provides usage metering data to any client, such as Service Management Portal and API, that wants to collect it. This data comprises all the usage metering metrics for all the virtual machines that a tenant uses, provided that those virtual machines are being monitored by System Center 2012 – Operations Manager and that the data is being stored in Operations Manager Data Warehouses. The role of Service Provider Foundation is to collect metrics from all the data warehouses and aggregate them for billing and analysis purposes.

WAP Usage Service

Inside Windows Azure Pack, the Usage Service consists of three components:

• Usage Collector
• Usage (REST API)
• Usage Database

Usage Collector

Usage collector consists of a scheduler that triggers the collection of information from the different resource providers in a circular fashion

Each resource provider has a list of data sets that the usage collector will collect.

The time span of the collection cycle is configurable, by default the usage collector spends a max of 3 minutes by default on every resource provider. So every collection cycle is 3 min * Number of Resource Providers in the System. I.e. (Resource Providers: VM, SQL Server, MySQL, WEB) *  3 minutes = 9 mins maximum for a cycle

The Usage system interacts with each resource provider one by one, if the provider returns with an empty response it moves to the next provider in the cycle.

Usage Database

Data from the usage collector stored in the central data repository. The data stored comes from the different resource providers and also contains relationships to tenants and plan.

Data is stored for 40 days by default

Usage

The Usage REST API exposes a REST endpoint for data access. This is used to allow access to the content in the usage database by billing providers and analytics services

The process is as follows for VM Clouds (IaaS) Resources:

1. Usage Collector (a sub component of Usage Service) collect usage data from every Resource Provider.

2. The Usage Collector for VM Cloud Resource collects needed data like CPU, memory and disk.

3. The Usage Collector interacts with the Service Provider Foundation (SPF) metering endpoint (Usage) to collect data for VM Clouds.

4. Service Provider Foundation usage metering module calls Operations Manager Data Warehouse and returns the data to the Usage Collector service.

5. The Usage Collector stores the data in the Usage Database.

Orange: Internal Modules

Yellow: REST endpoint

Blue: WAP / SPF Component

Green: WM Clouds Resource Providers and sub-components (SPF & System Center)

Service Management & Usage Data Model

The usage data model is used to store the usage data for the tenants consuming the Resource Provider services collected through Usage Collector.

This enables mapping between a Resource provider instance (e.g. VM) usage to be associated with a tenant (user) and the subscription that a given tenant signed up for.

For Service Management the following concepts are used:

Tenant: One or more individuals consuming services from the Windows Azure Pack via a subscription.

Usage Record: A data record that indicates the resource consumption per Tenant, per subscription and per provider.

Subscription: The mapping between a tenant and one or more plans.

Plan: Contains one or more services (Resource Providers: VM, DB, Web, Service Bus) offered and the quota for each service. Subscriptions, Add-on and Resource Providers are all associated to a plan.

Add-on (Plan Add-on): Allows customer upgrades on quota or add capabilities that can be combined with a plan (a service can be optional) to extend functionality or quota within a plan as an extra service.

Resource Providers: Services offered within Windows Azure Pack with mapping to resources (e.g. Web Sites or Virtual Machines).

The Usage Service collects data from Service Management module by using REST API and stores this data in the Usage Database

The usage database now contains both usage data from the different resource providers and the relation ships between tenants, plans and resource providers.

Data analyzed by Service Reporting

A new component introduced in System Center Orchestrator R2 is Service Reporting. This component does macro and microanalysis based on data collected from various resource providers in the Windows Azure Pack. (VM, DB, Service Bus & Web).

1. Service Reporting uses ETL (Extract, Transfer Load) standard to collect data (This is much similar to the way Service Manager Data Warehouse collect data from Service Manager DB).

2. The Extract process will contact the Windows Azure Pack Usage API to extract data.

3. WAP Usage API will return the data from the usage database to the extract process.

4. After completing the ETL process, the data is transferred and stored in Cubes for analytics purpose.

5. After this has completed, the data can be analyzed by Excel (PowerPivot) or SharePoint (PerformancePoint).

Below are two examples for usage data analyzed in Excel (PowerPivot) & SharePoint (PerformancePoint)

 

To summarize the flow the following diagram shows the process end to end:

I hope that this blog post provides a basic understanding of how Usage Metering & Reporting for IaaS works in System Center 2012 R2 and the Windows Azure Pack.

In upcoming blog posts, I will describe how each component described above can be configured to make use of IaaS usage and metering.

• • Configuring VMM and OM for IaaS usage and metering.
  • Configuring SPF and Windows Azure Pack for IaaS usage and metering.
  • Installing & configuring Service Reporting for IaaS usage and metering.  

I would like to thank the Windows Azure Pack and Reporting Services product groups for providing inputs and reviewing this blog post.

As always, I am interested in feedback and suggestions.

Comments

• Anonymous
  January 01, 2003
  Have the data set windows been reduced from daily to some number of minutes?  I had heard data sets for chargebacks are now available in 15 minute increments instead of daily.

• Anonymous
  January 01, 2003
  It depends on the kind of data being collected for VMs. Most performance data is every hour. VM start/stop is available in 15 minute granularity while the other metrics are at 1 hour fidelity. Hope this answers the question. Anders Ravnholt MSFT

• Anonymous
  December 12, 2013
  Is there have a real VM perforamnce monitoring solution provided by windows azure pack?

• Anonymous
  December 12, 2013
  The comment has been removed

• Anonymous
  December 20, 2013
  Pingback from Windows Azure Pack Blog Post Overview on Building Clouds & TechNet - Building Clouds Blog - Site Home - TechNet Blogs

• Anonymous
  December 29, 2013
  Microsoft’s very own Anders Ravnholt published a great step-by-step on how to configure Operations Manager

• Anonymous
  December 29, 2013
  Anders Ravnholt has a new article that describes how to deploy usage metering & reporting in System

• Anonymous
  January 06, 2014
  Pingback from Windows Azure Pack Blog Post Overview on Building Clouds & TechNet

• Anonymous
  January 06, 2014
  Pingback from IaaS Usage Metering & Reporting mit System Center 2012 R2 & Windows Azure Pack

• Anonymous
  March 04, 2014
  In this final section of the Success with Hybrid Cloud series, I want to get technical. The following

• Anonymous
  March 13, 2014
  Throughout this Success with Hybrid Cloud series, I’ve emphasized the importance of linking private and

• Anonymous
  March 25, 2014
  Microsoft’s very own Anders Ravnholt published a great step-by-step on how to configure Operations Manager

• Anonymous
  March 30, 2014
  Hi,
  In usage data only CPU|RAM|DISK are available what about network of Virtual VM ? how do we fetch that data?

• Anonymous
  April 01, 2014
  Where is this REST API documented?

• Anonymous
  May 20, 2014
  Pingback from Success with Hybrid Cloud: Getting Deep – Windows Azure Pack | Dot Net RSS

• Anonymous
  May 20, 2014
  Pingback from Success with Hybrid Cloud: Getting Deep – Windows Azure Pack | Dot Net RSS

• Anonymous
  May 20, 2014
  Pingback from Success with Hybrid Cloud: Getting Deep – Virtualization & Templates | Dot Net RSS