Joseph Merrell - BI/SQL Consultant @ Microsoft

Trending with Temporal tables

Joseph Merrell — Wed, 07 Jun 2017 11:12:07 GMT

Trending is a very important component for BI work. For a customer, it's important know where they've been relative to where they are now; as well as help indicate where they're headed. In the past, this was typically accomplished by storing historical records in the same table or in history tables. This certainly is an acceptable option, but it does require management within the code base to know where and how to retrieve the historical data. Wouldn't it be nice to have SQL Server intrinsically manage that historical data? Well, beginning with SQL Server 2016, Temporal tables were introduced as means to provide system versioning for data. In this post, I'll discuss using Temporal tables to enable trending capabilities. While the scope of this post is to cover the use of Temporal tables, it's important to also cover some of the basic requirements of Temporal tables in an effort to level-set knowledge. As defined in the Microsoft docs (https://docs.microsoft.com/en-us/sql/relational-databases/tables/temporal-tables):

Temporal tables must have a Primary Key defined.
Temporal tables must have only 1 PERIOD FOR SYSTEM_TIME defined with two datetime2 columns, declared as GENERATED ALWAYS AS ROW START / END
The history table must always be schema-aligned with the current or temporal table, in terms of number of columns, column names, ordering and data types.

It's probably a good idea to review all of the official Microsoft docs for Temporal tables (https://docs.microsoft.com/en-us/sql/relational-databases/tables/temporal-tables). This will be very helpful in understanding all the different permutations of this feature that are well beyond the scope of this post. Now on to the good stuff...In my blog series "Reporting on SCOM Data", I covered how we're reporting current SCOM Health State and Alert counts. This gives us a nice snapshot of what's going on now. What it doesn't do is provide a history of Health State or Alert counts. What if we wanted to see our Health State or Alert counts from 2 weeks back? What if we wanted to see how our Health and/or Alert counts have changed over the last 2 weeks? To support this, the data would need to be maintained over time rather than just a current snap shot. In the past, we would manually maintain that history. However, with the introduction of Temporal tables, that functionality is provided with an out-of-the-box feature. I've modified the Host table from my post Reporting on SCOM Data: Piecing the data together (Part 1) for simplification and to include our new datetime2 columns. As you can see, the T-SQL is very similar to the creation of a normal table.

CREATE TABLE Host (
    [HostId] [int] IDENTITY(1,1) NOT NULL PRIMARY KEY CLUSTERED,
    [DnsHostName] [nvarchar](255) NULL,
    [EntityHealth] [tinyint] NULL,
    [AvailabilityHealth] [tinyint] NULL,
    [ConfigurationHealth] [tinyint] NULL,
    [PerformanceHealth] [tinyint] NULL,
    [SecurityHealth] [tinyint] NULL,
    [WarningOpenAlertCount] [int] NULL,
    [CriticalOpenAlertCount] [int] NULL,
    [SysStartTime] [datetime2] GENERATED ALWAYS AS ROW START NOT NULL,
    [SysEndTime] [datetime2] GENERATED ALWAYS AS ROW END NOT NULL,
    PERIOD FOR SYSTEM_TIME (SysStartTime, SysEndTime)
)
WITH (
    SYSTEM_VERSIONING = ON (HISTORY_TABLE = dbo.HostHistory)
);

Here's what the new table looks like in SSMS. The current (Temporal) "Host" table is indicated as "System-Versioned" and has a new child node called "HostHistory". This is the history table. We've now got a structure that we can use to store data. Let's add some rows to it.

INSERT INTO 
    Host (DnsHostName, EntityHealth, AvailabilityHealth, ConfigurationHealth, PerformanceHealth, SecurityHealth, WarningOpenAlertCount, CriticalOpenAlertCount)
VALUES
    ('HOST1.DOMAIN.COM', 1, 1, 1, 1, 1, 0, 0)

INSERT INTO 
    Host (DnsHostName, EntityHealth, AvailabilityHealth, ConfigurationHealth, PerformanceHealth, SecurityHealth, WarningOpenAlertCount, CriticalOpenAlertCount)
VALUES
    ('HOST2.DOMAIN.COM', 1, 1, 1, 1, 1, 0, 0)

INSERT INTO 
    Host (DnsHostName, EntityHealth, AvailabilityHealth, ConfigurationHealth, PerformanceHealth, SecurityHealth, WarningOpenAlertCount, CriticalOpenAlertCount)
VALUES
    ('HOST3.DOMAIN.COM', 1, 1, 1, 1, 1, 0, 0)

INSERT INTO 
    Host (DnsHostName, EntityHealth, AvailabilityHealth, ConfigurationHealth, PerformanceHealth, SecurityHealth, WarningOpenAlertCount, CriticalOpenAlertCount)
VALUES
    ('HOST4.DOMAIN.COM', 1, 1, 1, 1, 1, 0, 0)

With data now added to the Host table, let's take a look at the records in both tables. Selecting directly from each table individually is exactly the same as a normal (non system-versioned) table.

SELECT
    HostId,
    DnsHostName,
    EntityHealth,
    AvailabilityHealth,
    ConfigurationHealth,
    PerformanceHealth,
    SecurityHealth,
    WarningOpenAlertCount,
    CriticalOpenAlertCount,
    SysStartTime,
    SysEndTime
FROM
    Host

SELECT
    HostId,
    DnsHostName,
    EntityHealth,
    AvailabilityHealth,
    ConfigurationHealth,
    PerformanceHealth,
    SecurityHealth,
    WarningOpenAlertCount,
    CriticalOpenAlertCount,
    SysStartTime,
    SysEndTime
FROM
    HostHistory

As you can see, we've only got records in the current (Temporal) Host table. Note the "SysEndTime" column values of "9999-12-31 23:59:59.999999". This value indicates that this record is the current version of that record. This value comes in handy when we sort on all the current and historical versions. We'll see this later. Now let's update a couple of the Host records and check the tables again.

UPDATE Host SET WarningOpenAlertCount = 3 WHERE HostId = 2
UPDATE Host SET CriticalOpenAlertCount = 5 WHERE HostId = 3

Using the same SELECT statements as before, here are what the records look like. In the second table, you can see the previous versions of the rows that we updated. These new records in that table demonstrate the intrinsic versioning functionality of Temporal tables. Querying the Temporal and history tables directly is great, but we want to return history so we can report trends on that data. Let's update the same records again so we'll have a little more data to report on.

UPDATE Host SET WarningOpenAlertCount = 4 WHERE HostId = 2
UPDATE Host SET CriticalOpenAlertCount = 8 WHERE HostId = 3

Here's what that SELECT statement would look like to retrieve the current version and all historical versions of the records for Hosts 2 & 3.

SELECT
    HostId,
    DnsHostName,
    EntityHealth,
    AvailabilityHealth,
    ConfigurationHealth,
    PerformanceHealth,
    SecurityHealth,
    WarningOpenAlertCount,
    CriticalOpenAlertCount,
    SysStartTime,
    SysEndTime
FROM
    Host FOR SYSTEM_TIME ALL
WHERE
    HostId = 2
    OR HostId = 3
ORDER BY
    HostId,
    SysStartTime,
    SysEndTime

Above you can see the current and all of the historical records for each Host. Note how the SysEndTime values I mentioned earlier in the post sort exactly how we would want with our ORDER BY clause. The best part of all is that we didn't have to manage additional tables or records to leverage this functionality. In our example, we're only looking at the Open Alert Counts, but the same techniques could be used for any of the Health States as well. Taking things further, building a visualization for this data would be fairly simple. An example in Power BI might look something like this. While we are using a small sampling of data for our example, hopefully this gives you enough of an idea of how you can leverage Temporal tables to provide trending capabilities for the data that you work with. Viel Spaß! Joseph

Reporting on SCOM Data: Piecing the data together (Part 2)

Joseph Merrell — Mon, 05 Jun 2017 17:01:19 GMT

In my last blog post, Reporting on SCOM Data: Piecing the data together (Part 1), I discussed collecting host details from the SCOM DW db. This provided a great foundation for correlating Health State and Alert data into the existing hierarchy that our customer wanted to see. In this post, I'll focus on querying the Health State and Alert aggregates by host and updating our existing tables to enable reporting on that data. This data is updated every 10 minutes in our reporting database. Again, a huge shout out to Jimmy Harper for his help with the queries and imparting his extensive SCOM knowledge on me. :) Here are the steps that we followed to populate the Health State and Alert counts for each host:

Populate the Host_HealthLoad table with current Health State (from the SCOM OperationsManager db) for each host.
Populate the Host_AlertCountLoad table with the current count of Warning and Critical Alerts (from the SCOM DW db) for each host.
Update the Host_HealthLoad table with on Alert counts from the Host_AlertCountLoad table.
Update the Host table with current Health State and Alert Counts from the Host_HealthLoad table.

First, we need our Host_HealthLoad table. This is our loading/staging table for Health State & Alert Counts for each host.

CREATE TABLE Host_HealthLoad (
    [HealthLoadId] [int] IDENTITY(1,1) NOT NULL,
    [DnsHostName] [nvarchar](255) NULL,
    [EntityHealth] [tinyint] NULL,
    [AvailabilityHealth] [tinyint] NULL,
    [ConfigurationHealth] [tinyint] NULL,
    [PerformanceHealth] [tinyint] NULL,
    [SecurityHealth] [tinyint] NULL,
    [WarningOpenAlertCount] [int] NULL,
    [CriticalOpenAlertCount] [int] NULL
)

To populate the Host_HealthLoad table with Health State data, we used the following query against the SCOM OperationsManager db in an SSIS package scheduled every 10 minutes (all other queries in this blog are tasks in the same SSIS package). This query leverages the State table and MonitorView view to retrieve the current Health State of specific Monitor Categories. In our case, we're looking for following monitor categories for each host: "Entity Health" (which is used to display Overall Health of the host), "Availability", "Configuration", "Performance", and "Security". We're also setting WarningOpenAlertCount and CriticalOpenAlertCount equal to zero (0) in this query since we're populating them from the SCOM DW db (more on that in a bit). It's also important to note that we're truncating the Host_HealthLoad table immediately before inserting this results of this query into it.

SELECT
    DnsHostName = bme.DisplayName,
    EntityHealth = 
                    (SELECT TOP 1
                        s.HealthState
                    FROM
                        State s WITH (NOLOCK)
                    INNER JOIN
                        MonitorView mv WITH (NOLOCK)
                        ON mv.Id = s.MonitorId
                    WHERE
                        s.BaseManagedEntityId = bme.BaseManagedEntityId
                        AND mv.DisplayName = 'Entity Health'),
    AvailabilityHealth = 
                    (SELECT TOP 1
                        s.HealthState
                    FROM
                        State s WITH (NOLOCK)
                    INNER JOIN
                        MonitorView mv WITH (NOLOCK)
                        ON mv.Id = s.MonitorId
                    WHERE
                        s.BaseManagedEntityId = bme.BaseManagedEntityId
                        AND mv.DisplayName = 'Availability'),
    ConfigurationHealth = 
                    (SELECT TOP 1
                        s.HealthState
                    FROM
                        State s WITH (NOLOCK)
                    INNER JOIN
                        MonitorView mv WITH (NOLOCK)
                        ON mv.Id = s.MonitorId
                    WHERE
                        s.BaseManagedEntityId = bme.BaseManagedEntityId
                        AND mv.DisplayName = 'Configuration'),
    PerformanceHealth = 
                    (SELECT TOP 1
                        s.HealthState
                    FROM
                        State s WITH (NOLOCK)
                    INNER JOIN
                        MonitorView mv WITH (NOLOCK)
                        ON mv.Id = s.MonitorId
                    WHERE
                        s.BaseManagedEntityId = bme.BaseManagedEntityId
                        AND mv.DisplayName = 'Performance'),
    SecurityHealth = 
                    (SELECT TOP 1
                        s.HealthState
                    FROM
                        State s WITH (NOLOCK)
                    INNER JOIN
                        MonitorView mv WITH (NOLOCK)
                        ON mv.Id = s.MonitorId
                    WHERE
                        s.BaseManagedEntityId = bme.BaseManagedEntityId
                        AND mv.DisplayName = 'Security'),
    WarningOpenAlertCount = 0,
    CriticalOpenAlertCount = 0
FROM
    BaseManagedEntity bme WITH (NOLOCK)
INNER JOIN
    ManagedType mt WITH (NOLOCK)
    ON mt.ManagedTypeId = bme.BaseManagedTypeId
WHERE
    bme.IsDeleted = 0
    AND mt.TypeName = 'Microsoft.Windows.Computer'

Next, we'll need to populate our Host_AlertCountLoad table. Here's the schema for that table:

CREATE TABLE Host_AlertCountLoad (
    [AlertCountLoadId] [int] IDENTITY(1,1) NOT NULL,
    [DnsHostName] [nvarchar](255) NULL,
    [Severity] [tinyint] NULL,
    [AlertCount] [int] NULL
)

To improve efficiency of the query, we decided to UNION the results for Warning and Critical Alert counts that we're retrieving from the SCOM DW db. This is also the reason that we update the Host_HealthLoad table with these counts after we've inserted them into the Host_AlertCountLoad table. In addition, by breaking out the Health State data load from the Alert data load, we were able to support pulling these data sets from different data sources (e.g. - OperationsManager db & DW db). This query leverages the vAlert, vAlertResolutionState, and vManagedEntity views. The results are inserted into the Host_AlertCountLoad table. Like the Host_HealthLoad table, it's important to note that we're truncating the Host_AlertCountLoad table immediately before inserting this results of this query into it.

DECLARE @ParamDuration INT = 7 -- we only go back 7 days
SELECT
    DnsHostName = a.Path,
    Severity = 1,
    AlertCount = COUNT(a.AlertGuid)
FROM
    (SELECT
        me2.Path,
        a.AlertGuid,
        a.AlertName,
        a.Severity,
        a.DBCreatedDateTime
    FROM
        Alert.vAlert a WITH (NOLOCK)
    INNER JOIN
        (SELECT
            ars.AlertGuid,
            CurrentResolutionState = MAX(ars.ResolutionState)
        FROM
            Alert.vAlertResolutionState ars WITH (NOLOCK)
        GROUP BY
            ars.AlertGuid) crs
        ON crs.AlertGuid = a.AlertGuid
    INNER JOIN
        vManagedEntity me2 WITH (NOLOCK)
        ON me2.ManagedEntityRowId = a.ManagedEntityRowId
    WHERE
        a.Severity = 1
        AND a.DBCreatedDateTime > (GETUTCDATE() - @ParamDuration)
        AND crs.CurrentResolutionState = 0) a
GROUP BY
    a.Path
UNION
SELECT
    DnsHostName = a.Path,
    Severity = 2,
    AlertCount = COUNT(a.AlertGuid)
FROM
    (SELECT
        me2.Path,
        a.AlertGuid,
        a.AlertName,
        a.Severity,
        a.DBCreatedDateTime
    FROM
        Alert.vAlert a WITH (NOLOCK)
    INNER JOIN
        (SELECT
            ars.AlertGuid,
            CurrentResolutionState = MAX(ars.ResolutionState)
        FROM
            Alert.vAlertResolutionState ars WITH (NOLOCK)
        GROUP BY
            ars.AlertGuid) crs
        ON crs.AlertGuid = a.AlertGuid
    INNER JOIN
        vManagedEntity me2 WITH (NOLOCK)
        ON me2.ManagedEntityRowId = a.ManagedEntityRowId
    WHERE
        a.Severity = 2
        AND a.DBCreatedDateTime > (GETUTCDATE() - @ParamDuration)
        AND crs.CurrentResolutionState = 0) a
GROUP BY
    a.Path

Now that we have the Alert counts in the Host_AlertCountLoad table, we simply update the Host_HealthLoad table with those results.

BEGIN TRAN
UPDATE
    hl
SET
    hl.WarningOpenAlertCount = 
        ISNULL((SELECT
            SUM(acl.AlertCount)
        FROM
            Host_AlertCountLoad acl
        WHERE
            acl.ServerName = hl.DnsHostName
            AND acl.Severity = 1), 0)
FROM
    Host_HealthLoad hl
    
UPDATE
    hl
SET
    hl.CriticalOpenAlertCount = 
        ISNULL((SELECT
            SUM(acl.AlertCount)
        FROM
            Host_AlertCountLoad acl
        WHERE
            acl.ServerName = hl.DnsHostName
            AND acl.Severity = 2), 0)
FROM
    Host_HealthLoad hl
COMMIT TRAN

With a loaded and updated Host_HealthLoad table, we simply update the Host table with our current values.

UPDATE 
    h
SET
    h.EntityHealth = l.EntityHealth,
    h.AvailabilityHealth = l.AvailabilityHealth,
    h.ConfigurationHealth = l.ConfigurationHealth,
    h.PerformanceHealth = l.PerformanceHealth,
    h.SecurityHealth = l.SecurityHealth,
    h.WarningOpenAlertCount = l.WarningOpenAlertCount,
    h.CriticalOpenAlertCount = l.CriticalOpenAlertCount
FROM
    Host h WITH (NOLOCK)
INNER JOIN
    Host_HealthLoad l WITH (NOLOCK)
    ON l.DnsHostName = h.DnsHostName

We've now got the data to support a picture of SCOM Health State and Alert counts across the entire SCOM enterprise being updated every 10 minutes. You can review my first blog post (Reporting on SCOM Data: A Single Pane of Glass) in this series for more details, but here are the types of visualizations that we created using this data. Health State Map Drill down for Health State by Monitor Category Host-level details for Health State and Alert count, linking off to the SCOM Web Console for more in-depth Alert and Performance Monitoring details. In summary, we've been able to provide a "Single Pane of Glass" for Health State and Alert data across our customer's enterprise. These reports have been incredibly well received by all levels of the customer and has enabled us to increase our new work efforts with them. I hope you've found these posts helpful and hope to hear from some of you that may have implemented some or all of the solution, along with any improvements that you've made. Viel Spass! Joseph

Reporting on SCOM Data: Piecing the data together (Part 1)

Joseph Merrell — Thu, 18 May 2017 00:32:57 GMT

In my last post, Reporting on SCOM Data: Single Pane of Glass, I provided an overview of what we provided our customer that enabled them to view the Health State of their entire enterprise in one location. In this post, I'm going to begin covering how we pulled together the data to present those views. Special shout out to Jimmy Harper for help with the SCOM queries. Jimmy is the best SCOM guy that I know, so if you're into SCOM then you might want to check out his blog. Our customer wanted to see data aggregated from the host/server level up to the Region and Office level. They had a predetermined list of Regions and Offices that we loaded into some related tables and were able to generate our PK's for each of those entities. Those tables are fairly simple in nature:

CREATE TABLE Office (
 [OfficeId] [int] IDENTITY(1,1) NOT NULL,
 [OfficeName] [nvarchar](max) NULL,
 [TimeAdded] [datetime] NOT NULL,
 [Latitude] [numeric](18, 6) NULL,
 [Longitude] [numeric](18, 6) NULL,
)

CREATE TABLE Region (
 [RegionId] [int] IDENTITY(1,1) NOT NULL,
 [RegionName] [nvarchar](max) NULL,
)

CREATE TABLE RegionOffice (
 [RegionOfficeId] [int] IDENTITY(1,1) NOT NULL,
 [RegionId] [int] NOT NULL,
 [OfficeId] [int] NOT NULL,
)

In addition to the Region and Office tables that we want to roll up with, we need to define the Host table and its load table.

CREATE TABLE Host (
 [HostId] [int] IDENTITY(1,1) NOT NULL,
 [OfficeId] [int] NOT NULL,
 [DnsHostName] [nvarchar](255) NULL,
 [OU] [nvarchar](max) NULL,
 [ADSiteName] [nvarchar](128) NULL,
 [TimeAdded] [datetime] DEFAULT GETUTCDATE(),
 [EntityHealth] [tinyint] NULL,
 [AvailabilityHealth] [tinyint] NULL,
 [ConfigurationHealth] [tinyint] NULL,
 [PerformanceHealth] [tinyint] NULL,
 [SecurityHealth] [tinyint] NULL,
 [WarningOpenAlertCount] [int] NULL,
 [CriticalOpenAlertCount] [int] NULL,
 [BaseManagedEntityId] [uniqueidentifier] NULL,
 [ManagementGroup] [nvarchar](128) NULL,
 [DWManagedEntityRowId] [int] NULL
)

CREATE TABLE Host_Load (
 [HostLoadId] [int] IDENTITY(1,1) NOT NULL,
 [OfficeId] [int] NULL,
 [DnsHostName] [nvarchar](255) NULL,
 [OU] [nvarchar](max) NULL,
 [ADSiteName] [nvarchar](128) NULL,
 [BaseManagedEntityId] [uniqueidentifier] NULL,
 [ManagementGroup] [nvarchar](128) NULL,
 [DWManagedEntityRowId] [int] NULL
)

With the custom hierarchy tables created (and loaded where necessary), we need the list of hosts from SCOM. In this case, we queried host information out of the SCOM DW db by selecting from the vManagedEntity view and joining it with vManagedEntityType, vManagementGroup and vManagedEntityManagementGroup views. In addition, we performed subqueries against vManagedEntityPropertySet and vManagedEntityTypeProperty views to return specific host details like ADSiteName, OU, DnsHostName, etc. It's important to note that our customer has multiple SCOM Management Groups so we joined with the aforementioned ManagementGroup views to pull back Management Group details (namely ManagementGroupDefaultName and ToDateTime). Here is a snippet of that main query:

SELECT
 me.ManagedEntityRowId,
 me.TopLevelHostManagedEntityRowId,
 BaseManagedEntityId = me.ManagedEntityGuid,
 ServerName = me.Name, 
 ManagementGroup = mg.ManagementGroupDefaultName,
 ADSiteName = 
  (SELECT TOP 1
   ADSiteName = ps_ADSite.PropertyValue
  FROM 
   vManagedEntity me2 WITH (NOLOCK)
  INNER JOIN 
   vManagedEntityPropertySet ps_ADSite WITH (NOLOCK) 
   ON ps_ADSite.ManagedEntityRowId = me2.ManagedEntityRowId
  INNER JOIN 
   vManagedEntityTypeProperty tp_ADSite WITH (NOLOCK) 
   ON tp_ADSite.PropertyGuid = ps_ADSite.PropertyGuid 
    AND tp_ADSite.PropertySystemName = 'ActiveDirectorySite'
  WHERE 
   me2.ManagedEntityRowId = me.ManagedEntityRowId
   AND ps_ADSite.ToDateTime IS NULL),
 DomainDNSName = 
  (SELECT TOP 1
   DomainDNSName = ps_DomainDNSName.PropertyValue
  FROM 
   vManagedEntity me2 WITH (NOLOCK)
  INNER JOIN 
   vManagedEntityPropertySet ps_DomainDNSName WITH (NOLOCK) 
   ON ps_DomainDNSName.ManagedEntityRowId = me2.ManagedEntityRowId
  INNER JOIN 
   vManagedEntityTypeProperty tp_DomainDNSName WITH (NOLOCK) 
   ON tp_DomainDNSName.PropertyGuid = ps_DomainDNSName.PropertyGuid 
    AND tp_DomainDNSName.PropertySystemName = 'DomainDNSName'
  WHERE 
   me2.ManagedEntityRowId = me.ManagedEntityRowId
   AND ps_DomainDNSName.ToDateTime IS NULL),
 OrganizationalUnit =
  (SELECT TOP 1
   OrganizationalUnit = ps_OrganizationalUnit.PropertyValue
  FROM 
   vManagedEntity me2 WITH (NOLOCK)
  INNER JOIN 
   vManagedEntityPropertySet ps_OrganizationalUnit WITH (NOLOCK) 
   ON ps_OrganizationalUnit.ManagedEntityRowId = me2.ManagedEntityRowId
  INNER JOIN 
   vManagedEntityTypeProperty tp_OrganizationalUnit WITH (NOLOCK) 
   ON tp_OrganizationalUnit.PropertyGuid = ps_OrganizationalUnit.PropertyGuid 
    AND tp_OrganizationalUnit.PropertySystemName = 'OrganizationalUnit'
  WHERE 
   me2.ManagedEntityRowId = me.ManagedEntityRowId
   AND ps_OrganizationalUnit.ToDateTime IS NULL)
FROM 
 vManagedEntity me WITH (NOLOCK)
INNER JOIN 
 vManagedEntityType met WITH (NOLOCK) 
 ON met.ManagedEntityTypeRowId = me.ManagedEntityTypeRowId 
  AND met.ManagedEntityTypeSystemName = 'Microsoft.Windows.Computer'
INNER JOIN 
 vManagementGroup mg WITH (NOLOCK) 
 ON mg.ManagementGroupRowId = me.ManagementGroupRowId
INNER JOIN 
 vManagedEntityManagementGroup memg WITH (NOLOCK) 
 ON memg.ManagedEntityRowId = me.ManagedEntityRowId
WHERE 
 memg.ToDateTime IS NULL
ORDER BY
 me.Name

Using the above query in an SSIS package, we loaded the results into the Host_Load table. Using our customer's host naming convention, we were able to determine which Office each host is assigned to. There's some customer-specific formulas to make those determinations, so in the effort of privacy I'm leaving those details out. However, using those formulas, we were able to update the OfficeId field for each host in the load table. From there we used a MERGE statement to merge the records with the existing host table.

MERGE INTO 
 Host AS h
USING 
 Host_Load  AS l
ON 
 h.DnsHostName = l.DnsHostName
WHEN NOT MATCHED BY TARGET THEN
 INSERT (OfficeId, DnsHostName, OU, ADSiteName, TimeAdded, BaseManagedEntityId, ManagementGroup, DWManagedEntityRowId)
 VALUES (l.OfficeId, l.DnsHostName, l.OrganizationalUnit, l.ADSiteName, GETUTCDATE(), l.BaseManagedEntityId, l.ManagementGroup, l.DWManagedEntityRowId))
WHEN MATCHED THEN
 UPDATE SET
  h.OfficeId = l.OfficeId,
  h.DnsHostName = l.ServerName, 
  h.OU = l.OrganizationalUnit, 
  h.ADSiteName = l.ADSiteName,
  h.BaseManagedEntityId = l.BaseManagedEntityId, 
  h.ManagementGroup = l.ManagementGroup,
  h.DWManagedEntityRowId = l.DWManagedEntityRowId
WHEN NOT MATCHED BY SOURCE THEN
  DELETE;

Using these tables and a regularly scheduled load, we've now got a foundation that will support rolling up host aggregates at the Region and Office level. In addition, we can now use these structures to navigate live data in the SCOM DW db in our customer's desired fashion. Viel spass! Joseph UPDATED 5 June 2017 - Part 2 of this topic has been posted and is available here.

Reporting on SCOM Data: A Single Pane of Glass

Joseph Merrell — Thu, 13 Apr 2017 13:11:25 GMT

Over the past 6 months or so, I've been building SSRS and Power BI reports based on SCOM data. At first glance, this sounds super simple. Just write some reports on top of the SCOM DW db. However, rarely are things ever that simple. My customer had requirements that everyone from executives to technicians be able to use these reports. In effect, they're wanting a "single pane of glass" across their entire enterprise providing useful information up and down their management chain. You might ask, why not just give everyone access to the SCOM console and be done with them? I'm no SCOM guy (yes, that's a disclaimer), but apparently there is a limitation of about 50 users on the SCOM console. This particular customer had a much higher number that need/want access to it. This means we'll really have to limit access to ensure acceptable performance. Finally, the customer wanted their own hierarchical categorization (from AD) of the data. This would enable them to slice and dice in a manner that matches their business practices. Executives (and other managers) wanted aggregated maps and charts of Health and Alert data with drill-down capabilities to the host level. Technicians want to see the details they need to fix the issue (e.g. - Alerts, Performance metrics). Aggregating and rolling up data is easy enough for the execs and mangers, but providing meaningful real-time data to the technicians can be another beast. Do we report against the SCOM DW db? Do we ETL everything we need into our own db? That can be tricky, especially when/if that server is experiencing performance issues. I've seen cases with the SCOM DW db where data is late arriving and doesn't get picked up in the normal query window. Then your data and reports are missing records. Not a good experience when wanting to see real-time data. And what about the hierarchical categorization? How does that fit in? Ultimately, we settled on a fairly simple, hybrid approach of ETL'ing into our database for reporting the aggregates/rollups and querying directly from the SCOM DW db for real-time data. This enabled us to address use cases for executives, technicians, and everyone in between. Specifically for the technician-level users, we built reports on top of the SCOM DW db, but linked to them from the drill-down reports querying the reporting db. This addressed a vast majority of their needs. If they needed anything deeper, then SCOM console is still available. This approach enabled us to correlate the hosts and aggregates with the customer's hierarchical categorizations. At the same time, we were able to give the technicians the tools that they need to troubleshoot and address problems in real-time, but not overload requests from the SCOM console. Here's small snippet of the North American offices as identified by their aggregated SCOM Health across their core services (think Directory Services, Collaboration, Database, Mail, etc.). Selecting a particular office, you can drill down into the core services being monitored in that office. Drilling down from the above map, we have an example of deeper core services health monitoring. Here we can see how many servers in the enterprise are in a given SCOM Health State. Clicking each of the slices drills into a report of hosts within that Health State for that given service. This snapshot is a report that shows the hosts within a given Health State for a given service, along with links to the SCOM Web Console Health Explorer (Server Name), Alert Views (Open Alerts), and Performance Monitor. We're also aggregating Alert counts with a tree map for the most common occurrences, enabling task prioritization for technicians. Clicking each Alert drills down in the hosts that are reporting the specific Alert. And Alert counts over the last 7 days provides trending some trending. The result of all of this? A reporting solution that provided visibility into the customer's entire enterprise that they had never had before. We were able to proactive monitoring capabilities and the customer loved it! So much so that they've extended our engagement for integration with other data sources like SCCM, network scanning records results, and network device data. In the end, deciding when to ETL data from SCOM for reporting purposes came down to determining the cutoff point of where the aggregations stopped and the real-time data began. Following this blue print with the upcoming data sources, we'll be able to bring even more value to the customer by giving them a much deeper picture of their enterprise. Viel spass! Joseph 23 Apr 2017 UPDATE: My follow up post can be found here.