Inserting Operational Data

Article
09/05/2006

I received a few requests to talk about operational data insertion in SCOM 2007. I was a bit hesitant to take this on now, mostly because I haven't really covered a lot of the core concepts required to truly understand the data insertion process, but I decided that people can ask clarifying questions if they have any while getting this out there was an important step in helping customers and partners move from 2005 to 2007.

There are two objects that are used for inserting operational data; CustomMonitoringEvent and CustomMonitoringPerformanceData. Inserting these objects is supported both via the SDK directly as well as the MCF web-service.

This data can be inserted against any discovered object (unlike MOM 2005 where only programmatically inserted computers were supported for event/performance data insertion in the SDK) via the InsertCustomMonitoringEvent(s) and InsertCustomMonitoringPerformanceData methods on the PartialMonitoringObject class as well as the InsertMonitoringEvents and InsertMonitoringPerformanceData methods on the MCF endpoint.

The below sample illustrates finding all computers in the management group and inserting a single performance data value against each:

using System;

using System.Collections.ObjectModel;

using Microsoft.EnterpriseManagement;

using Microsoft.EnterpriseManagement.Configuration;

using Microsoft.EnterpriseManagement.Monitoring;

namespace Jakub_WorkSamples

{

partial class Program

{

static void InsertCustomMonitoringPerformanceData()

{

// Connect to the sdk service on the local machine

ManagementGroup localManagementGroup = new ManagementGroup("localhost");

// Get the MonitoringClass representing a Computer

MonitoringClass computerClass =

localManagementGroup.GetMonitoringClass(SystemMonitoringClass.Computer);

// Use the class to retrieve partial monitoring objects

ReadOnlyCollection<PartialMonitoringObject> computerObjects =

localManagementGroup.GetPartialMonitoringObjects(computerClass);

// Loop through each computer

foreach (PartialMonitoringObject computer in computerObjects)

{

// Create a CustomMonitoringPerformanceData object

CustomMonitoringPerformanceData perfData =

new CustomMonitoringPerformanceData("CPU", "CPU Threshold", 21.3);

perfData.InstanceName = computer.Name;

// Insert the data

computer.InsertCustomMonitoringPerformanceData(perfData);

}

The pattern for events is very similar, there are just more properties available on CustomMonitoringEvent.

In order to actually use these objects and methods, it's important to understand what happens when any of the insert calls complete. First, these objects get converted to their runtime counterpart as xml; for events this is the System.Event.LinkedData* as defined in the System.Library* management pack and for performance data this is System.Performance.LinkedData* as defined in the System.Performance.Library* management pack. These items are then inserted into the PendingSdkDataSource table in the database. Once this happens, the call succeeds and returns, even though the data has not yet been processed.

In order to actually pick up and utilize the inserted data, I also wrote several DataSourceModuleTypes (this is a management pack level concept that describes data sources (providers in MOM 2005) that are available for use in rules and monitors) that read from the PendingSdkDataSource table in the db and process the newly inserted objects. "Normal" performance data and event rules use the system defined DataSourceModuleTypes that read the system performance counters and the event log, respectively. The data inserted via the SDK will not be processed if using these data sources. All the data sources that facilitate SDK insertion are on a fixed polling interval of 30 seconds, and wake up that often to process any new data in the database. There are two DataSourceModuleTypes available for both events and performance data, all defined in the Microsoft.SystemCenter.Library*:

Microsoft.SystemCenter.SdkEventProvider - This data source will output a System.Event.LinkedData object for every CustomMonitoringEvent inserted via the SDK, regardless of the object it was inserted against.

Microsoft.SystemCenter.TargetEntitySdkEventProvider - This data source will only output a System.Event.LinkedData object for CustomMonitoringEvents inserted via the SDK that were inserted against the target of the workflow that is using the DataSourceModuleType. For instance, if you create a new rule and target it to the System.Computer type and use this DataSourceModuleType as the data source of the rule, the only events that will come out of the data source will be events that were inserted against objects of the System.Computer class.

Microsoft.SystemCenter.SdkPerformanceDataProvider - The same as the SdkEventProvider, only for System.Performance.LinkedData and CustomMonitoringPerformanceData.

Microsoft.SystemCenter.TargetEntitySdkPerformanceDataProvider - The same as the TargetEntitySdkEventProvider, only for System.Performance.LinkedData and CustomMonitoringPerformanceData.

So, in order to actually drive state of discovered objects, or perform other actions based on the data inserted via the SDK, you will need to write rules or monitors that use the aforementioned DataSourceModuleTypes. We do ship and install by default in the Microsoft.SystemCenter.Internal* management pack two rules that automatically collect all the data inserted via the SDK; Microsoft.SystemCenter.CollectSdkEventData* and Microsoft.SystemCenter.CollectSdkPerformanceData*. These rules are both targeted at the RootManagementServer* class and will only be instantiated, as such, on the Principal Management Server.

One very important thing to note: if you write rules or monitors that use the sdk data sources, they must be executed on a server that has database access AND the account the rule or monitor is running under must have the required database permissions. In general a rule or monitor is executed on the machine that discovered the object of a particular class; i.e. if you discover an instance of SQL Server on computer A and computer A has an agent on it, all rules and monitors targeted to SQL Server will be run on that particular SQL Server objects behalf on the agent on computer A. That's a bit of a mouthful, but I hope it makes sense.

* These names are subject to change prior to RTM.

Comments

Anonymous
September 07, 2006
Is it possible to provide some more documentation and/or sample how to create a monitor based on this SDKEvent ?
Anonymous
September 08, 2006
Absolutely. For my next post I will go over using the data sources I described here in a managemment pack.

To get you started though, you can check out the monitor types as defined in the Microsoft.Windows.Library management pack. If you look at the Microsoft.Windows.SingleEventLogManualReset2StateMonitorType, you will notice that it is compromised of an event data source followed by an expression filter. You would use the SDK event data source in much the same way, creating a monitor type that uses it, then creating monitors that use that monitor type.

Again, the big thing to note here is that the SDK event data source will produce events already attributed to monitoring objects, regardless of the target of the monitor or rule of which it is a part. Thus, targeting for monitors and rules using this data source would almost always be against the Prinicpal Management Server, unless they were targeted against a custom user-defined class that is always discovered and managed by the Principal Management Server, in which case targeting the monitor or rule to that class will have the same affect and you can utilize the Microsoft.SystemCenter.TargetEntitySdkEventProvider.
Anonymous
November 20, 2006
Is it possible to override the date/time at which the performance data is received? And is it possible to get multiple performance counters from single rule? or do i need to write seperate rules for each performance counter? The example i'm testing is using a Windows.TimedScript.PerformanceProvider, my performance data will be periodically populated into a file. So i need to check the file and load the data in SCOM. So every time i need to bulk load many performance counters and to be accurate i need to follow the date/time provided in the file.
Anonymous
November 21, 2006
As long as your data source can read multiple counters, your rule should be able to write them to the database. It looks like with this data source you cannot do multiple counters nor can you set the time. You can use the SDK to accomplish this by reading from the file then inserting the data via MonitoringObject.InsertCustomMonitoringPerformanceData(). There you can set the time generated for the performance data.
Anonymous
November 21, 2006
Is it possible to write custom "data sources" or "data providers" using the SDK and use that data source from the Rules?
Anonymous
November 27, 2006
The only public way this is supported is through scripts. We don't have any resources available to help partners build managed code modules at this time.
Anonymous
February 12, 2008
Can you point me to the place where I find official or non-official documentation on 'things' such as SdkPerformanceDataProvider, Microsoft.Windows.Library, etc? I am able to figure out some of the things by going over the cool examples provided here. However often I run into things that are not used in any of the examples. Right now I am trying to figure out what configurable properties available in SdkPerformanceDataProvider data source.
Anonymous
February 12, 2008
There is no single place where you can find documentation like this; some things are not documented at all, like the SdkPerformanceDataProvider for instance. For that specifically, you should open the MP it's defined in using your favorite xml editor or the authoring console and take a look that way.
Anonymous
February 12, 2008
Found it. Thanks! Lack of documentation is not much of problem because you provide unbelievably quick response on the blog. This is going to be the first place to look!
Anonymous
January 20, 2009
I'm trying to insert some performance data for a MP I created using MonitoringObject.InsertCustomMonitoringPerformanceData(). If I now make a rule with Microsoft.SystemCenter.TargetEntitySdkPerformanceDataProvider as DataSource, what module would I have to use as write action?
Anonymous
January 20, 2009
If you just want to insert, we ship a rule OOB that already does this. Are you trying to do something special?
Anonymous
January 20, 2009
Don't think I'm doing anything special. I made a custom class. By using the SDK I inserted a few instances with my own connector. Now by using InsertCustomMonitoringPerformanceData()I inserted some performance data against those instances. When I open the PendingSdkDataSource table I can see the inserted data, but it never seems to be inserted into the database as performance data. Reading your article I would guess I have to use TargetEntitySdkPerformanceDataProvider as data source in a rule, but I am not sure how to create the rest of the rule.
Anonymous
January 21, 2009
This is the rule that ships in the Microsoft.SystemCenter.2007 MP for perf counter collection from SDK: <Rule ID="Microsoft.SystemCenter.CollectSdkPerformanceData" Comment="this rule collects all performance data from the sdk" Enabled="true" Target="SCLibrary!Microsoft.SystemCenter.RootManagementServer" ConfirmDelivery="true" Remotable="true" Priority="Normal" DiscardLevel="100"> <Category>PerformanceCollection</Category> <DataSources> <DataSource ID="DS" TypeID="SCLibrary!Microsoft.SystemCenter.SdkPerformanceDataProvider" /> </DataSources> <WriteActions> <WriteAction ID="WriteToDB" TypeID="SCLibrary!Microsoft.SystemCenter.CollectPerformanceData" /> </WriteActions> </Rule>
Anonymous
January 21, 2009
Thanks for your help Jakub, that works very well!
Anonymous
February 07, 2009
Hi Jakub, Is it correct that performace data in the past doesn't get inserted? For example perfData.TimeSampled = DateTime.Now.AddSeconds(-(double)30); doesn't seem to be inserted.
Anonymous
February 07, 2009
I figured it out. It's possible to insert perf data in the past, jus not before any other data sampled.
Anonymous
March 30, 2009
Please clarify it for me - I insert the data using InsertCustomMonitoringPerformanceData and the OOB CollectSdkPerformanceData rule is enabled. I can see the performance metrics in the console. However when I define the monitors (Unit Monitors) to trigger the state changes etc - these stay grayed-out - I assume the reason is that the metrics are not actually inserted...right? So what do I need to do to be able to use the counters in the Unit Monitors? Thanks, David
Anonymous
March 30, 2009
Did you create the monitor with the SDK data source as in the sample?
Anonymous
March 30, 2009
No, and I guess this is the problem... and I dont see any attachments in this post, could you post a link to the sample? thx
Anonymous
March 30, 2009
I think I found the example ... I implemented the datasource/monitortype according to it (imported MP ok) but still getting the monitors grayed out. I had to guess the element names of the SDK Performance Counter so this might be the reason... could you look at the code fragment below - if anything stands out? Also is there a way to debug this? Thx <MonitorTypes> <UnitMonitorType ID="CCMSMetricMonitorType" Accessibility="Public"> <MonitorTypeStates> <MonitorTypeState ID="SDKMetricUnderThreshold" /> <MonitorTypeState ID="SDKMetricOverThreshold" /> </MonitorTypeStates> <Configuration> <xsd:element name="CounterName" type="xsd:string" minOccurs="1" maxOccurs="1" /> <xsd:element name="ObjectName" type="xsd:string" minOccurs="1" maxOccurs="1" /> <xsd:element name="Frequency" type="xsd:int" minOccurs="1" maxOccurs="1" /> <xsd:element name="Threshold" type="xsd:double" minOccurs="1" maxOccurs="1" /> </Configuration> <MonitorImplementation> <MemberModules> <DataSource ID="DS" TypeID="SC!Microsoft.SystemCenter.SdkPerformanceDataProvider" /> <ConditionDetection TypeID="System!System.ExpressionFilter" ID="CDSDKMetricOver"> <Expression> <And> <Expression> <SimpleExpression> <ValueExpression> <XPathQuery>ObjectName</XPathQuery> </ValueExpression> <Operator>Equal</Operator> <ValueExpression> <Value>$Config/ObjectName$</Value> </ValueExpression> </SimpleExpression> </Expression> <Expression> <SimpleExpression> <ValueExpression> <XPathQuery>CounterName</XPathQuery> </ValueExpression> <Operator>Equal</Operator> <ValueExpression> <Value>$Config/CounterName$</Value> </ValueExpression> </SimpleExpression> </Expression> <Expression> <SimpleExpression> <ValueExpression> <XPathQuery>Value</XPathQuery> </ValueExpression> <Operator>Greater</Operator> <ValueExpression> <Value>$Config/Threshold$</Value> </ValueExpression> </SimpleExpression> </Expression> </And> </Expression> </ConditionDetection> <ConditionDetection TypeID="System!System.ExpressionFilter" ID="CDSDKMetricUnder"> <Expression> <And> <Expression> <SimpleExpression> <ValueExpression> <XPathQuery>ObjectName</XPathQuery> </ValueExpression> <Operator>Equal</Operator> <ValueExpression> <Value>$Config/ObjectName$</Value> </ValueExpression> </SimpleExpression> </Expression> <Expression> <SimpleExpression> <ValueExpression> <XPathQuery>CounterName</XPathQuery> </ValueExpression> <Operator>Equal</Operator> <ValueExpression> <Value>$Config/CounterName$</Value> </ValueExpression> </SimpleExpression> </Expression> <Expression> <SimpleExpression> <ValueExpression> <XPathQuery>Value</XPathQuery> </ValueExpression> <Operator>LessEqual</Operator> <ValueExpression> <Value>$Config/Threshold$</Value> </ValueExpression> </SimpleExpression> </Expression> </And> </Expression> </ConditionDetection> </MemberModules> <RegularDetections> <RegularDetection MonitorTypeStateID="SDKMetricOverThreshold"> <Node ID="CDSDKMetricOver"> <Node ID="DS" /> </Node> </RegularDetection> <RegularDetection MonitorTypeStateID="SDKMetricUnderThreshold"> <Node ID="CDSDKMetricUnder"> <Node ID="DS" /> </Node> </RegularDetection> </RegularDetections> </MonitorImplementation> </UnitMonitorType> </MonitorTypes> </TypeDefinitions> <Monitoring> <Monitors> <UnitMonitor ID="SystemAvailabilityMonitor" Accessibility="Public" Enabled="true" Target="SAPSystem" ParentMonitorID="Health!System.Health.AvailabilityState" Remotable="true" Priority="Normal" TypeID="CCMSMetricMonitorType" ConfirmDelivery="true"> <Category>Custom</Category> <OperationalStates> <OperationalState ID="SystemAvailabilityMonitor.Under" MonitorTypeStateID="SDKMetricUnderThreshold" HealthState="Warning" /> <OperationalState ID="SystemAvailabilityMonitor.Over" MonitorTypeStateID="SDKMetricOverThreshold" HealthState="Success" /> </OperationalStates> <Configuration> <CounterName>RFCConnect</CounterName> <ObjectName>Availability</ObjectName> <Frequency>300</Frequency> <Threshold>0.99</Threshold> </Configuration> </UnitMonitor> <UnitMonitor ID="AppServerAvailabilityMonitor" Accessibility="Public" Enabled="true" Target="AppServer" ParentMonitorID="Health!System.Health.AvailabilityState" Remotable="true" Priority="Normal" TypeID="CCMSMetricMonitorType" ConfirmDelivery="true"> <Category>Operations</Category> <OperationalStates> <OperationalState ID="AppServerAvailabilityMonitor.Under" MonitorTypeStateID="SDKMetricUnderThreshold" HealthState="Warning" /> <OperationalState ID="AppServerAvailabilityMonitor.Over" MonitorTypeStateID="SDKMetricOverThreshold" HealthState="Success" /> </OperationalStates> <Configuration> <CounterName>Value</CounterName> <ObjectName>Availability</ObjectName> <Frequency>300</Frequency> <Threshold>0.99</Threshold> </Configuration> </UnitMonitor> <UnitMonitor ID="AppServerCPUMonitor" Accessibility="Public" Enabled="true" Target="AppServer" ParentMonitorID="Health!System.Health.PerformanceState" Remotable="true" Priority="Normal" TypeID="CCMSMetricMonitorType" ConfirmDelivery="true"> <Category>Operations</Category> <OperationalStates> <OperationalState ID="AppServerCPUMonitor.Under" MonitorTypeStateID="SDKMetricUnderThreshold" HealthState="Success" /> <OperationalState ID="AppServerCPUMonitor.Over" MonitorTypeStateID="SDKMetricOverThreshold" HealthState="Error" /> </OperationalStates> <Configuration> <CounterName>CPU_Utilization</CounterName> <ObjectName>CPU</ObjectName> <Frequency>900</Frequency> <Threshold>95</Threshold> </Configuration> </UnitMonitor> </Monitors> </Monitoring>
Anonymous
March 30, 2009
Have you looked at the sample here? http://blogs.msdn.com/jakuboleksy/archive/2006/10/17/more-with-alert-and-state-change-insertion.aspx
Anonymous
March 31, 2009
yes - I based my attempt on it. Several things are not clear to me yet:

Is it sufficient to define the MonitorType and a Monitor? Or an additional rule has to there there to trigger it?
What is the XML structure of the Performance Counter as produced by Microsoft.SystemCenter.SdkPerformanceDataProvider?
Is it possible to take a peek at the implementation of System.Performance.ThresholdMonitorType? Thx

Anonymous
March 31, 2009

Yes, it should be enough.
It is the same as all performance data.
Not that I know of. The best I could offer is a strategy on how to debug this, namely making the most simple monitor you can do, maybe without a condition detection even, and making sure it is all running properly. You can also use the tasks to see running and failed workflows and make sure your monitor is running on the RMS.

Anonymous
March 31, 2009
Thx, appreciate your help! One last (hopefully) question - for the 2) - where can I find the structure for the Performance Data XML?
Anonymous
March 31, 2009
I don't know honestly, but one way you could get it is look at the Context field of an alert generated from performance data.
Anonymous
March 31, 2009
Found it. FYI here it is: < DataItem type =" MOM.MOMNTPerfData " time =" 2009-03-31T10:52:48.0468750-07:00 " sourceHealthServiceId =" A198AC0B-4420-BCFF-0AF0-44F155E45288 " > < ObjectName > Availability </ ObjectName > < CounterName > RFCConnect </ CounterName > < InstanceName /> < IsNull Type =" Boolean " > false </ IsNull > < Value > 1 </ Value > < ManagedEntityId > {E0D05B9E-9A6C-1192-9E97-CC2EE2DF364D} </ ManagedEntityId > < RuleId > {47047E61-0565-BCC8-D47C-9A889FFF06FE} </ RuleId > </ DataItem >
Anonymous
May 28, 2009
Hello, Thanks for the example. I am having an issue when inserting multiple events targeted at the same monitoring object. These are seperate and distinct events with data read from a log file. If I insert these events without pausing I get the correct number of events in OpsMgr but all with the same event message. However, if I pause after each insertion I get the proper number of events in OpsMgr and with the proper event message. I have double checked my code and it seems correct. Any ideas?
Anonymous
May 28, 2009
Can you elaborate on what you mean by pausing? Do you have a code sample you could share?
Anonymous
May 28, 2009
Certainly, What I mean is this. If I run my app which reads a log file and then inserts events based on data in that log file I get the weird behavior of the same description for each event. If I put a break point at the end of the method I use to insert the event into opsmgr and then wait for the event to show up in opsmgr it is correct. I just looked at the PendingSdkDataSource table and I see the correct events. Are the records in this table supposed to be processed and then removed from the table?
Anonymous
May 28, 2009
The records are groomed later. If yo look at the resulting events in the DB, are they wrong too. That is very strange behavior that I have never seen before and doesn't reproduce for me.
Anonymous
May 28, 2009
Jakub, The events in PendingSdkDataSource are correct, however I didn't look at every single one. The events showing in the OpsMgr console are incorrect. I am going to create a simple console app that can be used to demonstrate this behavior. If it does I will post it. Thanks for your help.
Anonymous
May 28, 2009
Jakub, I was able to reproduce with the following code. using System; using System.Collections.Generic; using System.Collections.ObjectModel; using System.Linq; using System.Text; using System.IO; using Microsoft.EnterpriseManagement; using Microsoft.EnterpriseManagement.Administration; using Microsoft.EnterpriseManagement.Common; using Microsoft.EnterpriseManagement.Configuration; using Microsoft.EnterpriseManagement.Monitoring; namespace OpsMgrEventInsert { class OpsMgrEventInsert { private static ManagementGroup mgmtGroup; private static CustomMonitoringEvent custMonEvent; static void Main(string[] args) { mgmtGroup = new ManagementGroup("<insert your rms here>"); custMonEvent = new CustomMonitoringEvent("MySource", 999); custMonEvent.Channel = "Application"; custMonEvent.LoggingComputer = "<insert your logging computer>"; custMonEvent.User = "<insert your user>"; custMonEvent.LevelId = 1; string classQuery = "Name = 'Microsoft.Windows.Computer'"; MonitoringClassCriteria computerClassCriteria = new MonitoringClassCriteria(classQuery); ReadOnlyCollection<MonitoringClass> monitoringClasses = mgmtGroup.GetMonitoringClasses(computerClassCriteria); if (monitoringClasses.Count != 1) { throw new InvalidOperationException("Expected one monitoring object"); } string objectQuery = "NetbiosComputerName = '<insert NetbiosComputerName here>'"; MonitoringObjectCriteria objectCriteria = new MonitoringObjectCriteria(objectQuery, monitoringClasses[0]); ReadOnlyCollection<MonitoringObject> monitoringObjects = mgmtGroup.GetMonitoringObjects(objectCriteria); if (monitoringObjects.Count != 1) { throw new InvalidOperationException("Expected one monitoring object"); } string logPath = @"<insert path to log file here. One entry per line>"; using (StreamReader reader = new StreamReader(new FileStream(logPath, FileMode.Open, FileAccess.Read, FileShare.ReadWrite))) { string line = ""; while ((line = reader.ReadLine()) != null) { custMonEvent.Message = new CustomMonitoringEventMessage(line); monitoringObjects[0].InsertCustomMonitoringEvent(custMonEvent); } } } } }
Anonymous
May 28, 2009
So you're saying that when you run that code, all the events have the same message, but if you put a Thread.Sleep() after the insert, all is well?
Anonymous
May 29, 2009
When I run that code all of the event messages are correct in PendingSdkDataSource however they are never removed (Are the supposed to be after they are processed?). But in the Operations Manager Console the events all have the same message. Since PendingSdkDataSource is the first stop for the events and they are correc there maybe the CollectSDKEventData module doesn't like something about the events? I didn't use Thread.Sleep(), I just manually "paused" it using a break point so I could verify state. However, I am sure Thread.Sleep() would work.
Anonymous
May 29, 2009
Seems really fishy. What makes it into the PendingSdkDataSource table is what gets converted into alerts, and pausing or not pausing doesn't change that (as you have described), so I am guessing there is some sort of bug or misunderstanding in what you are looking at.
Anonymous
May 29, 2009
It doesn't make sense to me either. Is there anything wrong with my example code? Were you able to run my example code to see if you get the same behavior?
Anonymous
June 02, 2009
Jakub, Is there anything I can provide you or have you decided this isn't worth your time to look at?
Anonymous
June 02, 2009
Have you tried anything simpler, for instance two insert statements one after the other with manual data, no loops or reading from a file? Have you tried retreiving the events from the SDK and seeing that they are in fact incorrect?
Anonymous
June 03, 2009
I will try both of those. However, if I cannot read a file and submit events based on entries in the file this project is sunk. As far as the retreiving the events from the SDK, I figured they would show as incorrect since they show as incorrect in the Operations Manager console, but I will give it a shot anyway.
Anonymous
June 03, 2009
Yes, I am just trying to remove variable. When I do what I just asked you to do on R2, I see different event data for each.
Anonymous
July 09, 2009
Jakub, who is responsible for PendingSdkDataSource records extraction and insertion into other tables? We have latency problem with all our custom metrics. Table contains 540000 records and continue increasing every minute. SCOM 2007 SP1
Anonymous
July 09, 2009
PendingSdkDataSource contains actual data but PerformanceDataAllView does not. Lag between PerformanceDataAllView and PendingSdkDataSource starts with 5-10 mins, then lag is increasing to 27 hours and continue increasing. if we created custom rule and one of write actions is "write to DW". what is intermediate table for DW db? as we have the same problem for data in DW db.
Anonymous
July 09, 2009
There is a workflow that uses an special data source the reads from this table to do the insertion. This post discusses what the modules are that do this. How much perf data are you inserting via the SDK? Keep in mind, this particular solution was never meant to support collecting a lot of data.
Anonymous
July 09, 2009
Jakub, thank you for promt responce. We insert >1500 perf. metrics every 5 minutes. Are there any way to speed up workflow? Will moving OpsDB/RMS to more powerful HW help up? Will more aggressive grooming of the table help us? If yes, cna we change somehow internal SP p_PendingSdkDataSourceGrooming and Table PartitionAndGroomingSettings?
Anonymous
July 09, 2009
and will adding indexes to that table help up? Could you kindly please shed some light on mentioned workflow logic.
Anonymous
July 10, 2009
The comment has been removed
Anonymous
July 10, 2009
I'll have to get back to you on these on Monday, will have to dig up the code again. The 1k pull is either every 30 or 60 seconds. Grooming can be changed to be more aggressive, which would likely help. Indexes might help and better HW would likely help too.
Anonymous
July 11, 2009
We dug further. According to sql trace log. every 30 sec - pull next 1000 records where ObjectType = 0 (?events) every 5 mins - pull next 1000 records where ObjectType = 1 (performance). If it is correct, then our problem make sense. As we feed table faster than internal workflow eats them. Looks like we need patch from MS with ability to configure max row count to pull.
Anonymous
July 11, 2009
The polling interval for both events and performance data is 30 seconds, however, the module will wait until it receives an ACK for all outstanding data items. I am guesing performance data insertion is generally behind in your environment.
Anonymous
July 12, 2009
in this case we have bottleneck in RMS or DB server performance. Jakub, thank you for support.
Anonymous
August 17, 2009
The comment has been removed
Anonymous
August 18, 2009
There isn't a way to update existing values and to answer your question, it will store both.

Share via

Inserting Operational Data

Comments

Additional resources