Partager via

Quick and Dirty Large Scale Eventing for Windows

  • Article

One of the least known yet most powerful management features to ship with Windows Vista and Windows Server 2008 is built-in Event Forwarding which enables large scale health and state monitoring of a Windows environment (assuming health and state can be determined from Windows Events - which they usually can). Not only is this feature built into the latest versions of Windows, but it's also available for down-level OSs like Windows XP SP2+ and Windows Server 2003 SP1+ (here).

Note:  True enterprise class Windows eventing is included with enterprise monitoring solutions like System Center Operations Manager.

This new Windows Event Forwarding (also known as Windows Eventing 6.0) is exceptional for the following reasons:

  1. Standards Based: No really! It leverages the DMTF WS-Eventing standard which allows it to interoperate with other WS-Man implementations (see OpenWSMAN at SourceForge).
  2. Agentless:  Event Forwarding and Event Collection are included in the OS by default
  3. Down-Level Support: Event Forwarding is available for Windows XP SP2+ and Windows Server 2003 SP1+
  4. Multi-Tier: Forwarding architecture is very scalable where a "Source Computer" may forward to a large number of collectors and collectors may forward to collectors
  5. Scalable: Event Collection is very scalable (available in Windows Vista as well) where the collector can maintain subscriptions with a large number of "Source Computers" as well as process a large number of events per second
  6. Group Policy Aware: The entire model is configurable by Group Policy
  7. Schematized Events: Windows Events are now schematized and rendered in XML which enables many scripting and export scenarios
  8. Pre-Rendering: Forwarded Windows Events can now be pre-rendered on the Source Computer negating the need for local applications to render Windows Events
  9. Resiliency: Designed to enable mobile scenarios where laptops may be disconnected from the collector for extended periods of time without event loss (except when logs wrap) as well as leveraging TCP for guaranteed delivery
  10. Security: Certificate based encryption via Kerberos or HTTPS

This implementation will walk through the following example design where via Group Policy a domain computer group will be configured to forwared Windows Events to a single collector:

Implementation steps are as follows:

  • Step 1: Create Event Forwarding Subscription
  • Step 2: Configure WinRM Group Policy
  • Step 3: Configure Event Forward Group Policy
  • Step 4: Test

Step 1: Create the Event Forwarding Subscription on the Event Collector

In the Windows Event Forwarding architecture, the subscription definition is held and maintained on the Collector in order to reduce the number of touch-points in case a subscription needs to be created or modified. Creating the subscription is accomplished through the new Event Viewer user interface by selecting the 'Create Subscription' action when the 'Subscriptions' branch is highlighted. The Subscription may also be created via the "WECUTIL" command-line utility.

Note: Both Windows Vista and Windows Server 2008 can be event collectors (this feature is not supported for down-level). Although there are no built-in limitations when Vista is a collector, Server 2008 will scale much better in high volume scenarios.

Although the above subscription is configured to leverage Group Policy, the subscription can be configured in a stand-alone mode (see the "Collector Initiated" option). In addition, this subscription is designed to gather all events from the "Application" and "System" logs that have a level of "Critical", "Error", or "Warning". This event scope can be expanded to gather all events from these logs or even add additional logs (like the "Security" log).

Lastly, the subscription is configured to forward events as quickly as possible with the advanced settings delivery option of "Minimize Latency". The default setting of "Normal" would only forward events every 15 minutes (which may be more desirable depending the the Collector and Source Computer resources).

If Group Policy is not being used, configure the "Subscription type" to be "Collector Initiated". In this case Source Computers will need to be manually added to the Subscription either through the Subscription configuration or the "WECUTIL" command-line utility (which can also be scripted using PowerShell, but that's another topic).

Note: In cases where there Source Computer is generating a large volume of forwarded events (e.g. Security events from a Domain Controller), use WECUTIL on the collector to disable event rendering for the subscription. The task of pre-rendering an event on the source computer can be CPU intensive for a large number of events.

Step 2: Configure Group Policy to enable Windows Remote Management on the Source Computers (clients)

Group Policy can be used to enable and configure Windows Remote Management (WinRM or WS-Man) on the Source Computers. WinRM is required by Windows Event Forwarding as WS-Man is the protocol used by WS-Eventing. The following shows the Group Policy branch locations for configuring both WinRM and Event Forwarding:

The following GP setting will enable WinRM on the client as well as configure a Listener that will accept packets from ANY source.

Note: This Listener configuration should only be used in a trusted network environment. If the environment is not trusted (like the Internet), then configure only specific IP Addresses or ranges in the IPv4 and IPv6 filters.

To configure WinRM outside of Group Policy, run the following command on the Source Computer (also see the above Note):

winrm quickconfig

Step 3: Configure Group Policy to enable Windows Event Forwarding on the Source Computers

As with WinRM, Group Policy can be used to configure Source Computers (Clients) to forward events to a collector (or set of collectors). The policy is very simple. It merely tells the Source Computer to contact a specific FQDN (Fully Qualified Domain Name) or IP Address and request subscription specifics. All of the other subscription details are held on the Collector.

If Group Policy is not being used, then there is nothing to do here as the "Collector Initiated" Subscription will proactively reach out to the Source Computer.

Step 4: Test Event Forwarding

If all of the Event Forwarding components are functioning (and there's minimal network latency), a test event created on the Source Computer should arrive in the Collector's "Forwarded Events" log within 60 seconds. Create a test event with the following command:

eventcreate /id 999 /t error /l application /d "Test event."

This event should appear on the Collector as follows:

If the event doesn't appear, perform the following troubleshooting steps:

Troubleshooting Step 1: Has Policy Been Applied to the Source Computer?

This can be forced by running the following command on the Source Computer:

gpupdate /force

Troubleshooting Step 2: Can the Collector Reach The Source Computer via WinRM?

Run the following command on the Collector

winrm id /r:<Source Computer> /a:none

Troubleshooting Step 3: Is the Collector Using the Right Credentials?

Run the following command on the Collector

winrm id /r:<Source Computer> /u:<username> /p:<password>

Note: These are the credentials defined in the Subscription on the Collector. The credentials don't need to be in the local Administrators group on the Source Computer, they just need to be in the "Event Log Readers" group on the Source Computer (local Administrators will also work).

Troubleshooting Step 4: Has the Source Computer Registered with the Collector?

Run the following command on the Collector

wecutil gr <subscription name>

This will list all the registered Source Computers (note if the Subscription is "Collector Initiated" then this will list all configured Source Computers), their state (from the Collector's perspective), and their last heartbeat time.

Enjoy!

Comments

  • Anonymous
    January 01, 2003
    PingBack from http://computers.buckethunt.com/1969/12/31/quick-and-dirty-large-scale-eventing-for-windows/

  • Anonymous
    January 01, 2003
    Confused by the title? No problem. Most of my titles are confusing, even to me, and I write them. I&#39;ve

  • Anonymous
    August 11, 2008
    This is a great new feature.  Such a nice thing to have as more virtualized systems in small organizations become a reality.   This is also something that is very powerful when combined with powershell and task scheduler's new event and task integration with powershell...  Far easier to configure and monitor than any linux with python and chron job implementation I've used. Looking forward to the new remote powershell stuff.

  • Anonymous
    September 14, 2010
    Hi, Thanks for this excellent post, I hope I can ask your help with a problem I'm struglling with in the last few days. I have 3 Win2K3 servers that I want them to forward some of their events to a Win7 machine. All the Win2K3 servers are NOT members of the domain (the Win7 machine is) and they also communicate through two firewalls. I have opened ports 80, 443, icmp and 5985 and configured almost everything (I guess) and I get an event in the Win2K3 servers' Microsoft-Windows-Forwarding/Operational event log saying 'Access Denied', here is the full error message: The forwarder is having a problem communicating with subscription manager at addressmy_win7.mvs.co.il/.../WEC.  Error code is 5 and Error Message is <f:WSManFault xmlns:f="schemas.microsoft.com/.../wsmanfault" Code="5" Machine="mvs" xml:lang="he-IL"><f:Message>Access is denied. </f:Message></f:WSManFault>. Any thoughts?

  • Anonymous
    May 04, 2011
    So I have event's landing in the 'Forwarded Events' log. Now what? These don't seem to want to dump out of this long as easily as the other event logs, using the PowerShell command: Get-EventLog -Newest 3 -LogName  ForwardedEvents There's something different about the Forwarded Events log, and for now, the messages I have in it seem to be locked away. How do I get them out?

  • Anonymous
    May 05, 2011
    Great post.  Question for you, have you seen any stability issues with event log forwarding?  I have a situation where it works for a bit, then just stops complaining about user authentication.  But why, if it starts to work why would it fail after a few minutes?  Any quick thoughts or items to look at?  I'm using GP to set it up. Thanks in advance