During SQL Server startup the number of log writer workers is determined. 

If multiple log write workers are not allowed use a single log writer

Otherwise

• Total number NUMA nodes * 2
• Count the number of available CPUs on the NUMA node (affinity mask can impact this count)

       Take the smaller of the above calculations and cap by the MAX_LOG_WRITERS allowed (4 or 8 depending on version)

Bob Dorr

Reported Symptoms

·         Random connection failures from both SQL Authentication and AD based logins

·         Failures occur from remote clients or sqlcmd executed directory on the server (/opt/mssql-tools/bin)

·         Unpredictable when failures occur

·         Non-yielding scheduler reports

·         SQL Server 2017 CU10 does not show behavior

·         SQL Server 2017 CU12 exhibits the behavior

·         Server is a Superdome with lots of CPUs and RAM

Health Session Information

One of the first stops in debugging this issue is the SQL Server health session traces located in the /var/opt/mssql/log/system_health*.xel files. You may view and adjust the XEvent health session from SQL Server Management Studio (SSMS).

-          Watch live data

-          Right mouse | Properties to adjust the session

-          Right mouse | Script Session as …

Several event types are included in the health session, some of which include predicates to remove noise from the system health session.

-          sqlclr.clr_allocation_failure

-          sqlclr.clr_virtual_alloc_failure

-          sqlos.memory_broker_ring_buffer_recorded

-          sqlos.memory_node_oom_ring_buffer_recorded

-          sqlos.process_killed

-          sqlos.scheduler_monitor_deadlock_ring_buffer_recorded

-          sqlos.scheduler_monitor_non_yielding_iocp_ring_buffer_recorded

-          sqlos.scheduler_monitor_non_yielding_ring_buffer_recorded

-          sqlos.scheduler_monitor_non_yielding_rm_ring_buffer_recorded

-          sqlos.scheduler_monitor_stalled_dispatcher_ring_buffer_recorded

-          sqlos.scheduler_monitor_system_health_ring_buffer_recorded

-          sqlos.wait_info

-          sqlos.wait_info_external

-          sqlserver.connectivity_ring_buffer_recorded

-          sqlserver.error_reported

-          sqlserver.security_error_ring_buffer_recorded

-          sqlserver.sp_server_diagnostics_component_result

-          sqlserver.sql_exit_invoked

-          sqlserver.xml_deadlock_report

The non-yielding, wait_info* and connectivity events provide the required information for this issue.  Looking at the health records the following pattern repeatedly appeared.

The events logged in the system health show a non-yield beginning, then a login timeout occurring and the non-yield ending.

connectivity_ring_buffer_record

Drilling into the connectivity ring buffer record details emerge.

When a connection attempt fails the type=LoginTimers record is recorded, providing timings for the login activity.

Starting with the total_login_time_ms = 23390 the events point the failed login attempt of 23.3 seconds.  The SSL exchange time was low but the login_trigger_and_resource_governor_ms value consumed the majority of the time.

The find_login_ms event was another consumer and is child of the login_trigger_and_resource_governor_ms.

How would one know which timings are parent or child values?  Looking at the LoginTimer ring buffer entry you uncover the relationships.

select * from sys.dm_os_ring_buffers

       where ring_buffer_type = 'RING_BUFFER_CONNECTIVITY'

The ring buffer, XML format outputs the parent and child relationships.   This LoginTimes snippet shows the timing information (full record contains additional information for SSL, SSPI, reads, writes, etc.)

<Record id="2" type="RING_BUFFER_CONNECTIVITY" time="591262106">

  <ConnectivityTraceRecord>

    <RecordType>LoginTimers</RecordType>

    <LoginTimersInMilliseconds>

      <TotalTime>23309</TotalTime>

      <EnqueueTime>0</EnqueueTime>

      <NetWritesTime>0</NetWritesTime>

      <NetReadsTime>0</NetReadsTime>

       <TriggerAndResGovTime>

        <TotalTime>23266</TotalTime>

        <FindLogin>23265</FindLogin>

        <LogonTriggers>0</LogonTriggers>

        <ExecClassifier>0</ExecClassifier>

        <SessionRecover>0</SessionRecover>

      </TriggerAndResGovTime>

    </LoginTimersInMilliseconds>

  </ConnectivityTraceRecord>

This information quickly rules out a login trigger or classifier function as the cause of the problem.  As you might imagine from the name SQL Server is looking up the login information and contacting the AD if necessary, as part of the login validation.  This means we are reading database pages from the master, acquiring locks, and making calls to the domain controller (DC/KDC.)

wait_info*

Another observation is that the system health session did not conatain wait_info* events longer than 5-seconds.  The lack of wait_info* events is revealing because if we are waiting for a database read, lock or preemptive, external call for more than 5 seconds an event would have been logged.   This means that we are unlikely encountering issues with database reads, locking or a preemptive, external call. 

Network Trace

Looking at network traces helps build our understanding of the problem.

Normal SQL Login, execute select @@VERSION and logout

Here is an example of a valid SQL Server login.

SYN

1.       TCP connection establishment from the ODBC CLIENT to the LINUX SQL SERVER (SYN)

2.       TDS pre-login exchange

3.       512 byte response (use database context xxxxxx, default language xxxxxxx, test size xxxxx)

4.       SQLBatch (select @@VERSION)

5.       FIN – Close of TCP connection | Disconnect

Failing Login

Here is an example of a failed SQL Server login.

1.       TCP connection establishment from the ODBC CLIENT to the LINUX SQL SERVER (SYN)

2.       TDS pre-login exchange

3.       Login response

4.       Keep alive kicks in and time begins to elapse without other activity

5.       RST – TCP connection is reset by client (OS Error: 10054)

There were several failure variations but the most common waits occurred handling the login response.

Network Layer Overloaded

In contrast to the failing SQL Server login, this is what a network trace looks like when the network layer becomes overloaded and is not completing requests to the SQL Server.  Usually, you start seeing retransmission of the SYN.

Attempting A Reproduction

Now we have an idea about the pattern, a reproduction would be helpful to track the problem and make sure it is properly resolved.  This is where my old friend (ostress) came in handy.

"C:\Program Files\Microsoft Corporation\RMLUtils\ostress" -Usa -Pxxxxxxx  -Q"--" -Stcp:10.193.17.114 -q -n64 -r99999999 -cc:\temp\sample.ini -T146 -oc:\temp\output

-          Use -U and -P for SQL authentication

-          Use -E for AD authentication

-          -q ignores query results

-          -S target server name forcing tcp protocol

-          -Q with a query of --  (comment only to do nothing)

-          -n controls the number of threads

-          -r controls the number of loops for each thread

-          -c the control file

-          -T146 disable encrypted TDS

-          -o output location

In the control file set DisconnectPct=100.0.  This directs ostress to connect, execute the query --, disconnect 100% of the time and then repeat -r times for each of the -n threads.

Tracing XEvent login and logout events you can see the connect and disconnect activities.  The following is a simple query for looking at the rate of login and logouts the test is accomplishing.

drop table #before
go
drop table #after
go

select * into #before
from sys.dm_os_performance_counters where counter_name in ('Logins/sec', 'Logouts/sec')

waitfor delay '00:00:10'

select * into #after       from sys.dm_os_performance_counters where counter_name in ('Logins/sec', 'Logouts/sec')

select b.object_name, a.cntr_value - b.cntr_value as [diff], (a.cntr_value - b.cntr_value) / 10.0 as [Rate/sec]
       from #before b
              inner join #after a on a.counter_name = b.counter_name

I started with a cmd file script exercising the connection path.

start "ostress" "C:\Program Files\Microsoft Corporation\RMLUtils\ostress" -E -Q"--" -Stcp:xxxxxxx -q -n32 -r99999999 -cc:\temp\sample.ini -T146 -oc:\temp\output

start "ostress" "C:\Program Files\Microsoft Corporation\RMLUtils\ostress" -E -Q"--" -Stcp:xxxxxxx -q -n32 -r99999999 -cc:\temp\sample.ini -T146 -oc:\temp\output2

:top

       sqlcmd -E -Stcp:xxxxxxx -Q"select @@VERSION"

       ping -n 10 xxxxxxxx

goto top

Windows TCP Settings

As I cranked up the stress level on a 2TB, 144CPU system I hit some TCP issues.

Running along and all the sudden no traffic occurring at the SQL Server for a few seconds, then stress kicked back in.   During the downturn in activity a local /opt/mssql-tools/bin/sqlcmd connection continued to work.

I started using the following bash script running locally on the Linux SQL Server system to see successful connections while the remote connections were failing.

while [ 1 -gt 0 ]

do

        echo

        echo $(date)

        ./sqlcmd -Usa -Pxxxxxxx -Q"select @@VERSION" -Stcp:.

        sleep 2

done

The stall and go behavior occurred because I was running out of Windows client TCP ports.  Using netsh we adjusted the client port range and the stop and go behavior went away.

netsh int ipv4 show dynamicport tcp
netsh int ipv6 show dynamicport tcp

netsh int ipv4 set dynamicport tcp start=10000 num=50000
netsh int ipv6 set dynamicport tcp start=10000 num=50000

Now I began driving high levels of connect and disconnect operations until I started seeing network saturation (SYN with retransmits) but I was not seeing the non-yield pattern.

Disrupt The Domain Controller (keytab)

Knowing the find login path was doing domain controller calls the next thing I did was pause the domain controller VM and poof – non-yielding occurred with connection timeouts, high times recorded in find login.  My collogue was able to add entries to the keytab file (invalid DC targets) that exhibited long KDC/DC query behavior as well.

We then found a few code paths in find login that did not switch to preemptive before making a call to SSSD or LDAP for Linux, AD activities.   Since SQL Server did not switch preemptive there is no wait_type* records in the system health log.

Non-Yielding Scheduler

Non-yielding is the key to the issue and the associated behaviors.  The following diagram help highlight the problems experienced.

The owning worker made a call to the domain controller without switching off the SQL Server scheduler (preemptive.)  You have a SQL authentication and AD authentication workers part of the way through the pre-login sequence and yielded.  When the owning worker yields the runnable list is used to determine the next worker to execute.  When the runnable list has reached max workers for the scheduler new requests are queued.

The KDC/DC query is taking more than 5 seconds and SchedulerMonitor (one per NUMA node) reports a non-yielding condition (BEGIN).   Every 5 seconds the SchedulerMonitor checks progress, incrementing a pass count.  When that pass count has reached the limit (12 == 60 seconds) a non-yielding situation is reported.

In the customer’s case the non-yield generally lasted 5 to 15 seconds, reported in the system health session with non-yielding BEGIN and END events bracketing connection timeout events.

Various login timeouts are then triggered.  When the client detects the connection, timeout is reached the FIN is sent to close the TCP connection, usually bubbling up to SQL Server as error 10054 – connection closed.

·         If the login was in the middle of processing pre-login the ring buffer entry may show time spent in SSL, reads, etc and the disconnect.

·         If the login was stuck on the DC call the time generally accumulates in find login. 

·         If the login was queued to the scheduler you see 1 or 0ms elapsed times for login failure events. By the time the a worker was allowed to pick up the new task the client has already sent the 10054 and as soon as SQL Server attempts to process the login the connection is detected as closed and fails.

SQL Authentication Impact

Now you can see how an AD authentication could lead to SQL authentication, login timeouts.  If the SQL authentication request was on the same scheduler as the stalled KDC/DC query the quest is impacted.

Other Schedulers Impacted

This code path does not engage in much SQL locking so a stalled user running the KDC/DC query usually does not impact other schedulers and the logins associated with those schedulers.   Other schedulers are impacted by the KDC/DC query in two ways.

1.       If other schedulers are trying to process AD logins and the KDC/DC queries are slow the queries are likely to encounter the same behavior.

2.       If a non-yield is detected a dump is invoked.  The dump suspends the SQL Server process.  If the dump time exceeded the login timeout the client closes the connection and you commonly see a burst of 10054 TCP errors across the SQL Server schedulers.

Recap

The bug is in the find login code not going preemptive before querying the domain controller, leading to delays and login timeouts. 

When fixed, individual AD login attempts still encounter login timeouts when experiencing delays from the KDC/DC query, but other logins and queries are no longer impacted.

Bob Dorr

Thank to test folks for helping track down and correct this issue.

Dylan Gray, Mike Habben, Mitchell Sternke, Val Menn, Kapil Thacker, Ben Ko, Madeline MacDonald, and Ethan Moffat

For an X64 installation, using the sp_configure ‘max worker threads’ value of 0 SQL Server uses the following calculation.

On small environments SQL Server always uses 256 as the target.

Running on a larger environment 512 workers are the default and are adjusted based on CPUs count.

When more than 4 CPUs are present additional the worker limit is increased from the 512 worker level to 512 plus the remaining CPUs times 32 or 16.  The additional worker level is increased by 32 workers per CPU when the system presents 64 or more CPUs.

The design takes into account smaller environments which few CPU and Memory resources as well as larger CPU and Memory systems.

The worker calculation only applies to visible schedulers.  Hidden schedulers and the Dedicated Admin Connection (DAC) schedulers are not impacted by the calculation.

Bob Dorr

In my previous blog posts, I highlighted the importance of using Event Sequence as the sort key.

How It Works: XEvent Output and Visualization
SQL Server Management Studio Provides–“XE Profiler”
Use the SSMS XEvent Profiler

Some of the feedback has been that sorting can be a cumbersome operation because:

• Using sys.fn_xe_file_target_read_file the XML has to be queried and indexed which can take time and space, followed by the order by event sequence query
• Using SQL Server Management Studio (SSMS) and storing the events in a table requires reading the events, streaming to table storage followed by an order by event sequence query
• Large traces can encounter the 32-bit SSMS memory limitations

Several years ago, I created a XEProfiler based loosely on the SSMS implementation, built as a 64-bit application to avoid the 32-bit memory limitations and using parallel activities to speed up the processing.

The XEProfiler Sample

·         Creates an empty DataGrid control

·         Makes a pass over the files to obtain the Event Locators

·         Sorts the EventLocators

·         Leverages the display location to paint the information on the screen

Note: By tracking and using EventLocators, the overall memory is reduced, and the 64-bit address space allows for a larger set of trace data.

TSQL Sample Session

CREATE EVENT SESSION [EventSequence] ON SERVER
ADD EVENT sqlserver.sql_batch_completed(
    ACTION(package0.event_sequence,sqlserver.client_app_name,sqlserver.session_id)),
ADD EVENT sqlserver.sql_batch_starting(
    ACTION(package0.event_sequence,sqlserver.client_app_name,sqlserver.session_id))
ADD TARGET package0.event_file(SET filename=N'c:\temp\EventSequence',max_file_size=(1024),max_rollover_files=(100))
WITH (MAX_MEMORY=1024 MB,EVENT_RETENTION_MODE=ALLOW_SINGLE_EVENT_LOSS,MAX_DISPATCH_LATENCY=30 SECONDS,MAX_EVENT_SIZE=0
KB,MEMORY_PARTITION_MODE=PER_CPU,TRACK_CAUSALITY=ON,STARTUP_STATE=OFF)

Form1.cs

using System;
using System.Collections.Generic;
using System.Drawing;
using System.IO;
using System.Linq;
using System.Threading;
using System.Threading.Tasks;
using System.Windows.Forms;
using Microsoft.SqlServer.XEvent.Linq; 

namespace XEProfiler
{
    public partial class xeProfilerForm : Form
    {
        // Event locator index needs
        //
        private class LocationInfo
        {
            public QueryableXEventData Stream
            {
                get;
                private set;
            } 

            public EventLocator Location
            {
                get;
                private set;
            } 

            public ulong Sequence
            {
                get;
                private set;
            } 

            // File can consume memory - for sample purposes only
            //
            public string File
            {
                get;
                private set;
            } 

            public LocationInfo(string file, ulong sequence, EventLocator location, QueryableXEventData stream)
            {
                File = file;
                Sequence = sequence;
                Location = location;
                Stream = stream;
            }
        } 

        private SolidBrush textBrush;
        private Font textFont;
        private StringFormat stringFormat = new StringFormat();
        private LocationInfo[] sortedLocators = null; 

        // Constructor to make display loop a bit like SQL Profiler
        //
        public xeProfilerForm()
        {
            InitializeComponent();
            stringFormat.Alignment = StringAlignment.Near;
            stringFormat.LineAlignment = StringAlignment.Near;
            stringFormat.FormatFlags = StringFormatFlags.NoWrap; 

            textBrush = new SolidBrush(Color.Black);
            textFont = new System.Drawing.Font("Courier New", 8.25F, System.Drawing.FontStyle.Regular, System.Drawing.GraphicsUnit.Point, ((byte)(0))); 

            dataGrid.RowHeadersWidth = 10;
            dataGrid.AllowUserToAddRows = false;
            dataGrid.CellPainting += DataGrid_CellPainting;
        } 

        // TODO: Make this meta data driven with targets for fields vs actions to remove need for switch statement
        //
        string[] ColIndexNames = { "timestamp", "event_sequence", "name", "batch_text", "event_file" }; 

        // Draw the proper text for the current display region
        //
        private void DataGrid_CellPainting(object sender, DataGridViewCellPaintingEventArgs e)
        {
            if (e.ColumnIndex > -1 && e.RowIndex > -1)
            {
                e.PaintBackground(e.ClipBounds, false);
                LocationInfo info = sortedLocators[e.RowIndex];
                PublishedEvent pubEvent = info.Stream.EventProvider.RetrieveEvent(info.Location);
                PublishedAction action;
                PublishedEventField field;

                string strText = string.Empty; 

                switch (e.ColumnIndex)
                {
                    case 0:
                        strText = pubEvent.Timestamp.UtcDateTime.ToString();
                        break; 

                    case 1:
                        if (pubEvent.Actions.TryGetValue(ColIndexNames[e.ColumnIndex], out action))
                        {
                            strText = action.Value.ToString();
                        }
                        break; 

                    case 2:
                        strText = pubEvent.Name;
                        break; 

                    case 3:
                        if (pubEvent.Fields.TryGetValue(ColIndexNames[e.ColumnIndex], out field))
                        {
                            strText = field.Value.ToString();
                        }
                        break; 

                    case 4:
                        strText = info.File;
                        break;
                }              

                e.Graphics.DrawString(strText, textFont, textBrush, e.CellBounds, stringFormat);
                e.Handled = true;
            }
        } 

        // Load the event locators and activate the display
        //
        private void Form1_Load(object sender, EventArgs e)
        {

            dataGrid.Enabled = false;
            string[] files = Directory.GetFiles(@"c:\temp", @"EventSequence*", SearchOption.TopDirectoryOnly); 

            object locationLock = new object();
            int rows = 0;
            List<LocationInfo> eventLocators = new List<LocationInfo>(); 

            // Load the EventLocators in parallel
            //
            Parallel.ForEach (files, file  =>
            {
                LoadEvents(ref rows, ref locationLock, ref eventLocators, file);
            }); 

            // Get array sorted by sequence values, allowing parallel Linq activities
            //
            sortedLocators = (from locator in eventLocators select locator)
                                 .OrderBy(x => x.Sequence)
                                 .ToArray(); 

            SetRowCount(rows, true);
        } 

        // Load the events into the list to be sorted and used as index for display
        //
        private void LoadEvents(ref int rows, ref object locationLock, ref List<LocationInfo> eventLocators, string file)
        {
            string fileName = Path.GetFileNameWithoutExtension(file);

            // Open and track the stream so we can use later to access event data
            //
            QueryableXEventData stream = new QueryableXEventData(file); 

            foreach (PublishedEvent pubEvent in stream)
            {
                // Sequence may have holes due to options such as Event Loss
                // All events must be read before final sort can be closed.
                //
                ulong sequence = (ulong)pubEvent.Actions["event_sequence"].Value;
                EventLocator locator = pubEvent.Location;
                LocationInfo info = new LocationInfo(fileName, sequence, locator, stream); 

                lock (locationLock)
                {
                    eventLocators.Add(info);
                } 

                // Simple progress
                //
                int rowCount = Interlocked.Increment(ref rows);
                if (rowCount % 128 == 0)
                {
                    SetRowCount(rowCount, false);
                }
            }
        } 

        // Once row count is complete display headers (TODO: based on metadata)
        // and update the rowcount in status bar
        //
        private void SetRowCount(int rows, bool finished)
        {
            toolStripRowCount.Text = String.Format("{0}", rows);
            toolStripRowCount.Invalidate(); 

            if (finished)
            {
                textBoxDisplay.Text = String.Empty;
                dataGrid.Columns.Clear(); 

                DataGridViewColumn newColumn = new DataGridViewTextBoxColumn();
                newColumn.Name = "timestamp";
                newColumn.HeaderText = "timestamp";
                newColumn.ToolTipText = "Timestamp of the captured event";
                dataGrid.Columns.Add(newColumn); 

                newColumn = new DataGridViewTextBoxColumn();
                newColumn.Name = "sequence";
                newColumn.HeaderText = "sequence";
                newColumn.ToolTipText = "Event sequence of the captured event";
                dataGrid.Columns.Add(newColumn); 

                newColumn = new DataGridViewTextBoxColumn();
                newColumn.Name = "name";
                newColumn.HeaderText = "name";
                newColumn.ToolTipText = "Name of the captured event";
                dataGrid.Columns.Add(newColumn); 

                newColumn = new DataGridViewTextBoxColumn();
                newColumn.Name = "batch_text";
                newColumn.HeaderText = "batch_text";
                newColumn.ToolTipText = "Batch text";
                newColumn.Width = 200;
                dataGrid.Columns.Add(newColumn); 

                newColumn = new DataGridViewTextBoxColumn();
                newColumn.Name = "file";
                newColumn.HeaderText = "file";
                newColumn.ToolTipText = "Event source file";
                newColumn.Width = 200;
                dataGrid.Columns.Add(newColumn); 

                //  Change the dataGrid to an OwnerDraw (OnPaint) so we only have to
                //  draw the data and not really load it into the data grid rows
                //
                dataGrid.RowCount = rows;
                if (rows > 0)
                {
                    dataGrid.Rows[1].Selected = true;
                } 

                dataGrid.Enabled = true;
                dataGrid.Invalidate();
                dataGrid.Refresh();
            }
        } 

        // Batch text display
        //
        private void dataGrid_SelectionChanged(object sender, EventArgs e)
        {
            textBoxDisplay.Text = String.Empty; 

            if (0 != dataGrid.SelectedRows.Count)
            {
                List<DataGridViewRow> orderedSelection = (from DataGridViewRow row in dataGrid.SelectedRows select row)
                                                          .OrderBy(x=>x.Index)
                                                          .ToList(); 

                foreach (DataGridViewRow selectedRow in orderedSelection)
                {
                    if (textBoxDisplay.Text != String.Empty)
                    {
                        textBoxDisplay.Text += "\r\ngo\r\n";
                    } 

                    LocationInfo info = sortedLocators[selectedRow.Index];
                    PublishedEvent pubEvent = info.Stream.EventProvider.RetrieveEvent(info.Location);
                    PublishedEventField field;
                    string strText = string.Empty; 

                    if (pubEvent.Fields.TryGetValue("batch_text", out field))
                    {
                        strText = field.Value.ToString();
                    } 

                    textBoxDisplay.Text += strText;
                }
            }
        }
    }
}
 

Form1.Designer.cs

namespace XEProfiler
{
    partial class xeProfilerForm
    {
        /// <summary>
        /// Required designer variable.
        /// </summary>
        private System.ComponentModel.IContainer components = null;

        /// <summary>
        /// Clean up any resources being used.
        /// </summary>
        /// <param name="disposing">true if managed resources should be disposed; otherwise, false.</param>
        protected override void Dispose(bool disposing)
        {
            if (disposing && (components != null))
            {
                components.Dispose();
            }
            base.Dispose(disposing);
        }

        #region Windows Form Designer generated code

        /// <summary>
        /// Required method for Designer support - do not modify
        /// the contents of this method with the code editor.
        /// </summary>
        private void InitializeComponent()
        {
            System.Windows.Forms.DataGridViewCellStyle dataGridViewCellStyle1 = new System.Windows.Forms.DataGridViewCellStyle();
            System.Windows.Forms.DataGridViewCellStyle dataGridViewCellStyle2 = new System.Windows.Forms.DataGridViewCellStyle();
            System.ComponentModel.ComponentResourceManager resources = new System.ComponentModel.ComponentResourceManager(typeof(xeProfilerForm));
            this.dataGrid = new System.Windows.Forms.DataGridView();
            this.statusBar = new System.Windows.Forms.StatusStrip();
            this.labelRowCount = new System.Windows.Forms.ToolStripStatusLabel();
            this.toolStripRowCount = new System.Windows.Forms.ToolStripStatusLabel();
            this.menuStrip1 = new System.Windows.Forms.MenuStrip();
            this.fileToolStripMenuItem = new System.Windows.Forms.ToolStripMenuItem();
            this.editToolStripMenuItem = new System.Windows.Forms.ToolStripMenuItem();
            this.viewToolStripMenuItem = new System.Windows.Forms.ToolStripMenuItem();
            this.toolsToolStripMenuItem = new System.Windows.Forms.ToolStripMenuItem();
            this.windowToolStripMenuItem = new System.Windows.Forms.ToolStripMenuItem();
            this.helpToolStripMenuItem = new System.Windows.Forms.ToolStripMenuItem();
            this.textBoxDisplay = new System.Windows.Forms.TextBox();
            this.splitContainer1 = new System.Windows.Forms.SplitContainer();
            ((System.ComponentModel.ISupportInitialize)(this.dataGrid)).BeginInit();
            this.statusBar.SuspendLayout();
            this.menuStrip1.SuspendLayout();
            ((System.ComponentModel.ISupportInitialize)(this.splitContainer1)).BeginInit();
            this.splitContainer1.Panel1.SuspendLayout();
            this.splitContainer1.Panel2.SuspendLayout();
            this.splitContainer1.SuspendLayout();
            this.SuspendLayout();
            //
            // dataGrid
            //
            this.dataGrid.AllowUserToAddRows = false;
            this.dataGrid.AllowUserToDeleteRows = false;
            this.dataGrid.AllowUserToOrderColumns = true;
            this.dataGrid.ColumnHeadersHeightSizeMode = System.Windows.Forms.DataGridViewColumnHeadersHeightSizeMode.AutoSize;
            dataGridViewCellStyle1.Alignment = System.Windows.Forms.DataGridViewContentAlignment.MiddleLeft;
            dataGridViewCellStyle1.BackColor = System.Drawing.SystemColors.Window;
            dataGridViewCellStyle1.Font = new System.Drawing.Font("Courier New", 8.25F, System.Drawing.FontStyle.Regular, System.Drawing.GraphicsUnit.Point, ((byte)(0)));
            dataGridViewCellStyle1.ForeColor = System.Drawing.SystemColors.ControlText;
            dataGridViewCellStyle1.SelectionBackColor = System.Drawing.SystemColors.Highlight;
            dataGridViewCellStyle1.SelectionForeColor = System.Drawing.SystemColors.HighlightText;
            dataGridViewCellStyle1.WrapMode = System.Windows.Forms.DataGridViewTriState.False;
            this.dataGrid.DefaultCellStyle = dataGridViewCellStyle1;
            this.dataGrid.Dock = System.Windows.Forms.DockStyle.Fill;
            this.dataGrid.Location = new System.Drawing.Point(0, 0);
            this.dataGrid.Name = "dataGrid";
            this.dataGrid.ReadOnly = true;
            dataGridViewCellStyle2.Alignment = System.Windows.Forms.DataGridViewContentAlignment.MiddleLeft;
            dataGridViewCellStyle2.BackColor = System.Drawing.SystemColors.Control;
            dataGridViewCellStyle2.Font = new System.Drawing.Font("Courier New", 8.25F, System.Drawing.FontStyle.Regular, System.Drawing.GraphicsUnit.Point, ((byte)(0)));
            dataGridViewCellStyle2.ForeColor = System.Drawing.SystemColors.WindowText;
            dataGridViewCellStyle2.SelectionBackColor = System.Drawing.SystemColors.Highlight;
            dataGridViewCellStyle2.SelectionForeColor = System.Drawing.SystemColors.HighlightText;
            dataGridViewCellStyle2.WrapMode = System.Windows.Forms.DataGridViewTriState.True;
            this.dataGrid.RowHeadersDefaultCellStyle = dataGridViewCellStyle2;
            this.dataGrid.RowTemplate.Height = 18;
            this.dataGrid.Size = new System.Drawing.Size(1089, 277);
            this.dataGrid.TabIndex = 0;
            this.dataGrid.SelectionChanged += new System.EventHandler(this.dataGrid_SelectionChanged);
            //
            // statusBar
            //
            this.statusBar.Items.AddRange(new System.Windows.Forms.ToolStripItem[] {
            this.labelRowCount,
            this.toolStripRowCount});
            this.statusBar.Location = new System.Drawing.Point(0, 479);
            this.statusBar.Name = "statusBar";
            this.statusBar.Size = new System.Drawing.Size(1089, 22);
            this.statusBar.TabIndex = 1;
            this.statusBar.Text = "statusBar";
            //
            // labelRowCount
            //
            this.labelRowCount.Name = "labelRowCount";
            this.labelRowCount.Size = new System.Drawing.Size(41, 17);
            this.labelRowCount.Text = "Rows: ";
            //
            // toolStripRowCount
            //
            this.toolStripRowCount.Name = "toolStripRowCount";
            this.toolStripRowCount.Size = new System.Drawing.Size(108, 17);
            this.toolStripRowCount.Text = "toolStripRowCount";
            //
            // menuStrip1
            //
            this.menuStrip1.Items.AddRange(new System.Windows.Forms.ToolStripItem[] {
            this.fileToolStripMenuItem,
            this.editToolStripMenuItem,
            this.viewToolStripMenuItem,
            this.toolsToolStripMenuItem,
            this.windowToolStripMenuItem,
            this.helpToolStripMenuItem});
            this.menuStrip1.Location = new System.Drawing.Point(0, 0);
            this.menuStrip1.Name = "menuStrip1";
            this.menuStrip1.Size = new System.Drawing.Size(1089, 24);
            this.menuStrip1.TabIndex = 2;
            this.menuStrip1.Text = "menuStrip1";
            //
            // fileToolStripMenuItem
            //
            this.fileToolStripMenuItem.Name = "fileToolStripMenuItem";
            this.fileToolStripMenuItem.Size = new System.Drawing.Size(37, 20);
            this.fileToolStripMenuItem.Text = "File";
            //
            // editToolStripMenuItem
            //
            this.editToolStripMenuItem.Name = "editToolStripMenuItem";
            this.editToolStripMenuItem.Size = new System.Drawing.Size(39, 20);
            this.editToolStripMenuItem.Text = "Edit";
            //
            // viewToolStripMenuItem
            //
            this.viewToolStripMenuItem.Name = "viewToolStripMenuItem";
            this.viewToolStripMenuItem.Size = new System.Drawing.Size(44, 20);
            this.viewToolStripMenuItem.Text = "View";
            //
            // toolsToolStripMenuItem
            //
            this.toolsToolStripMenuItem.Name = "toolsToolStripMenuItem";
            this.toolsToolStripMenuItem.Size = new System.Drawing.Size(47, 20);
            this.toolsToolStripMenuItem.Text = "Tools";
            //
            // windowToolStripMenuItem
            //
            this.windowToolStripMenuItem.Name = "windowToolStripMenuItem";
            this.windowToolStripMenuItem.Size = new System.Drawing.Size(63, 20);
            this.windowToolStripMenuItem.Text = "Window";
            //
            // helpToolStripMenuItem
            //
            this.helpToolStripMenuItem.Name = "helpToolStripMenuItem";
            this.helpToolStripMenuItem.Size = new System.Drawing.Size(44, 20);
            this.helpToolStripMenuItem.Text = "Help";
            //
            // textBoxDisplay
            //
            this.textBoxDisplay.Dock = System.Windows.Forms.DockStyle.Fill;
            this.textBoxDisplay.HideSelection = false;
            this.textBoxDisplay.Location = new System.Drawing.Point(0, 0);
            this.textBoxDisplay.Multiline = true;
            this.textBoxDisplay.Name = "textBoxDisplay";
            this.textBoxDisplay.ReadOnly = true;
            this.textBoxDisplay.ScrollBars = System.Windows.Forms.ScrollBars.Both;
            this.textBoxDisplay.Size = new System.Drawing.Size(1089, 174);
            this.textBoxDisplay.TabIndex = 3;
            //
            // splitContainer1
            //
            this.splitContainer1.Dock = System.Windows.Forms.DockStyle.Fill;
            this.splitContainer1.Location = new System.Drawing.Point(0, 24);
            this.splitContainer1.Name = "splitContainer1";
            this.splitContainer1.Orientation = System.Windows.Forms.Orientation.Horizontal;
            //
            // splitContainer1.Panel1
            //
            this.splitContainer1.Panel1.Controls.Add(this.dataGrid);
            this.splitContainer1.Panel1MinSize = 150;
            //
            // splitContainer1.Panel2
            //
            this.splitContainer1.Panel2.Controls.Add(this.textBoxDisplay);
            this.splitContainer1.Panel2MinSize = 150;
            this.splitContainer1.Size = new System.Drawing.Size(1089, 455);
            this.splitContainer1.SplitterDistance = 277;
            this.splitContainer1.TabIndex = 4;
            //
            // xeProfilerForm
            //
            this.AutoScaleDimensions = new System.Drawing.SizeF(6F, 13F);
            this.AutoScaleMode = System.Windows.Forms.AutoScaleMode.Font;
            this.ClientSize = new System.Drawing.Size(1089, 501);
            this.Controls.Add(this.splitContainer1);
            this.Controls.Add(this.statusBar);
            this.Controls.Add(this.menuStrip1);
            this.Icon = ((System.Drawing.Icon)(resources.GetObject("$this.Icon")));
            this.MainMenuStrip = this.menuStrip1;
            this.Name = "xeProfilerForm";
            this.Text = "XEProfiler";
            this.WindowState = System.Windows.Forms.FormWindowState.Maximized;
            this.Load += new System.EventHandler(this.Form1_Load);
            ((System.ComponentModel.ISupportInitialize)(this.dataGrid)).EndInit();
            this.statusBar.ResumeLayout(false);
            this.statusBar.PerformLayout();
            this.menuStrip1.ResumeLayout(false);
            this.menuStrip1.PerformLayout();
            this.splitContainer1.Panel1.ResumeLayout(false);
            this.splitContainer1.Panel2.ResumeLayout(false);
            this.splitContainer1.Panel2.PerformLayout();
            ((System.ComponentModel.ISupportInitialize)(this.splitContainer1)).EndInit();
            this.splitContainer1.ResumeLayout(false);
            this.ResumeLayout(false);
            this.PerformLayout();

        }

        #endregion

        private System.Windows.Forms.DataGridView dataGrid;
        private System.Windows.Forms.StatusStrip statusBar;
        private System.Windows.Forms.ToolStripStatusLabel labelRowCount;
        private System.Windows.Forms.ToolStripStatusLabel toolStripRowCount;
        private System.Windows.Forms.MenuStrip menuStrip1;
        private System.Windows.Forms.ToolStripMenuItem fileToolStripMenuItem;
        private System.Windows.Forms.ToolStripMenuItem editToolStripMenuItem;
        private System.Windows.Forms.ToolStripMenuItem viewToolStripMenuItem;
        private System.Windows.Forms.ToolStripMenuItem toolsToolStripMenuItem;
        private System.Windows.Forms.ToolStripMenuItem windowToolStripMenuItem;
        private System.Windows.Forms.ToolStripMenuItem helpToolStripMenuItem;
        private System.Windows.Forms.TextBox textBoxDisplay;
        private System.Windows.Forms.SplitContainer splitContainer1;
    }
}

MergeXEvents

MergeXEvents consumes XEvent files, using optimized parallel sort algorithms and outputting events into the physical file in event_sequence order.  This was a project I created in 2014 to help Microsoft SQL Server support process incoming XEvent files, including the ability to merge XEvent files from multiple servers such as replication or high availability XEvent capture.  The project uses parallel threads to perform scatter/gather activities as well as tournament sorting to optimize the sorting and merging activities.

Program.cs

/*
*      Designed to merge XEvents from multiple files into a single stream sorted by Event Sequence, Time and Server name
*     
*      This allows capture of large XEvent traces, using per CPU partitioning, and then post sorting in sequence order so the
*      stream is in physical sequence order.
*
*      Arrays are owned by ## of partitions and dynamically expanded as events are read and added to them
*      Once all data is read each array is sorted in ascending or decending order (by a a group of parallel workers)
*      Once sorted the tournament sort is used by a single reader to gather the array streams and write to the output file
*     
*      Output File[n..] <----- Parallel Task [Ordered Output Array Segment 1] |                                                           | <----- Array[1]       
*                                                                             | <---- FWriteEntry <-- Gather Streams  <-- Parellel sorts  | <----- Array[2] <----- Distribute streams | Reader
*      Output File[n..] <----- Parallel Task [Ordered Output Array Segment N] |                                                           | <----- Array[3]
*
*      NOTE:  Testing shows parallel Output files do not have performance gains.  The original EventProvider, which is not thread safe,
*              prevents you opening and using a second copy of the input stream and a second EventProvider and adding lock() activities between the
*              RetreiveObject(EventLocator) and the Serialize causes more context swithing than just doing the work on a single thread.
* */

#define SEPERATE_OUTPUT

using System;
using System.Reflection;
using System.Collections;
using System.Collections.Generic;
using System.Text;
using System.Threading.Tasks;
using Microsoft.SqlServer.XEvent.Linq;
using System.IO;
using VersionInfo;
using System.Security.AccessControl;
using System.Runtime.InteropServices;
using ScatterGatherParallelNS;
using System.Threading;

namespace SortManager
{
    abstract class ISortLogOutput
    {
        abstract public void LogOutputMessage(string strMsg);
    }

#region DynamicList
    public sealed class TDynamicList<_T> : IDisposable
    {
        internal TDynamicList()
        {
            m_Items = new List<_T>();
        }

        public void Dispose()
        {
            if (null != m_Items)
            {
                m_Items.Clear();
            }

            m_Items = null;
        }

        internal void Add(_T newEntry)
        {
            lock (m_Items)
            {
                m_Items.Add(newEntry);
            }
        }

        internal bool FMoveNext()
        {
            return m_Walker.MoveNext();
        }

        internal _T GetCurrentEntry
        {
            get
            {
                System.Diagnostics.Debug.Assert(null != m_Walker.Current);
                return m_Walker.Current;
            }
        }

        internal bool FHasData
        {
            get
            {
                return null != m_Walker.Current;
            }
        }

        internal Int32 StoredItems
        {
            get { return m_Items.Count; }
        }

        internal void Sort(IComparer<_T> comparer)
        {
            if (null != m_Items && m_Items.Count > 0)
            {
                m_Items.Sort(comparer);
            }

            m_Walker = m_Items.GetEnumerator();
        }

        //      Members
        List<_T>                  m_Items;
        List<_T>.Enumerator       m_Walker;
    }
#endregion

#region DynamicArray
    public sealed class TDynamicArray<_T> : IDisposable
    {
        internal TDynamicArray()
        {
            m_Items = ArrayList.Synchronized(new ArrayList());
            m_iCurrentReadPos = -1;
        }

        public void Dispose()
        {
            if (null != m_Items)
            {
                m_Items.Clear();
            }

            m_iCurrentReadPos = -1;
            m_Items = null;
        }

        internal void Add(_T newEntry)
        {
            m_Items.Add(newEntry);
        }

        internal bool FMoveNext()
        {
            m_iCurrentReadPos++;
            return m_iCurrentReadPos < m_Items.Count;
        }

        internal _T GetCurrentEntry
        {
            get
            {
                return (_T) m_Items[m_iCurrentReadPos];
            }
        }

        public _T Item(int iItem)
        {
            return (_T) m_Items[iItem];
        }
 
        internal Int32 StoredItems
        {
            get { return m_Items.Count;  }
        }

        internal int Count
        {
            get { return m_Items.Count;  }
        }
       
        //      Members
        ArrayList           m_Items;
        System.Int32        m_iCurrentReadPos;           //     Current reader position 
    }
#endregion

#region SortMatchTree
    internal class TSortMatchingTree<_T> : IDisposable
    {
        internal TSortMatchingTree(TDynamicList<_T>[] partitions, bool bAscending = true)
        {
            m_bAscending = bAscending;
            m_Partitions = partitions;

            m_bWinners = new Int16[MAX_LEVELS+1, MAX_PARTITIONS];
                     
            //      Lowest level is always self
            for(Int16 iPartition = 0; iPartition < MAX_PARTITIONS; iPartition++)
            {
                m_bWinners[MAX_LEVELS, iPartition] = iPartition;
            }
        }

        public void Dispose()
        {
            m_bWinners = null;
        }
       
        public void PrimeEntries(IComparer<_T> comparer)
        {
            m_Comparer = comparer;

            for (Int16 iSlot = 0; iSlot < MAX_PARTITIONS; iSlot++)
            {
                FTreeInsert(iSlot);
            }
        }

        /*=========================================================
               0     1            Parents
            0  1  2  3           
           01 23 45 56 ...        Pairs
        =========================================================*/   
        //    Given an index who are we playing this match agianst
        //
        internal Int16 GetMatchTargetIndex(Int16 iPartition)
        {
            Int16 iMatchIndex = iPartition;
           
            iMatchIndex += (0 == iPartition % 2) ? (Int16)1 : (Int16)(-1);

            return iMatchIndex;
        }

        internal Int16 GetParentIndex(Int16 iPartition)
        {
            Int16 iParent;

            iParent = (Int16) (iPartition / 2);

            return iParent;
        }

        internal bool FTreeInsert(Int16 iPartitionIn)
        {
            bool    bRC = true;
               
            //===========================================================================
            //            https://en.wikipedia.org/wiki/Tournament_sort
            //            https://en.wikipedia.org/wiki/Heapsort
            //            https://en.wikipedia.org/wiki/Merge_sort
            //===========================================================================       
            Int16 iPartition = iPartitionIn;
           
            for (Int16 iCurrentLevel = MAX_LEVELS; iCurrentLevel > 0; iCurrentLevel--)
            {
                Int16 iPartnerPartition = GetMatchTargetIndex(iPartition);
                Int16 iParent = GetParentIndex(iPartition);

                Int16 iPartnerCompare = m_bWinners[iCurrentLevel, iPartnerPartition];
                Int16 iCurrentCompare = m_bWinners[iCurrentLevel, iPartition];

                Int16 iPrevWinner = m_bWinners[iCurrentLevel - 1, iParent];

                if (INVALID_PARTITION == iPartnerCompare || false == m_Partitions[iPartnerCompare].FHasData)
                {
                    m_bWinners[iCurrentLevel - 1, iParent] = iCurrentCompare;
                }
                else if (INVALID_PARTITION == iCurrentCompare || false == m_Partitions[iCurrentCompare].FHasData)
                {
                    m_bWinners[iCurrentLevel - 1, iParent] = iPartnerCompare;
                }
                else
                {
                    if (true == m_bAscending)
                    {
                        if(m_Comparer.Compare(m_Partitions[iCurrentCompare].GetCurrentEntry,  m_Partitions[iPartnerCompare].GetCurrentEntry) < 0)
                        {
                            m_bWinners[iCurrentLevel - 1, iParent] = iCurrentCompare;
                        }
                        else
                        {
                            m_bWinners[iCurrentLevel - 1, iParent] = iPartnerCompare;
                        }
                    }
                    else
                    {
                        if (m_Comparer.Compare(m_Partitions[iCurrentCompare].GetCurrentEntry, m_Partitions[iPartnerCompare].GetCurrentEntry) > 0)
                        {
                            m_bWinners[iCurrentLevel - 1, iParent] = iCurrentCompare;
                        }
                        else
                        {
                            m_bWinners[iCurrentLevel - 1, iParent] = iPartnerCompare;
                        }
                    }
                }

                if (iPrevWinner == m_bWinners[iCurrentLevel - 1, iParent] && iPrevWinner != iPartitionIn)        //     I changed so we have to recheck assumptions of previous matches
                {
                    break;
                }

                iPartition = iParent;
            }
           
            return bRC;
        }

        internal bool RemoveHead(out _T entry)
        {
            bool bRC = false;
            Int16 iWinner = m_bWinners[0, 0];

            if (INVALID_PARTITION == iWinner || false == m_Partitions[iWinner].FHasData)
            {
                ;
            }
            else
            {
                entry = m_Partitions[iWinner].GetCurrentEntry;

                m_Partitions[iWinner].FMoveNext();        //  FTreeInsert has to be called in true or false case
                FTreeInsert(iWinner);

                bRC = true;
                return bRC;
            }

            entry = default(_T);
            return bRC;
        }

        internal void ValidateAllProcessed()
        {
            for (int iPartition = 0; iPartition < MAX_PARTITIONS; iPartition++)
            {
                System.Diagnostics.Debug.Assert( false == m_Partitions[iPartition].FHasData);
            }
        }

        //      Members and Constants
        public const int MAX_PARTITIONS     = 32;           //     Has to be even number to allow matching to work properly - must be power of 2 to handle max levels
        public const int INVALID_PARTITION  = 255;          //     No further entries in the partition
        public const int MAX_LEVELS         = 5;
       
        TDynamicList<_T>[]          m_Partitions;           //  Reference to the partitions for actual values
        IComparer<_T>               m_Comparer;             //  Reference to comparision
        Int16[,]                    m_bWinners;             //  Tournament tree storage
        bool                        m_bAscending;           //  Sort direction
    }
   
    public abstract class ISortReaderWriter<_T>
    {
        abstract public bool FReadEntry(out _T entry, ref int iEnumerator, int iStepping);
        abstract public bool FWriteEntry(_T entry);

        abstract public void ClearReadAhead();
        abstract public void DoReadAhead(int iSourceFileSlot);
    }

    public class TSortManager<_T> : ISortReaderWriter<_T>, IDisposable
    {
        const Int32 OUTPUT_CHIMB = 250000;

        public override void ClearReadAhead()
        {
            throw new NotImplementedException();
        }
        public override void DoReadAhead(int iSourceFileSlot)
        {
            throw new NotImplementedException();
        }

        internal TSortManager(ISortLogOutput oSortOutput)
        {
            m_oSortOutput = oSortOutput;

            m_PartitionedItemStorage = new TDynamicList<_T>[TSortMatchingTree<_T>.MAX_PARTITIONS];

            for (int iSlot = 0; iSlot < TSortMatchingTree<_T>.MAX_PARTITIONS; iSlot++)
            {
                m_PartitionedItemStorage[iSlot] = new TDynamicList<_T>();
            }

            m_MatchingTree = new TSortMatchingTree<_T>(m_PartitionedItemStorage);
           
            m_i32ActivePartition = 0;
            m_ui64TotalReadEntries = 0;
            m_ui64TotalWriteEntries = 0;

            m_stopWatch = new System.Diagnostics.Stopwatch();
        }

        public void Dispose()
        {
            Dispose(true);
            GC.SuppressFinalize(this);
        }

        protected virtual void Dispose(bool bDisposing)
        {
            if (true == bDisposing)
            {
                m_MatchingTree.Dispose();
                m_MatchingTree = null;

                if (null != m_PartitionedItemStorage)
                {
                    for (Int32 iSlot = 0; iSlot < TSortMatchingTree<_T>.MAX_PARTITIONS; iSlot++)
                    {
                        m_PartitionedItemStorage[iSlot] = null;
                    }
                }

                m_PartitionedItemStorage = null;
                m_stopWatch = null;
            }
        }

        public void LogOutputMessage(string strMsg)
        {
            m_oSortOutput.LogOutputMessage(strMsg);
        }

        internal int ReadCPS
        {
            get
            {
                return Math.Min(Math.Max(2, Environment.ProcessorCount - 1), 4);
            }
        }
       
        static int m_iReaderThreadId = -1;
        internal void ReaderThread()
        {
            int iEnumerator = System.Threading.Interlocked.Increment(ref m_iReaderThreadId);
            int iCPUs = ReadCPS;
          
            _T entry;

            while (true == FReadEntry(out entry, ref iEnumerator, iCPUs))
            {
                Add(entry);

                System.Int64 i64Read = System.Threading.Interlocked.Increment(ref m_ui64TotalReadEntries);
                if (0 == i64Read % OUTPUT_CHIMB)
                {
                    LogOutputMessage(String.Format("     Entries read: {0} : {1} per sec", i64Read, (UInt64)((double)i64Read / Math.Max(1, m_stopWatch.ElapsedMilliseconds / 1000.0))));
                }
            }
        }

        internal void DoSort(IComparer<_T> comparer)
        {
            _T entry;

            m_stopWatch.Start();

            LogOutputMessage("Loading partitions");

            int iCPUs = ReadCPS;
            System.Threading.Thread[] readThreads = new System.Threading.Thread[iCPUs];

            for (int iThread = 0; iThread < iCPUs; iThread++)
            {
                readThreads[iThread] = new System.Threading.Thread(ReaderThread);
                readThreads[iThread].Start();
            }

            foreach (System.Threading.Thread thread in readThreads)
            {
                thread.Join();
            }

            LogOutputMessage("Reads complete, invoking parallel sort of partitions");
            SetupSortForExternalConsumption(comparer);

            ClearReadAhead();
          
            LogOutputMessage("Gathering partitions and spooling output");

            Int64 ui64StartWriteMS = m_stopWatch.ElapsedMilliseconds;
           
            while(true == RemoveHead(out entry))
            {
                m_ui64TotalWriteEntries++;
                if (false == FWriteEntry(entry))
                {
                    break;
                }
               
                if (0 == m_ui64TotalWriteEntries % OUTPUT_CHIMB)
                {
                    LogOutputMessage(String.Format("     Entries streamed: {0} : {1} per sec  {2}%", m_ui64TotalWriteEntries,
                       (UInt64)((double)m_ui64TotalWriteEntries / Math.Max(1, (m_stopWatch.ElapsedMilliseconds-ui64StartWriteMS) / 1000.0)),
                       (Int32) (((m_ui64TotalWriteEntries / (double) m_ui64TotalReadEntries) * 100.0))) );
                }
            }

            m_stopWatch.Stop();

            ClearReadAhead();

             System.Diagnostics.Debug.Assert(m_ui64TotalReadEntries == m_ui64TotalWriteEntries);
            m_MatchingTree.ValidateAllProcessed();
        }

        override public bool FReadEntry(out _T entry, ref int iEnumerator, int iStepping)
        {
            throw new NotImplementedException();
            //return false;
        }

        override public bool FWriteEntry(_T entry)
        {
            throw new NotImplementedException();
         }

        internal void Add(_T newEntry)
        {
            Int32 iPartition = System.Threading.Interlocked.Increment(ref m_i32ActivePartition) % TSortMatchingTree<_T>.MAX_PARTITIONS;
            m_PartitionedItemStorage[iPartition].Add(newEntry);
        }

        internal void SetupSortForExternalConsumption(IComparer<_T> comparer)
        {
            Parallel.ForEach(m_PartitionedItemStorage, (partitionList) =>
                                {
                                    partitionList.Sort(comparer);
                                    partitionList.FMoveNext();       //  Position to first entry in list or EOF
                                }
                );
           
             m_MatchingTree.PrimeEntries(comparer);
        }

        internal bool RemoveHead(out _T entry)
        {
            return m_MatchingTree.RemoveHead(out entry);
        }

        internal Int64 Write_Stats
        {
            get { return m_ui64TotalWriteEntries;  }
        }

        internal Int64 Read_Stats
        {
            get { return m_ui64TotalReadEntries; }
        }

        internal System.Diagnostics.Stopwatch StopWatch
        {
            get { return m_stopWatch;  }
        }

        //      Members 
        TDynamicList<_T>[]                  m_PartitionedItemStorage;
        Int32                               m_i32ActivePartition;
        TSortMatchingTree<_T>               m_MatchingTree;
        long                                m_ui64TotalReadEntries;
        long                                m_ui64TotalWriteEntries;
        System.Diagnostics.Stopwatch        m_stopWatch;
        SortManager.ISortLogOutput          m_oSortOutput;
    }
#endregion
   
}           //      End if generic sort manager namespace

namespace MergeXEvents
{
#region SortEntry
    public class CXESortEntry
    {
        internal CXESortEntry()
        {
            m_timestamp = default(DateTimeOffset);
            m_ui64EventSequence = 0;       //  0 indicates NOT set
            m_ui64ServerNameHash = 0;
            m_iCookie = 0;
            m_EventLocator = null;         /  This is only unique at a file level so the cookie used to get the proper file is important to making sure we get back to the proper offset

            m_uiUniqueIdentity = (UInt64) System.Threading.Interlocked.Increment(ref s_uiUniqueIdentity);
        }

        internal CXESortEntry(DateTimeOffset timestamp, UInt64 ui64EventSeq, UInt64 uiServerNameHash)
        {
            m_timestamp = timestamp;
            m_ui64EventSequence = ui64EventSeq;
            m_ui64ServerNameHash = uiServerNameHash;
            m_iCookie = 0;
            m_EventLocator = null;

            //      Wanted to use memory address of the entry but with GC moving things we needed a stable value
            //
            //      This is added because the data could have duplicate entries.  For example if you merge files from
            //      2 different servers the event sequence could be repeated (1 from each trace)
            //
            m_uiUniqueIdentity = (UInt64) System.Threading.Interlocked.Increment(ref s_uiUniqueIdentity);
        }

        internal UInt64 EventSequence
        {
            get { return m_ui64EventSequence; }
            set { m_ui64EventSequence = value; }
        }

        internal UInt64 ServerNameHash
        {
            get { return m_ui64ServerNameHash; }
            set { m_ui64ServerNameHash = value; }
        }

        internal DateTimeOffset Timestamp
        {
            get { return m_timestamp; }
            set { m_timestamp = value; }
        }

        internal UInt64 UniqueIdentity
        {
            get { return m_uiUniqueIdentity;  }
        }

        public EventLocator EventLocator
        {
            get { return m_EventLocator; }
            set { m_EventLocator = value; }
        }

        public Int32 Cookie
        {
            get { return m_iCookie;  }
            set { m_iCookie = value;  }
        }

        //      Members
        EventLocator    m_EventLocator;         //  So we can use this to get the event later
        Int32           m_iCookie;              //  External cookie for sort reader and writer logic
        DateTimeOffset  m_timestamp;            //  timestamp on the event
        UInt64          m_ui64EventSequence;    //  event_sequence action
        UInt64          m_ui64ServerNameHash;   //  GetHashCode of the server_principle_name action
        UInt64          m_uiUniqueIdentity;     //  Used to make sure we have a final sort order if timestamps and server names match without event sequence

        static long s_uiUniqueIdentity;
    }
#endregion

#region Comparision
    class CXEComparerSeqTimeServer : IComparer<CXESortEntry>
    {
        int IComparer<CXESortEntry>.Compare(CXESortEntry xeLeft, CXESortEntry xeRight)
        {
            Int32 iRC = 0;      //  Note we have UInt64s so we can't do simple match or it can overflow the Int32 return value

            bool bDoTimestamp = false;

            //      Event sequence
            if (0 != xeLeft.EventSequence && 0 != xeRight.EventSequence)
            {
                if (xeLeft.EventSequence < xeRight.EventSequence)
                    iRC = -1;
                else if (xeLeft.EventSequence > xeRight.EventSequence)
                    iRC = 1;
                else
                    bDoTimestamp = true;
            }
            else
            {
                bDoTimestamp = true;
            }
           
            if (true == bDoTimestamp)
            {
                if (xeLeft.Timestamp < xeRight.Timestamp)
                    iRC = -1;
                else if (xeLeft.Timestamp > xeRight.Timestamp)
                    iRC = 1;
            }

            if (0 == iRC)
            {
                if (0 != xeLeft.ServerNameHash && 0 != xeRight.ServerNameHash)
                {
                    if (xeLeft.ServerNameHash < xeRight.ServerNameHash)
                        iRC = -1;
                    else if (xeLeft.ServerNameHash > xeRight.ServerNameHash)
                        iRC = 1;
                }
            }

            if (0 == iRC)
            {
                if (xeLeft.UniqueIdentity < xeRight.UniqueIdentity)
                    iRC = -1;
                else if (xeLeft.UniqueIdentity > xeRight.UniqueIdentity)
                    iRC = 1;
            }
           
            return iRC;
        }

    }
#endregion

#region StreamToFile
    internal class CSourceFileEntry :  IDisposable
    {
        public CSourceFileEntry(String strFileName)
        {
            m_strXELName = strFileName;
            m_XEventSourceFile = null;
            m_XEEventEnumerator = null;
            #if SEPERATE_OUTPUT
                m_XEventSourceFileDuplicateForOutput = null;
                m_DuplicateOpenSemaphore = new Semaphore(0, 1);
            #endif
        }

        public void Dispose()
        {
            m_strXELName = String.Empty;

            if(null != m_XEventSourceFile)
            {
                m_XEventSourceFile.Dispose();
                m_XEventSourceFile = null;
            }

            #if SEPERATE_OUTPUT
                if(null != m_XEventSourceFileDuplicateForOutput)
                {
                    m_XEventSourceFileDuplicateForOutput.Dispose();
                    m_XEventSourceFileDuplicateForOutput = null;
                }

                if (null != m_DuplicateOpenSemaphore)
                {
                    m_DuplicateOpenSemaphore.Close();
                    m_DuplicateOpenSemaphore = null;
                }

            #endif
        }

        public String FileName
        {
            get { return m_strXELName;  }
        }

        public QueryableXEventData XELFile
        {
            get { return m_XEventSourceFile;  }
        }

        public IEnumerator<PublishedEvent> EventEnumerator
        {
            get { return m_XEEventEnumerator;  }
        }

        #if SEPERATE_OUTPUT
            public QueryableXEventData XELFileForOutput
            {
                get { return m_XEventSourceFileDuplicateForOutput; }
            }

            public IEnumerator<PublishedEvent> EventEnumeratorForOutput
            {
                get { return m_XEEventEnumeratorDuplicateForOutput; }
            }

            public void WaitForOutputOpen()
            {
                if (true == m_DuplicateOpenSemaphore.WaitOne())
                {
                    m_DuplicateOpenSemaphore.Release();
                }
            }
        #endif

        public void Open()
        {
            #if SEPERATE_OUTPUT
                //  Found if we double read the file into another provider we can pass the published
                //  event across these providers (same exact metadata and buffers) so we can seperate
                //  the retreive from serialize and increase the output speed
                //
                try
                {
                    m_XEventSourceFileDuplicateForOutput = new QueryableXEventData(m_strXELName);
                    m_XEEventEnumeratorDuplicateForOutput = m_XEventSourceFileDuplicateForOutput.GetEnumerator();
                }
                finally
                {
                    m_DuplicateOpenSemaphore.Release();
                }
            #endif

            m_XEventSourceFile = new QueryableXEventData(m_strXELName);
            m_XEEventEnumerator = m_XEventSourceFile.GetEnumerator();
        }

        public bool FIsOpen()
        {
            return null != m_XEEventEnumerator;
        }

        private QueryableXEventData         m_XEventSourceFile;
        private IEnumerator<PublishedEvent> m_XEEventEnumerator;
       
        #if SEPERATE_OUTPUT
                private QueryableXEventData         m_XEventSourceFileDuplicateForOutput;
                private IEnumerator<PublishedEvent> m_XEEventEnumeratorDuplicateForOutput;
                private Semaphore                   m_DuplicateOpenSemaphore;
        #endif
       
        private String                      m_strXELName;
    }

    public class CXEventSortStream : SortManager.TSortManager<CXESortEntry>
    {
        internal CXEventSortStream(SortManager.ISortLogOutput oLogOutput) :
            base(oLogOutput)
        {
            m_XEventSourceFiles = new SortManager.TDynamicArray<CSourceFileEntry>();
        }

        internal void DoSort()
        {
            DoSort(new CXEComparerSeqTimeServer());
        }

        override public bool FReadEntry(out CXESortEntry entry, ref int iEnumerator, int iStepping)
        {
            bool bRC = false;

            PublishedEvent publishedEvent = null;

            do
            {
                if (iEnumerator < m_XEventSourceFiles.StoredItems)
                {
                    if(false == m_XEventSourceFiles.Item(iEnumerator).FIsOpen())
                    {
                        m_XEventSourceFiles.Item(iEnumerator).Open();
                    }

                    if (true == m_XEventSourceFiles.Item(iEnumerator).EventEnumerator.MoveNext())
                    {
                        publishedEvent = m_XEventSourceFiles.Item(iEnumerator).EventEnumerator.Current;
                    }
                    else
                    {
                        iEnumerator += iStepping;
                    }
                }

            } while (null == publishedEvent && iEnumerator < m_XEventSourceFiles.StoredItems);

            if (null != publishedEvent)
            {
                entry = default(CXESortEntry);
                entry = new CXESortEntry();

                entry.EventLocator = publishedEvent.Location;
                entry.Cookie = iEnumerator;
                entry.Timestamp = publishedEvent.Timestamp;

                PublishedAction eventSeq;
                if (true == publishedEvent.Actions.TryGetValue(@"event_sequence", out eventSeq))
                {
                    entry.EventSequence = (UInt64) eventSeq.Value;
                }

                PublishedAction serverName;
                if (true == publishedEvent.Actions.TryGetValue(@"server_principal_name", out serverName))
                {
                    entry.ServerNameHash = (UInt64)serverName.Value.GetHashCode();
                }

                bRC = true;
            }
            else
            {
                entry = default(CXESortEntry);
            }

          
            return bRC;
        }

        override public bool FWriteEntry(CXESortEntry entry)
        {
            return true;
        }

        internal void AddInputFile(string strXELFile)
        {
            CSourceFileEntry se = new CSourceFileEntry(strXELFile);
            lock (m_XEventSourceFiles)
            {
                m_XEventSourceFiles.Add(se);
            }
        }

        internal SortManager.TDynamicArray<CSourceFileEntry> SourceFiles
        {
            get { return m_XEventSourceFiles; }
        }

        //      Members
        private SortManager.TDynamicArray<CSourceFileEntry> m_XEventSourceFiles;
    }

    class CParallelPayload
    {
        public CParallelPayload(CXEventFileToFileSortManager sortFileMgr, CXESortEntry sortEntry)
        {
            m_sortFileMgr = sortFileMgr;
            m_sortEntry = sortEntry;
            m_publishedEvent = null;
        }

        public CXESortEntry SortEntry
        {
            get { return m_sortEntry; }
        }

        public CXEventFileToFileSortManager SortFileMgr
        {
            get { return m_sortFileMgr; }
        }

        public PublishedEvent PublishedEvent
        {
            get { return m_publishedEvent; }
            set { m_publishedEvent = value;  }
        }

        private CXEventFileToFileSortManager m_sortFileMgr;
        private CXESortEntry m_sortEntry;
        private PublishedEvent m_publishedEvent;
    }

    internal class CScatterGatherStream : ScatterGatherParallelMgr<CParallelPayload>
    {
        public CScatterGatherStream() :
            base(2)
        {
        }

        override public void DoUnOrderedWork(CParallelPayload pPayload)
        {
            pPayload.SortFileMgr.DoUnorderedWork(pPayload);
        }
        override public void DoOrderedWork(CParallelPayload pPayload)
        {
            pPayload.SortFileMgr.DoOrderedWork(pPayload);
        }
       
        override public void OrderedWorkerStalled(CParallelPayload pPayload)
        {
        }

        override public void OrderedWorkerNoStall()
        {
        }

    }

    internal class CXEventFileToFileSortManager : CXEventSortStream
    {
        internal CXEventFileToFileSortManager(SortManager.ISortLogOutput oLogOutput)  :
            base(oLogOutput)
        {
            m_iCancelReadAhead = 0;
            m_ScatterGatherOutputStream = new CScatterGatherStream();
            m_strLastOutputName = String.Empty;
            m_lFileCount = 1;
            m_strOutputFile = String.Empty;
            m_strOutputDir = String.Empty;
            m_XEOutputStream = null;
            m_ui64RolloverCount = 0;

            m_dictReadAhead = new Dictionary<int, Thread>();
        }

        ~CXEventFileToFileSortManager()
        {
            CloseFile();
        }

        new internal void DoSort()
        {
            base.DoSort();
            m_ScatterGatherOutputStream.CompleteAllWork();
            CloseFile();
        }
      
        private void CloseFile()
        {
            if(String.Empty != m_strLastOutputName && null != m_XEOutputStream)
            {
                m_XEOutputStream.Dispose();
                m_XEOutputStream = null;
              
                string strDirectory = System.IO.Path.GetDirectoryName(m_strLastOutputName);
                strDirectory = System.IO.Path.GetFullPath(strDirectory);

                string strFileSpec = System.IO.Path.GetFileName(m_strLastOutputName);

                //  Strip extention
                strFileSpec = strFileSpec.Substring(0, strFileSpec.Length - 4);
                strFileSpec += "*.xel";

                IEnumerable<string> strFiles = System.IO.Directory.EnumerateFiles(strDirectory, strFileSpec);
               
                int iLoops = 0;
                foreach(string strFile in strFiles)
                {
                    if (0 != iLoops++)
                    {
                        SetNewOutputFileName();
                        iLoops++;
                    }

                    FileAttributes fa = File.GetAttributes(strFile);
                    if (FileAttributes.Compressed == (FileAttributes.Compressed & fa))
                    {
                        LogOutputMessage(" WARN: Performance may be impacted as output file appears to be using compression.");
                    }

                    System.IO.Directory.Move(strFile, m_strLastOutputName);
                }
                   
                m_strLastOutputName = String.Empty;
            }
        }

        internal void SetNewOutputFileName(CXESortEntry entry = null)
        {

            if (null != entry)
            {
                m_strLastOutputName = String.Format(@"{0}\{1}_0 ymd({2}-{3}-{4}) hms({5}-{6}-{7}.{8})_{9}.xel", m_strOutputDir, m_strOutputFile,
                            entry.Timestamp.ToUniversalTime().Year,
                            entry.Timestamp.ToUniversalTime().Month,
                            entry.Timestamp.ToUniversalTime().Day,
                            entry.Timestamp.ToUniversalTime().Hour,
                            entry.Timestamp.ToUniversalTime().Minute,
                            entry.Timestamp.ToUniversalTime().Second,
                            entry.Timestamp.ToUniversalTime().Millisecond,
                            m_lFileCount++);
            }
            else   //   Rollover path
            {
                string strCount = (m_lFileCount -1).ToString();     //  File count is incremented when the name was built
                string strFile = m_strLastOutputName.Substring(0, m_strLastOutputName.Length - 4 - strCount.Length);

                m_strLastOutputName = String.Format(@"{0}{1}.xel", strFile, m_lFileCount++);
            }

            if (true == File.Exists(m_strLastOutputName))
            {
                FileAttributes fa = File.GetAttributes(m_strLastOutputName);
                File.SetAttributes(m_strLastOutputName, fa & ~FileAttributes.ReadOnly);
                System.IO.File.Delete(m_strLastOutputName);
            }
        }

        internal void DoUnorderedWork(CParallelPayload payload)
        {
            lock (SourceFiles.Item(payload.SortEntry.Cookie).XELFile.EventProvider)
            {
                payload.PublishedEvent = SourceFiles.Item(payload.SortEntry.Cookie).XELFile.EventProvider.RetrieveEvent(payload.SortEntry.EventLocator);
            }
        }

#if DEBUG
        UInt64 m_ui64SeqCheck = 0;
#endif
        internal void DoOrderedWork(CParallelPayload payload)
        {
            //========================================================================
            //  Generate file name for output based on the timestamp so it is easy to see which file you want to open.
            //  Toggle close and open the output stream to force a numbered rollover
            //
            //      We use 1 millon because this is the current SSMS display limit
            //
            //      1,000,000 approaches 1GB too often so shrink the number of entries per output file
            //========================================================================
            if (null == m_XEOutputStream || 0 == (m_ui64RolloverCount + 1) % 500000)
            {
                CloseFile();
                SetNewOutputFileName(payload.SortEntry);

                //      Setup the file for output
                m_XEOutputStream = new XEventFileSerializer(m_strLastOutputName);
            }

#if DEBUG
            //  Safety check
            if (m_ui64SeqCheck < payload.SortEntry.EventSequence)
            {
                m_ui64SeqCheck = payload.SortEntry.EventSequence;
            }
            else
            {
                throw new Exception("UTILITY ERROR; Attempt to stream duplicate entry");
            }
#endif
            #if SEPERATE_OUTPUT
                SourceFiles.Item(payload.SortEntry.Cookie).XELFileForOutput.EventProvider.SerializeEvent(m_XEOutputStream, payload.PublishedEvent);
                m_ui64RolloverCount++;
            #else
                lock (SourceFiles.Item(payload.SortEntry.Cookie).XELFile.EventProvider)
                {
                    SourceFiles.Item(payload.SortEntry.Cookie).XELFile.EventProvider.SerializeEvent(m_XEOutputStream, payload.PublishedEvent);
                    m_ui64RolloverCount++;
                }
#endif
        }

        override public bool FWriteEntry(CXESortEntry entry)
        {
            CParallelPayload payload = new CParallelPayload(this, entry);
             
            //      Do inline work
             #if SEPERATE_OUTPUT
                m_ScatterGatherOutputStream.FQueueWorkItem(payload);
            #else
                m_ScatterGatherOutputStream.DoUnOrderedWork(payload);
                m_ScatterGatherOutputStream.DoOrderedWork(payload);
            #endif
      
            return true;
        }

        public override void ClearReadAhead()
        {
            //  Let everyone know they are done
            Interlocked.Increment(ref m_iCancelReadAhead);

            lock(m_dictReadAhead)
            {
                //  Make sure all active threads are done
                foreach(KeyValuePair<int, Thread> oKV in m_dictReadAhead)
                {
                    oKV.Value.Join();
                }

                m_dictReadAhead.Clear();
            }

            //  Reset for next execution
            m_iCancelReadAhead = 0;
        }
        public override void DoReadAhead(int iFileEnumSlot)
        {
            int iCookie = -1;
            Thread tReadAhead = new Thread(DoReadAheadInternal);

            lock (m_dictReadAhead)
            {
                Thread tReadAheadDummy;
                if (false == m_dictReadAhead.TryGetValue(iFileEnumSlot, out tReadAheadDummy))
                {
                    iCookie = iFileEnumSlot;
                    m_dictReadAhead.Add(iFileEnumSlot, tReadAhead);
                }
            }

            //  Do we need to start the read-ahead to suck data into file system cache
            if (iCookie > -1)
            {
                tReadAhead.Start(iCookie);
            }
        }

        internal void DoReadAheadInternal(object oParms)
        {
            try
            {
                int iCookie = (int) oParms;
                string strName = SourceFiles.Item(iCookie).FileName;
#if DEBUG
                LogOutputMessage(String.Format(" INFO: Starting file system cache read-ahead for {0}",  strName));
#endif
                //      We just open the file and read in chunks, allowing caching to fill file system cache quickly
                const int c_iMaxReadSize = ((1024 * 1024) * 4);
               
                using (FileStream s = new FileStream(strName, FileMode.Open, FileAccess.Read, FileShare.Read | FileShare.Write | FileShare.Delete, c_iMaxReadSize, FileOptions.SequentialScan))
                {
                    byte[] bData = new byte[c_iMaxReadSize];

                    ulong ulLoops = 0;
                    int iBytesRead = s.Read(bData, 0, c_iMaxReadSize);
                    while (0 == m_iCancelReadAhead && iBytesRead > 0)
                    {
                        iBytesRead = 0;
                        ulLoops++;
                        if(0 == ulLoops % 100)
                        {
                            Thread.Sleep(10);
                        }

                        iBytesRead = s.Read(bData, 0, c_iMaxReadSize);
                    }
                }

                #if SEPERATE_OUTPUT
                    SourceFiles.Item(iCookie).WaitForOutputOpen();
                    if (null != SourceFiles.Item(iCookie).EventEnumeratorForOutput)
                    {
                        while (true == SourceFiles.Item(iCookie).EventEnumeratorForOutput.MoveNext()) ;
                    }
                #endif
            }
            catch(Exception e)
            {
                LogOutputMessage(String.Format(" INFO: read ahead encounted exection for {0} ignoring.", e.ToString()));
            }
        }

        internal void SetOutputFile(string strXELFile)
        {
            m_strOutputFile = System.IO.Path.GetFileNameWithoutExtension(strXELFile);
            m_strOutputDir = System.IO.Path.GetDirectoryName(strXELFile);
        }
     
        //      Members
        Dictionary<int, Thread>                                 m_dictReadAhead;
        CScatterGatherStream                                    m_ScatterGatherOutputStream;
        string                                                  m_strOutputFile;
        string                                                  m_strOutputDir;
        XEventFileSerializer                                    m_XEOutputStream;
        string                                                  m_strLastOutputName;
        int                                                     m_lFileCount;
        int                                                     m_iCancelReadAhead;
        UInt64                                                  m_ui64RolloverCount;
    }
#endregion

    class CSortOutput : SortManager.ISortLogOutput, IDisposable
    {
        public CSortOutput(string strOutputPath)
        {
            m_OutputFile = null;

            string strDirectory = System.IO.Path.GetDirectoryName(strOutputPath);

            Program.FCreateDirectoryWithACLs(strDirectory);

            m_OutputFile = new System.IO.StreamWriter(String.Format(@"{0}\MergeEvents.log", strDirectory));
        }

        public void Dispose()
        {
            Dispose(true);
            GC.SuppressFinalize(this);
        }

        protected void Dispose(bool bDisposing)
        {
            if(true == bDisposing)
            {
                m_OutputFile.Close();
                m_OutputFile = null;
            }
        }

        override public void LogOutputMessage(string strMsg)
        {
            Program.LogToConsole(strMsg);
            if (null != m_OutputFile)
            {
                m_OutputFile.WriteLine("{0} {1} {2}", DateTime.Now, System.Threading.Thread.CurrentThread.ManagedThreadId, strMsg);
                m_OutputFile.Flush();
            }
        }

        private System.IO.StreamWriter m_OutputFile;
    }

    class Program
    {
        static internal void LogToConsole(string strMsg)
        {
            Console.WriteLine("{0} {1} {2}", DateTime.Now, System.Threading.Thread.CurrentThread.ManagedThreadId, strMsg);
        }

        static internal void ShowUsage()
        {
            LogToConsole("");
            LogToConsole("Usage");
            LogToConsole("------------------------------------------------------------------------");
            LogToConsole("/? Present this help information");
            LogToConsole("");
            LogToConsole("EXAMPLE:  MergeXEvents.exe <input spec> <output spec>");
            LogToConsole("          MergeXEvents.exe \"C:\\temp\\XEvent_0_130494871596480000.xel\" \"C:\\temp\\Sorted.xel\" ");
            LogToConsole("");
            LogToConsole("<input spec> any valid directory and wildcard.   All files matching wildcard in the directory will be processed as input");
            LogToConsole("<output spec> is any valid file name where output for the merge can be stored, including rollover files.");
            LogToConsole("");
        }

#region AssemblyInformation
        static public string AssemblyTitle
        {
            get
            {
                object[] attributes = Assembly.GetExecutingAssembly().GetCustomAttributes(typeof(AssemblyTitleAttribute), false);
                if (attributes.Length > 0)
                {
                    AssemblyTitleAttribute titleAttribute = (AssemblyTitleAttribute)attributes[0];
                    if (false == String.IsNullOrEmpty(titleAttribute.Title))
                        return titleAttribute.Title;
                }

                return System.IO.Path.GetFileNameWithoutExtension(Assembly.GetExecutingAssembly().CodeBase);
            }
        }

        static public string AssemblyVersion
        {
            get
            {
                return VersionStrings.VERSION_MAJOR_STR + VersionStrings.VERSION_MINOR_STR + VersionStrings.VERSION_BUILD_STR;
            }
        }

       static  public string AssemblyDescription
        {
            get
            {
                object[] attributes = Assembly.GetExecutingAssembly().GetCustomAttributes(typeof(AssemblyDescriptionAttribute), false);
                if (attributes.Length == 0)
                    return "";

                return ((AssemblyDescriptionAttribute)attributes[0]).Description;
            }
        }

        static public string AssemblyProduct
        {
            get
            {
                object[] attributes = Assembly.GetExecutingAssembly().GetCustomAttributes(typeof(AssemblyProductAttribute), false);
                if (attributes.Length == 0)
                    return "";
               
                return ((AssemblyProductAttribute)attributes[0]).Product;
            }
        }

        static public string AssemblyCopyright
        {
            get
            {
                object[] attributes = Assembly.GetExecutingAssembly().GetCustomAttributes(typeof(AssemblyCopyrightAttribute), false);
                if (attributes.Length == 0)
                    return "";

                return ((AssemblyCopyrightAttribute)attributes[0]).Copyright;
            }
        }

        static public string AssemblyCompany
        {
            get
            {
                object[] attributes = Assembly.GetExecutingAssembly().GetCustomAttributes(typeof(AssemblyCompanyAttribute), false);
                if (attributes.Length == 0)
                    return "";

                return ((AssemblyCompanyAttribute)attributes[0]).Company;
            }
        }
        #endregion

#region DIR_ACLS
        const int FILE_PERSISTENT_ACLS = 0x00000008;

        [DllImport("kernel32.dll", CharSet = CharSet.Unicode)]
        public static extern bool GetVolumeInformation(string lpRootPathName,
                                                            StringBuilder lpVolumeNameBuffer,
                                                            [MarshalAs(UnmanagedType.U4)] int nVolumeNameSize,
                                                            [MarshalAs(UnmanagedType.U4)] ref int lpVOlumeSerialNumber,
                                                            [MarshalAs(UnmanagedType.U4)] ref int lpMaxComponentLen,
                                                            [MarshalAs(UnmanagedType.U4)] ref int lpFileSytemFlags,
                                                            StringBuilder lpFileSystemNameBuffer,
                                                            [MarshalAs(UnmanagedType.U4)] int nFileSystemNameSize);

        static public bool FCreateDirectoryWithACLs(string strDirName)
        {
            bool bRC = false;

            System.IO.Directory.CreateDirectory(strDirName);
            DirectorySecurity dss = new DirectorySecurity();

            try
            {
                dss.AddAccessRule(new FileSystemAccessRule(@"NT AUTHORITY\ANONYMOUS LOGON",
                                                            FileSystemRights.FullControl,
                                                            AccessControlType.Deny));
            }
            catch (Exception e)
            {
                LogToConsole(String.Format("Exception encountered setting ANONYMOUS access rule {0}", e.ToString()));
                throw;
            }
       
            try
            {
                dss.AddAccessRule(new FileSystemAccessRule(System.Security.Principal.WindowsBuiltInRole.Guest.ToString(),
                                                            FileSystemRights.FullControl,
                                                            AccessControlType.Deny));
            }
            catch (Exception e)
            {
                LogToConsole(String.Format("Exception encountered setting GUEST access rule {0}", e.ToString()));
                throw;
            }

            try
            {
                dss.AddAccessRule(new FileSystemAccessRule(@"BuiltIn\Administrators",
                                                            FileSystemRights.ReadExtendedAttributes | FileSystemRights.ReadAttributes | FileSystemRights.Synchronize,
                                                            AccessControlType.Allow));
            }
            catch (Exception e)
            {
                LogToConsole(String.Format("Exception encountered setting Administrators access rule {0}", e.ToString()));
                throw;
            }

            String strCurrentUser = System.Security.Principal.WindowsIdentity.GetCurrent().Name;

            try
            {
                dss.AddAccessRule(new FileSystemAccessRule(strCurrentUser,
                                                           FileSystemRights.FullControl,
                                                           AccessControlType.Allow));
            }
            catch (Exception e)
            {
                LogToConsole(String.Format("Exception encountered setting current user access rule for: " + strCurrentUser, e));
                throw;
            }

            String strRoot = System.IO.Path.GetPathRoot(strDirName);
            if (false == strRoot.EndsWith("\\", StringComparison.Ordinal))
            {
                strRoot += "\\";
            }

            int iSerialNumber = 0;
            int iMaxComponent = 0;
            int iFlags = 0;

            bRC = GetVolumeInformation(strRoot, null, 0, ref iSerialNumber, ref iMaxComponent, ref iFlags, null, 0);
            if (false == bRC)
            {
                LogToConsole("Attempt to obtain volume information failed.");
            }
            else if (FILE_PERSISTENT_ACLS != (iFlags & FILE_PERSISTENT_ACLS))
            {
                LogToConsole("ERROR: Specified location does not support proper access controls.");
                bRC = false;
            }

            if(true == bRC)
            {
                System.IO.Directory.SetAccessControl(strDirName, dss);
                bRC = true;
            }

            return bRC;
        }
#endregion

        //==============================================================
        static int Main(string[] args)
        {
            LogToConsole(AssemblyTitle);
            LogToConsole(String.Format("Version {0}", AssemblyVersion));
            LogToConsole(AssemblyCopyright);
            LogToConsole("------------------------------------------------------------");
            LogToConsole("");
           
            if(args.Length != 2)
            {
                ShowUsage();
                return -1;
            }

            return DoWork(args[0], args[1]);
        }

        private static int DoWork(string strInput, string strOutput)
        {
            int iRC = -100;

            try
            {
                CXEventFileToFileSortManager xeSortMgr = new CXEventFileToFileSortManager(new CSortOutput(strOutput));
                string strDirectory = System.IO.Path.GetDirectoryName(strInput);
                strDirectory = System.IO.Path.GetFullPath(strDirectory);
                string strFileSpec = System.IO.Path.GetFileName(strInput);
                string[] strParts = strFileSpec.Split('_');

                strFileSpec = String.Empty;
               
                for (int iLoop = 0; iLoop < strParts.Length - 2; iLoop++ )
                {
                    strFileSpec += strParts[iLoop] + '_';
                }

                if(String.Empty != strFileSpec)
                {
                    strFileSpec += "*.xel";
                }

                int iFiles = 0;
                IEnumerable<string> strFiles = System.IO.Directory.EnumerateFiles(strDirectory, strFileSpec);
                Parallel.ForEach (strFiles, (strFile) =>
                {
                    LogToConsole(String.Format("VALIDATING INPUT SOURCE: {0}", strFile));
                
                    xeSortMgr.AddInputFile(strFile);
                    System.Threading.Interlocked.Increment(ref iFiles);

                    //      Since the XE Linq reader uses file mapping compression defeats the alignment attempts on sector boundaries
                    //
                    FileAttributes fa = File.GetAttributes(strFile);
                    if (FileAttributes.Compressed == (FileAttributes.Compressed & fa))
                    {
                        LogToConsole(" WARN: Performance may be impacted as input file appears to be using compression.");
                    }
                });

                xeSortMgr.ClearReadAhead();

                if(iFiles > 0)
                {
                    strDirectory = System.IO.Path.GetDirectoryName(strOutput);
                    strDirectory = System.IO.Path.GetFullPath(strDirectory);

                    if (false == FCreateDirectoryWithACLs(strDirectory))
                    {
                        LogToConsole(String.Format("ERROR: Unable to create or set ALCs on {0}", strDirectory));
                        iRC = -6;
                    }
                    else
                    {
                        strFileSpec = System.IO.Path.GetFullPath(strOutput);

                        LogToConsole(String.Format("      OUTPUT: {0}", strFileSpec));
                        {
                            LogToConsole("Removing files in output location with matching file name pattern.");

                            string strFileNameNoExt = System.IO.Path.GetFileNameWithoutExtension(strFileSpec);
                            string strLastOutputNamePattern = String.Format(@"{0}_0 ymd(*) hms(*)_*.xel", strFileNameNoExt);
                            strFiles = System.IO.Directory.EnumerateFiles(strDirectory, strLastOutputNamePattern);

                            foreach(string strFile in strFiles)
                            {
                                FileAttributes fa = File.GetAttributes(strFile);
                                File.SetAttributes(strFile, fa & ~FileAttributes.ReadOnly);
                                System.IO.File.Delete(strFile);
                            }
                        }

                        xeSortMgr.SetOutputFile(strFileSpec);

                        LogToConsole("");
                        LogToConsole("Beginning sort action ...");

                        xeSortMgr.DoSort();
                       
                        LogToConsole(String.Format("Sort action has completed (Elapsed {0} sec)", xeSortMgr.StopWatch.ElapsedMilliseconds / 1000.0));
                        LogToConsole("");

                        LogToConsole(String.Format("    Events read: {0}", xeSortMgr.Read_Stats));
                        LogToConsole(String.Format("Events streamed: {0}", xeSortMgr.Write_Stats));

                        if (xeSortMgr.Read_Stats != xeSortMgr.Write_Stats)
                            iRC = -3;
                        else if (0 == xeSortMgr.Read_Stats)
                            iRC = -4;
                        else
                            iRC = 0;
                    }
                }
                else
                {
                    LogToConsole("ERROR: Could not locate any input files.");
                    iRC = -2;
                }

            }
            catch (Exception e)
            {
                LogToConsole("");
                LogToConsole("Error / Exception encountered");
                LogToConsole("==========================================================");
                LogToConsole(e.ToString());
                iRC = -99;
            }

            return iRC;
        }

    }
}

ScatterGather.cs

using System;
using System.Collections.Concurrent;
using System.Collections;
using System.Threading;

namespace ScatterGatherParallelNS
{
    #region ScatterGatherParallelMgr_IMPL
    public abstract class ScatterGatherParallelMgr<_TPayload> : IDisposable
    {
        private BlockingCollection<CMsg<_TPayload>> m_FreeQueue;        //  Msg has been processed, it can handle other payload
        private BlockingCollection<CMsg<_TPayload>> m_UnOrderedQueue;   //  Msg is dirty and needs processed when on this queue
        private BlockingCollection<CMsg<_TPayload>> m_OrderedQueue;     //  Msg is ready for final processing when on this queue
        private ArrayList m_alUnOrderedThreads;
        private Thread m_OrderedThread;
        private bool m_bInitialized;
        private Int32 m_TotalUnOrderedThreads;
        private Int32 m_TotalMessages;
        private bool m_bOrderedIsCurrentlyStalled;
        private Int64 m_iTotalOrderStalls;

        public ScatterGatherParallelMgr(int iUnorderedLimit)
        {
            m_TotalUnOrderedThreads = Math.Max(2, Environment.ProcessorCount-1);
            m_TotalUnOrderedThreads = Math.Min(iUnorderedLimit, m_TotalUnOrderedThreads);
            m_TotalMessages = m_TotalUnOrderedThreads * 128;           //  ## per CPU
            m_bInitialized = false;
            m_FreeQueue = null;
            m_UnOrderedQueue = null;
            m_OrderedQueue = null;
            m_alUnOrderedThreads = null;
            m_OrderedThread = null;
            m_bOrderedIsCurrentlyStalled = false;
            m_iTotalOrderStalls = 0;
        }

        public bool FIsOrderedWorkerStalled
        {
            get { return m_bOrderedIsCurrentlyStalled; }
        }

        public Int64 TotalOrderedWorkerStalls
        {
            get { return m_iTotalOrderStalls; }
        }

        virtual public void AbortAllWorkers()
        {
            if (null != m_alUnOrderedThreads)
            {
                foreach (Thread t in m_alUnOrderedThreads)
                {
                    try
                    {
                        t.Abort();
                    }
                    catch (Exception)
                    {
                    }
                }
            }

            if (null != m_OrderedThread)
            {
                try
                {
                    m_OrderedThread.Abort();
                }
                catch (Exception)
                {
                }
            }
        }

        private void StartWorkersIfNecessary()
        {
            if (false == m_bInitialized)
            {
                m_bOrderedIsCurrentlyStalled = false;
                m_iTotalOrderStalls = 0;

                m_FreeQueue = new BlockingCollection<CMsg<_TPayload>>();            //  Order not required
                m_UnOrderedQueue = new BlockingCollection<CMsg<_TPayload>>();       //  Order not required

                ConcurrentQueue<CMsg<_TPayload>> q = new ConcurrentQueue<CMsg<_TPayload>>();  //  Order required
                m_OrderedQueue = new BlockingCollection<CMsg<_TPayload>>(q);

                for (Int32 iMsg = 0; iMsg < m_TotalMessages; iMsg++)
                {
                    m_FreeQueue.Add(new CMsg<_TPayload>());
                }

                m_alUnOrderedThreads = new ArrayList();

                for (Int32 iUnOrdered = 0; iUnOrdered < m_TotalUnOrderedThreads; iUnOrdered++)
                {
                    ThreadStart ts = new ThreadStart(UnOrderedEntryPoint);
                    Thread t = new Thread(ts);
                    m_alUnOrderedThreads.Add(t);
                    t.Start();
                }

                {
                    ThreadStart ts = new ThreadStart(OrderedEntryPoint);
                    m_OrderedThread = new Thread(ts);
                    m_OrderedThread.Start();
                }

                m_bInitialized = true;
            }
        }

        protected virtual void Dispose(bool bDispose)
        {
            CompleteAllWork();  //  If not shutdown do so now
        }

        [System.Diagnostics.CodeAnalysis.SuppressMessage("Microsoft.Design", "CA1063")]
        public virtual void Dispose()
        {
            Dispose(true);
            GC.SuppressFinalize(this);
        }

        public void CompleteAllWork()
        {
            if(null != m_UnOrderedQueue)
            {
                m_UnOrderedQueue.CompleteAdding();
            }

            if (null != m_alUnOrderedThreads)
            {
                foreach (Thread t in m_alUnOrderedThreads)
                {
                    t.Join();
                }

                m_alUnOrderedThreads = null;
            }

            if (null != m_OrderedQueue)
            {
                m_OrderedQueue.CompleteAdding();
            }

            if (null != m_OrderedThread && null != m_OrderedQueue)
            {
                m_OrderedThread.Join();
                m_OrderedThread = null;
            }

            if(null != m_FreeQueue)
            {
                m_FreeQueue = null;
            }

            if(null != m_UnOrderedQueue)
            {
                m_UnOrderedQueue = null;
            }

            if(null != m_OrderedQueue)
            {
                m_OrderedQueue = null;
            }

            m_bInitialized = false;
        }

        public abstract void DoUnOrderedWork(_TPayload pPayload);

        private void UnOrderedEntryPoint()
        {
            try
            {
                foreach (CMsg<_TPayload> cMsg in m_UnOrderedQueue.GetConsumingEnumerable())
                {
                    DoUnOrderedWork(cMsg.Payload);
                    cMsg.MarkReadyForOrderedProcessing();
                }
            }
            catch (ThreadAbortException)        //  Silent, main would have issued this
            {
            }
            catch (OperationCanceledException)
            {
            }
        }

        public abstract void DoOrderedWork(_TPayload pPayload);
        public abstract void OrderedWorkerStalled(_TPayload pPayload);
        public abstract void OrderedWorkerNoStall();

        private void OrderedEntryPoint()
        {
            Int32 iContigiousNoStalls   =   0;

            try
            {
                foreach (CMsg<_TPayload> cMsg in m_OrderedQueue.GetConsumingEnumerable())
                {
                    bool bAcquired = false;

                    bAcquired = cMsg.WaitForUnOrderedProcessing(0);
                    if (false == bAcquired)        //  Immediate grab attempt
                    {
                        m_iTotalOrderStalls++;
                        iContigiousNoStalls = 0;
                        m_bOrderedIsCurrentlyStalled = true;
                        OrderedWorkerStalled(cMsg.Payload); //  Signal we are going to a wait state on the specific payload
                    }
                    else
                    {
                        if (0 == (++iContigiousNoStalls % (m_TotalMessages * 0.25)))
                        {
                            OrderedWorkerNoStall(); //  Signal that prediction could feed us more
                            m_bOrderedIsCurrentlyStalled = false;
                        }
                    }

                    if (false == bAcquired)
                    {
                        cMsg.WaitForUnOrderedProcessing(-1);
                    }

                    DoOrderedWork(cMsg.Payload);

                    cMsg.MakeReadyForFreeQueue();
                    m_FreeQueue.Add(cMsg);
                }
            }
            catch (ThreadAbortException)        //  Silent, main would have issued this
            {
            }
            catch (OperationCanceledException)
            {
            }
        }

        public bool FQueueWorkItem(_TPayload pPayload)
        {
            bool bRC = false;
            StartWorkersIfNecessary();

            if (true == m_bInitialized)
            {
                CMsg<_TPayload> cMsg = m_FreeQueue.Take();
                cMsg.Payload = pPayload;

                //  Place on both queues
                m_UnOrderedQueue.Add(cMsg);
                m_OrderedQueue.Add(cMsg);
                bRC = true;
            }

            return bRC;
        }

        #region CMsg_IMPL
        public sealed class CMsg<_TPayloadType> : IDisposable
        {
            public CMsg()
            {
                m_Payload = default(_TPayloadType);
                m_OrderedReadyLock = new AutoResetEvent(false);
                MakeReadyForFreeQueue();
            }

            public void Dispose()
            {
                if (null != m_OrderedReadyLock)
                {
                    m_OrderedReadyLock.Close();
                    m_OrderedReadyLock.Dispose();
                    m_OrderedReadyLock = null;
                }

                GC.SuppressFinalize(this);
            }

            public void MakeReadyForFreeQueue()
            {
                m_Payload = default(_TPayloadType);
            }

            private _TPayloadType m_Payload;
            public _TPayloadType Payload
            {
                get { return m_Payload; }
                set { m_Payload = value; }
            }

            private AutoResetEvent m_OrderedReadyLock;
            public void MarkReadyForOrderedProcessing()
            {
                m_OrderedReadyLock.Set();
            }

            public bool WaitForUnOrderedProcessing(Int32 iTimeout)
            {
                return m_OrderedReadyLock.WaitOne(iTimeout);
            }
        }
        #endregion

    }
    #endregion
}

Powershell Test Driver

$core =   [Reflection.Assembly]::LoadFrom('Microsoft.SqlServer.XE.Core.dll');
$xelinq = [Reflection.Assembly]::LoadFrom('Microsoft.SqlServer.XEvent.Linq.dll');

$MergeXeventAssembly = [Reflection.Assembly]::LoadFrom('.\bin\x64\Release\MergeXEvents.exe');
if($null -eq $MergeXeventAssembly)
{
    throw 'ERROR: Unable to load assembly';
}

$bDumpRaw = $true;

if($true -eq $bDumpRaw)
{
    [MergeXEvents.Exporter]::GetIList('C:\temp\NewTestTraces\BigXEL\Test_0_130494871596480000.xel', 'C:\temp\NewTestTraces\BigXEL');
}
else
{
        $iRow = 0;

        $events = [MergeXEvents.Exporter]::GetIList('C:\temp\NewTestTraces\BigXEL\eEst_0_130494871596480000.xel', 'C:\temp\NewTestTraces\BigXEL');
        foreach($event in $events)
        {
             $iRow.ToString() + ' Event: ' + $event.Event_name + '|Package: ' + $event.Event_package_name + '|Timestamp: ' + $event.Event_timestamp  | format-table -auto;
             $iRow++;
        }
}

Bob Dorr - Principal Software Engineer SQL Server

“Calling fsync does not ensure that the entry in the directory containing the file has also reached disk. For that an explicit fsync on a file descriptor for the directory is also needed.”

The SQL Server Host Extension provides file level integrity.  When a move (rename), create, delete (remove/unlink) occurs the Host Extension issues a fsync on the parent directory as required by Linux. 

Note: Activates such as changes in the file size do not require synchronization of the direct parent.

Bob Dorr - Principal Software Engineer SQL Server

An example of the error: The application does a buffered write and receives success. (This means data can be stored in file system cache, non-stable media.)  An fsync/fdatasync is used to make sure the data is stored in stable media.  The stable media write(s) occurs during the sync operation and can fail for any number of reasons (out of disk space, lost connectivity to SAN, etc.) reporting an EIO error.

The issue, as described in the link, is that the sync returns the error but may clear the state of the cached pages.  The next sync returns ESUCCESS, meaning the the write(s), which were in cache, do not flush to stable media but applications were told they did.

Assume the database application opens the backup file, allowing file system caching (~_O_DIRECT.) SQL Server does NOT do this, in fact I made sure SQL Server on Linux does not allow this to occur.

· Backup is started using buffered I/O
· Writes occur to file system cache as the backup is progressing
· An external call to sync occurs causing file system cache writes to take place
· The sync encounters a write error on a buffer belonging to the backup.  The sync was called by an application outside database application so the database application is unaware of the failure.
· At the end of backup the database application issues fdatasync and instead of returning EIO, ESUCCESS is returned.
· A successful fdatasync indicates a hardened to stable media backup and the database application then truncates the inactive part of the transaction log.

The database no longer has the proper log records to perform transaction operations or recovery and the backup file, once thought to be valid, is invalid.

This issue does NOT impact SQL Server database, log and backup files.  SQL Server opens these file types using O_DIRECT to bypass file system cache. Even when SQL Server on Linux is running in Forced Flush mode the files are opened with O_DIRECT, avoiding the issue. 

Bob Dorr - Principal Software Engineer SQL Server

Overview

SQL Server relies on Forced-Unit-Access (Fua) I/O subsystem capabilities to provide data durability, detailed in the following documents: SQL Server 2000 I/O Basic and SQL Server I/O Basics, Chapter 2

Durability: “In database systems, durability is the ACID property which guarantees transactions that have committed will survive permanently. For example, if a flight booking reports that a seat has successfully been booked, then the seat will remain booked even if the system crashes.” - https://en.wikipedia.org/wiki/Durability_(database_systems)

Durability is a cornerstone of any database system and starting with SQL Server 2017 on Linux Cumulative Update 6 (CU6), SQL Server on Linux enables “Forced Flush” behavior as described in this article, improving durability on non-Fua optimized systems.

“Be sure to deploy SQL Server 2017 CU6 or newer for best data durability and performance results.“

Microsoft SQL Server Database Engine Input/Output Requirements

The Back Story

My involvement began when looking at the Write-Ahead-Logging (WAL) and the write behaviors on a Linux SQL Server. The investigation revealed that some of the Linux file systems may not be Fua optimized. Instead of passing the Fua bit with the write request (like Windows) Linux generates a separate device level flush request. Device level flushing may have an impact on your I/O caching, read ahead or other behaviors of the storage system. The “forced flush” changes in SQL Server avoid flushes, when possible, in order to improve performance on non-optimized Fua file systems.

Performance points of reference: The following charts highlight the increase in performance.

4 Socket, TPCC 9,108 – Old kernel 36,618 – Patched kernel

2 Socket, TPCC 9,606 – Old kernel 18,840 – Patched kernel

During my investigation, I traced, tested, reviewed, and instrumented the Linux kernel to understand the SQL Server I/O durability needs and associated impacts. This effort resulted in:

· Changes to the SQL Server Engine
· Changes to the SQL Server Linux Host Extension
· Validation with the Windows Storage team
· Validation with Windows Hyper-V team
· Validation with Windows Azure Storage team
· Conversations with Red Hat, HP, SUSE and Intel
· Changes in the Linux file system

Before all was said and done, all of these people and more helped with various activities:

· Eugene, Mike, Slava, Scott, Peter, Steve and others doing code reviews on my SQL Server and Host Extension changes.
· Val and Dylan performing power outage durability testing and adding new tests.
· Mitch and Ben doing functionality testing with the matrix of trace flags, mssql.conf options, Red Hat, Centos, Ubuntu, SUSE, and Windows using xfs, xfs ^opt, ext4, refs and ntfs.
· Pat validating Docker functionality and updating SQLIOSim to allow forced flush patterns.
· Patrick and Purvi doing performance and regression analytics.
· Kapil building and validating behavior on the Linux kernel changes for xfs (Note: I have tagged it xfs ^opt (xfs optimizations) in this document for reference.)
· Amit, Lee, Vaqar and Mike handling release activities and documentation.
· Neal, Matt, and others from Windows Storage, Windows Azure Storage, Windows Hyper-V, … validating Windows behaviors.
· Red Hat, SUSE, HPE, Intel helping and confirming Linux behaviors and making file system code changes.
· Pradeep, Venu and Suresh handling support issues. 

Special thanks to Dave Chinner and the Red Hat team, Dan Williams, Hamesh Patel and their colleagues from Intel, Scott Norton and his colleagues, Josh Paulson and members of the Microsoft Linux teams, Jan Kara from SUSE and many others.

Terms

Before diving into details the terminology is important.

Term	Type	Description
Stable Media	Concept	Any storage device that can survive a power outage.  In the 90’s we talked about saving data on spinning media.  SSDs, persistent memory, battery-backed controllers broaden the range of devices surviving power outage from traditional, spinning media.
WAL	Concept	Write-Ahead Logging or journaling.  The log records, which can be used to recreate the data, are stored in stable media before the data buffers are written and data buffers are stored in stable media before the supporting log records can be deleted.
Fua	Hardware	Fua = Forced Unit Access.  Fua is the standard used to issue an I/O write command in stable data storage.  Fua has been a long-standing part of the SCSI specifications.  Caution: Non-SCSI storage implementations often silently ignore the request.
Device Flush	Hardware	A device level flush forces any dirty buffer located in volatile storage onto stable media.  Flushing a non-battery backed disk or cache device can be performance impacting and repeated device flush commands reduce the optimized capabilities, such as, writes near current head placement, wear level reduction, caching mechanisms, etc.
FILE_FLAG_NO_BUFFERING	Windows	FILE_FLAG_NO_BUFFERING is the Win32, CreateFile API flags and attributes setting to bypass file system cache.  This flag does NOT imply flush behaviors, only the avoidance of the system file cache.  FILE_FLAG_NO_BUFFERING maps to the O_DIRECT flag for the Linux open command.   Warning: The flag causes the file system cache of a guest VM to be bypassed but the VM or container may not pass the option to the host.
O_DIRECT	Linux	Linux open command flag used to bypass file system cache.   Warning: The flag causes the file system cache of a guest VM to be bypassed but the VM or container may not pass the option to the host.
FILE_FLAG_WRITE_THROUGH	Windows	FILE_FLAG_WRITE_THROUGH is the Win32, CreateFile API flags and attributes setting generating Forced Unit Access (Fua) writes.  A write honoring Fua is issued with the Fua flag and does not return from the write command until transferred to stable media.    FILE_FLAG_WRITE_THROUGH maps to the O_DSYNC flag option for the Linux open command.   The FILE_FLAG_WRITE_THROUGH flag does NOT imply FILE_FLAG_NO_BUFFERING behavior. To achieve no buffering and Fua, both the FILE_FLAG_NO_BUFFERING and FILE_FLAG_WRITE_THROUGH flags must be specified.
O_DSYNC | fdatasync | RWF_ODSYNC	Linux	The O_DSYNC | RWF_ODSYNC | fdatasync indicate to the Linux file systems that the ‘data integrity standard’ is to be upheld for I/O operations.  The data integrity standard makes sure the data and supporting meta data, needed to retrieve the data, are stored in stable media.   O_SYNC and fsync indicate to the Linux file system to uphold the ‘file integrity standard.’  O_SYNC is a superset of the O_DSYNC as the standard indicates that all file metadata (access timestamps for example) are stored in stable media.   RWF_ODSYNC is a newer variation of O_DSYNC.  The RWF_ODSYNC can be set as a flag for each I/O request.  Whereas, opening the file with O_DSYNC establishes flushing behavior for all writes to the file.  SQL Server does NOT make use of RWF_ODSYNC.
FILE_FLAG_OVERLAPPED	Windows	FILE_FLAG_OVERLAPPED is the Win32, CreateFile API flags and attributes setting enabling asynchronous API behavior.  Control from APIs such as WriteFile and ReadFile returns as soon as the I/O is queued.  The operating system signals completion when the I/O stack finishes the request.
AIO | io_submit | Kernel I/O (kio)	Linux	Linux provides different system calls (syscall)/commands for asynchronous behavior.   ·         write command – synchronous syscall, does not return until the write completes. ·         aio_write | io_submit – asynchronous syscall, returning control as soon as the I/O request is queued. SQL Server uses aio* for asynchronous I/O operations.
Synchronized	Concept	Not to be confused with the synchronous behavior of the API/ABI invocations.  In the context of I/O, synchronized indicates that the storage and in-memory copies of the data are the same.  O_DSYNC often uses this term to indicate that the in-memory buffer is in sync with the storage media.
Data Integrity Standard	Concept	This data integrity standard is a subset of the file integrity standard.  In summary, the data integrity standard is the ability of the file system to store and retrieve data over a power outage.  This includes the data itself as well as the tracking structures for the storage.   This diagram shows a high-level view of a File.  The file holds the metadata, made up of attributes (time of last access, permissions, …) as well as mappings to actual storage (sectors on a disk, pages on an SSD, …)   Flush to stable media occurs for the data and storage mappings to achieve the data integrity standard.   Refer to the ISO documentation on data integrity for complete details. CSA ISO/IEC 9945-1:2005 (R2009) 3.375 Synchronized I/O Data Integrity Completion
File Integrity Standard	Concept	This file integrity standard is a superset of the data integrity standard.  The file integrity standard is the ability for the file system to store and retrieve data and metadata written over a power outage including attributes such as the time of last access and permissions.   Refer to the ISO documentation on file integrity for complete details: CSA ISO/IEC 9945-1:2005 (R2009) 3.376 Synchronized I/O File Integrity Completion
O_NONBLOCK	Linux	Linux open command flag value to avoid blocking during an I/O call.  This is not a FILE_FLAG_OVERLAPPED implementation for the Linux ABIs.  Instead, the flag is used to decide if a wait will occur while attempting to issue the I/O request.  If the thread would become blocked the attempt to issue the request is aborted and control is returned to the caller.   Overlapped will issue the I/O and return STATUS_PENDING, whereas the O_NONBLOCK returns EAGAIN and the caller must retry the entire I/O operation.
O_NOATIME	Linux	Linux open command flag to avoid updating the time of last access which can reduce metadata writes.  The O_NOATIME flag does NOT disable MTIME or CTIME modifications.
REQ_FUA	Linux	Block device write request using the Fua capabilities when performing a write.  The Fua bit is sent with the write signaling stable media storage is required.
REQ_PREFLUSH REQ_POSTFLUSH	Linux	A block device write request (0 bytes – write[10, 12 or 16 bytes] SCSI command) requesting a device flush.
DpoFua	Linux	DPO = Disable Page Out (for writes instructs I/O subsystem to avoid adding write data to the cache, for reads prevents replacing existing entries in cache.)   Fua and Dpo are separate bit values in the SCSI SCB.  SQL Server only requires Fua and not DPO capabilities.
Forced Flush Behavior	SQL Server	When enabled the Linux SQL Server opens the file with FILE_FLAG_NO_BUFFERING and without FILE_FLAG_WRITE_THROUGH (no FUA.)  SQL Server log and checkpoint made durable with calls to FlushFileBuffers.

When SQL Server performs a write, using a file opened with FILE_FLAG_WRITE_THROUGH(O_DSYNC), Linux performs the multi-step operation as a data write followed by the complete device flush request instead of adding the Fua bit to the data write request.

1. Issue write to block device for the data
2. Receive the data write complete event in the block device’s write completion routine
3. If O_DSYNC requested, issue block device flush
4. Wait for block device flush request completion
5. Return control to write caller

Every write, requiring O_DSYNC behavior, also invokes a block device flush on Linux.  The repeated use of a write request followed by a flush request may be detrimental to performance and will increase traffic on the I/O bus.

Note: The 4.18 Linux Kernel updates allow the XFS file system and a device supporting Fua writes to avoid the additional block device flush request.

Prior to the Linux Kernel 4.18 updates, Linux could use Fua but only for the file system journaling writes and not data writes.  Linux queries the device for the DpoFua support capabilities and when the device reports Fua support (DpoFua = 1) the journaling is optimized to use Fua writes.  The data storage always uses the multi-step flush process. 

Windows does NOT query the device for support, instead any file opened with FILE_FLAG_WRITE_THROUGH sends the Fua option with the write request where the target device is expected to honor the Fua option.  In some instances, the target silently ignores the request. The Windows implementation places the responsibility of making sure the I/O subsystem is Fua compliant on the database and system administrator.

Fua (Forced Unit Access)

Microsoft SQL Server requires Fua compliant storage (stable media) ensuring the data is stored in stable media before a write request returns to SQL Server.  For example, if you run SQL Server on an IDE or SATA drive, you may need to disable the physical disk cache to accommodate WAL (durability), requirements. The SQL Server 2017 CU6, forced flush feature, provides durability assistance on Linux systems without Fua.

Note:  Disabling the write cache using the Windows, properties dialog may not suffice.  Some drives are not compliant with the dialog options and require special utilities to configure the drive cache.  Linux may need similar utilities to control various hardware cache installations.

You can explore your system using the df and lsblk commands.  When Linux mounts a block device, Linux queries the attributes and capabilities of the block device.  One such capability is the ability to support DpoFua for write requests.  When DpoFua=1 is reported, the device is reporting support for Fua.

Note: These commands may require root permissions to execute.

df -T

Filesystem          Type     1K-blocks     Used Available Use% Mounted on

devtmpfs            devtmpfs   4062840        0   4062840   0% /dev

tmpfs               tmpfs      4078624        0   4078624   0% /dev/shm

tmpfs               tmpfs      4078624     8880   4069744   1% /run

tmpfs               tmpfs      4078624        0   4078624   0% /sys/fs/cgroup

/dev/mapper/cl-root xfs       52403200 13010400  39392800  25% /

/dev/sda1           xfs        1038336   431168    607168  42% /boot

/dev/mapper/cl-home xfs      114361796 22488816  91872980  20% /home

tmpfs               tmpfs       815728        0    815728   0% /run/user/1000

lsblk

NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT

fd0           2:0    1     4K  0 disk

sda           8:0    0   168G  0 disk

├─sda1        8:1    0     1G  0 part /boot

└─sda2        8:2    0   167G  0 part

  ├─cl-root 253:0    0    50G  0 lvm  /

  ├─cl-swap 253:1    0   7.9G  0 lvm  [SWAP]

  └─cl-home 253:2    0 109.1G  0 lvm  /home

sdb           8:16   0     1G  0 disk

sr0          11:0    1  1024M  0 rom

The /dev/sda1 entry reports an xfs file system and the device id of 8:1 (major:minor.)  The block device is sda (8:0.)  The entries for sda1, sda2 are partitions on the same block device.

You may also use lshw to explore disk information such as sector size and vendor which are helpful in determining additional device details.

lshw -class disk

       description: SCSI Disk

       product: Virtual Disk

       vendor: Msft

       physical id: 0.0.0

       bus info: scsi@0:0.0.0

       logical name: /dev/sda

       version: 1.0

       size: 168GiB (180GB)

       capabilities: partitioned partitioned:dos

       configuration: ansiversion=5 logicalsectorsize=512 sectorsize=4096 signature=000a0600

Use dmsg or sg_modes to reveal the DpoFua state.

dmesg | grep sda

[    1.266561] sd 0:0:0:0: [sda] 352321536 512-byte logical blocks: (180 GB/168 GiB)

[    1.266562] sd 0:0:0:0: [sda] 4096-byte physical blocks

[    1.266892] sd 0:0:0:0: [sda] Write Protect is off

[    1.266894] sd 0:0:0:0: [sda] Mode Sense: 0f 00 10 00

[    1.266949] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, supports DPO and Fua

[    1.271227]  sda: sda1 sda2

[    1.274579] sd 0:0:0:0: [sda] Attached SCSI disk

[    4.637845] XFS (sda1): Mounting V5 Filesystem

[    4.661270] XFS (sda1): Ending clean mount

Note:  /dev/sda is the block device and /dev/sda1, /dev/sda2 are partitions of the block device.

sg_modes /dev/sda1

    Msft      Virtual Disk      1.0    peripheral_type: disk [0x0]

Mode parameter header from MODE SENSE(10):

  Mode data length=20, medium type=0x00, WP=0, DpoFua=1, longlba=0

  Block descriptor length=0

>> Caching, page_control: current

00     08 0a 04 00 00 00 00 00  00 00 00 00

If your system supports Fua and you have the Linux Kernel 4.18 or newer updates, you can enable SQL Server trace flag -T3979 and use /opt/mssql/bin/mssql-conf set control.alternatewritethrough 0.  SQL Server will use Fua write behavior patterns instead of Forced Flush.  File systems supporting optimized Fua align SQL Server on Linux with SQL Server on Windows behavior and performance.

Note: If DpoFua=0 is reported the Linux file systems revert to the write, flush behavior to support O_DSYNC.  If you are unsure about the Fua support use forced flush behavior.

 

Note: DpoFua optimizations may be ignored during the first data write to an offset in the file, as documented here.

 

Note: Some systems embellish the Fua support capabilities.  You should always validate the Fua capabilities with your device and hardware vendor.

Beware The Cache

The following is a high-level flowchart of the settings and outcomes.

Safety and Performance

To help make sense of all of this I have broken down the various trace flag and configuration options by data safety and performance. 

Safe Installation (Best performance):
- SQL Server 2017 CU6 or newer
- Linux kernel 4.18 or newer with xfs storage
- T3979
- DpuFua=1
- control.writethrough = 1
- control.alternatewritethrough = 0

Safe Installation (Best Grouping Performance – Forced Flush from SQL Server and HE):
- SQL Server 2017 CU6 or newer
- T3982
- DpuFua = 0
- control.writethrough = 1
- control.alternatewritethrough =1

Safe Installation (Midrange Grouping Performance - Grouped device level flushes from HE):
- SQL Server 2017 CU6 or newer
- T3979
- DpuFua = 0
- control.writethrough = 1
- control.alternatewritethrough =1

Safe Installation (Slowest Performance - Linux kernel write, flush O_DSYNC behavior):
- SQL Server 2017 CU6 or newer
- T3979
- DpuFua = 0
- control.writethrough = 1
- control.alternatewritethrough =0

Alternative Installation (O_DIRECT Safe Reliance on I/O Subsystem):
- SQL Server 2017 CU6 or newer
- T3979
- DpuFua = 0
- control.writethrough = 0
- control.alternatewritethrough =0

      * Note: This achieves the prior configuration of SQL Server 2017 RTM to CU5 on Linux using only O_DIRECT are relying on O_DIRECT safe installations.

O_DIRECT safe refers to is the ability to perform writes with O_DIRECT and without O_DSYNC and maintain durability.  If the I/O subsystem avoids the proper buffering mechanism until it reaches stable media (battery-backed cache for example) then the Fua option may not be required.  However, to be O_DIRECT safe you must ensure the kernel, drivers, and hardware are able to place a write into non-volatile storage before indicating the write is complete.

I am unaware of tracing mechanisms to determine if the system is O_DIRECT safe.  Microsoft has done extensive testing of durability on systems believed to be O_DIRECT safe and SQL Server for Linux RTM thru CU5 was configured for O_DIRECT safe write activities.  However, Microsoft does not recommend reliance on O_DIRECT safe configurations and instead provides SQL Server 2017 CU6 and newer builds with explicit durability capabilities (forced flush or optimized Fua writes.)

If you choose to configure your system for O_DIRECT safe writes be sure to employ an appropriate backup or high availability strategy.

Caution

Unless you are using an updated file system, it is unsafe to assume the REQ_FUA behavior.  SQL Server automatically enables -T3982 (forced flush behavior) on SQL Server for Linux.  If you are able to, verify that a write on your system meets the following criteria, you may be able to achieve data integrity compliance without Fua capabilities as long as you are able to make sure the O_DIRECT path supports write completion when the data has been stored on stable media. – “O_DIRECT Safe.”

 

1.     I/O subsystem and drivers do not cache FILE_FLAG_NO_BUFFERING (O_DIRECT) in volatile cache. All cache involvement must be power outage hardened.

2.     All files in the database are on these stable media, compliant I/O paths (data, log, checkpoint files, standby recovery files, backups, …)

3.     All SQL Server support files are on the stable media (cluster files, registry files, …)

 

If you are unsure maintain the forced flush behavior configuration.

Virtual Environments (VMs and Containers)

Virtual environments may not provide the write behavior required to maintain durability.  Microsoft has validated Hyper-V, Windows Azure Virtual Machines, Windows Storage and various Docker implementations pass the flush requirements to the host properly.  When performing a Fua write the Fua requirement must be passed to the host and the host must carry the write in stable media.  If the virtualization or host caching does not support the Fua requirement you should use forced flush behaviors to maintain durability.

How did I get DpoFua=1 enabled for a Hyper-V installation of Linux?

On virtual systems, additional effort may be required to confirm the virtualization technology passes the Fua intent to the host and the host honors the request.  You may use a directly mounted device (direct pass-through drive for Hyper-V or iSCSI supporting Fua capabilities) and/or confirm the VHD | VHDx storage path supports Fua. 

WARNING: Use caution as the Hyper-V configuration does not query the device for Fua support and trusts the setting provided.

Using the Hyper-V, PowerShell cmdlets, from Windows 2016, you can configure Fua reporting for the guest virtual machine.

$a = Get-VMHardDiskDrive -VMName "xxxxxxxxxxx"
foreach ($b in $a) { $b; $b.WriteHardeningMethod }

Set-VMHardDiskDrive -VMName "xxxxxxxxxxx"  -OverrideCacheAttributes WriteCacheAndFuaEnabled

     Note: You may need to add a controller type such as -ControllerType SCSI

Linux Kernel 4.18 Updates

I have worked at Microsoft since 1994 and this issue truly points to the spirit of ‘Microsoft Loves Linux.’  I was strongly supported, funded and encouraged to track down the issue and make it better for customers.  The Linux committer and vendor engagements where insightful and helpful and, in the end, I am able to proudly say that Microsoft helped to improve Linux.

Several code changes were made to the imap, block device, and XFS components to accommodate optimized Fua writes.  The changes in the imap and block device areas can be leveraged by other file systems but as Dec 2018 (this post) only XFS has enabled optimized Fua data write capabilities.

Key Commit References in 4.18 and newer kernel releases

• ed5c3e6 xfs: move generic_write_sync calls inwards
• 4f8ff44 iomap: iomap_dio_rw() handles all sync writes
• 0ce9144 block: add blk_queue_fua() helper function
• 3460cac iomap: Use FUA for pure data O_DSYNC DIO writes

Support
Microsoft has tested and confirmed the Fua changes are included in Linux kernel build 4.18 and newer kernels while Red Hat has tested and confirmed the changes are included in RHEL 8.0.

Hint: Use the uname -sr command to discover the Linux Kernel version.

Tracing SQL Server Behavior (XEvent)

The following XEvents can be used to monitor the flush behavior employed by SQL Server.

XEvents (Debug channel)

·         flush*

·         make_durable*

·         checkpoint*

-T3979 (Force Flush Disabled) Checkpoint

-T3982 (Force Flush Enabled) Checkpoint

Notice the make_writes_durable and flush_file_buffers events for the database log and data files used for ACI(D)urability.

Testing and Futures (Warning: Plans May Change)

When it comes to database consistency there is no discussion on doing the right thing.  When I told Slava I was going to put in the forced flush behavior and it would have a performance impact, he simply said do it!  Not 2 seconds passed between us making the decision. As a team, we would figure out ways to improve the performance, but the safety of the data took priority.

When SQL Server 7.0 shipped Microsoft SQL Server made a distinct change to transaction log writes, aligning on sector boundaries and removing the 2K page rewrites of the 6.x and Sybase code to improve ACID properties of the SQL Server.  This generated lots of calls to Microsoft Support because it changed the I/O performance characteristics in ways customers didn’t expect and lead to the first code I added to SQL70IOStress, which I would later rewrite into what you know today as SQLIOSim. 

I mention this because folks like Keith Elmore, Bob Ward and I were running tests in the lab.  In fact, I was pulling the power between the power supply and the hard drive, waiting a few minutes, plugging the power back in and validating the durability and consistency provided by the I/O subsystem.  This lasted a couple of days until I electric-arc’ed the drive and burnt out the power supply. The lab manager would no longer allow me to manually pull the power.  It just so happened that Ed Harper was on our team and he built me a wire wrapped board.  It had a few D-Batteries and a switch on it with a plug on one end for the power supply and a plug on the other for the drive.  I used the generic driver sample and could signal the switch to open and close which I could then signal from SQL70IOStress and perform 1000s of ACID related tests overnight.  I won’t even tell you how many disk drives I have worn out doing I/O testing.

Creating SQL 2017 CU6 updates involved revisiting similar tests.  The SQL Server test infrastructure can inject fail points.  Dylan added additional fail points to checkpoint, lazy writer, and log writer.  Ben, Mitch, and Kapil performed physical pull the plug and kill the VM testing.  Pat added forced flush behavior to SQLIOSim, which we have been running in a Linux package, in our daily lab runs since before SQL Server 2017 was released.  We also added additional coverage to containers, high-availability, replication, backup, restore and recovery test suites.

Met Tejas, a long time Microsoft SQL Server Support Escalation Engineer, is a Program Manager for SQL Server on Linux.  He is working on the Linux experience, testing and capabilities surrounding SQL Server WAL and ACID capabilities.  Some of the areas he is currently looking into are as follows (as always these a goals we would like to achieve:)

As of this post, XFS on Linux 4.18 or newer is Fua optimized and we have also reached out to additional, Linux file system committers.  For example, the EXT4 committers are aware of the changes and evaluating the changes.

Odds and Ends

Updates and changes like this tend to expose some odds and ends.

Checkpoint Record Flush

SQL Server creates a log record (LOP_BUF_WRITE) to assist recovery. The record has an array of page ids written to disk. The checkpoint flush record has a limited size and is written whenever it becomes full.  The flushing of the checkpoint record requires FlushFileBuffers when forced flush behavior is enabled to maintain durability.

All Data Files For Same Database Flushed

Forced flush loops over all database files in the target database and issues FlushFileBuffers to complete a checkpoint. The data files may be on the same block device and the secondary flushes may be redundant but are inexpensive. Future SQL Server updates may optimize this behavior.

TEMPDB Is Faster

You may find that TEMPDB is faster using -T3982. TEMPDB is always rebuilt upon SQL Server restart and forced flush removes the FILE_FLAG_WRITE_THOUGH requirement of TEMPDB.  Since TEMPDB does not need to issue checkpoints, FlushFileBuffers may be skipped.  In fact, our performance team is evaluating a SQL Server 2019 change that never opens the TEMPDB file with the FILE_FLAG_WRITE_THROUGH.

RWF_ODSYNC

Another choice, provided in the Linux Fall 2017 Update, allows individual I/O requests (io_submit) to set the RWF_ODSYNC flag, forcing O_DSYNC for individual write requests. SQL Server does not use RWF_ODSYNC.

Simple Test and Results