How to: Programmatically Export the Crawl History to a CSV File

Article
10/16/2013

Microsoft Search Server 2008 tracks crawl statistics, including the following:

Crawl type (full or incremental)
Crawl start time and end time
Content source associated with the crawl
Number of items crawled successfully, and the number of warnings and errors that occurred

You can use the Search Administration object model to retrieve this information by using the Microsoft.Office.Server.Search.Administration.CrawlHistory class. This class contains the GetCrawlHistory function, which returns the crawl statistics in a DataTable object.

The following procedure shows how to write out the contents of the crawl history table to a comma separated values (.csv) file. You can then open that file in an application such as Microsoft Office Excel for further analysis.

To export the crawl history to a .csv file from a console application

In Microsoft Visual Studio, on the File menu, point to New, and then click Project.
In Project types, under C#, click Windows.
Under Templates, click Console Application. In the Name field, type CrawlHistorySample, and then click OK.
On the Project menu, click Add Reference.
On the .NET tab, select the following references, and then click OK:
- Microsoft Search component (Microsoft.Office.Server.Search.dll)
- Microsoft Office Server component (Microsoft.Office.Server.dll)
- Windows SharePoint Services (Microsoft.SharePoint.dll)
In your console application's class file, add the following using statements near the top of the code with the other namespace directives.
```
using System.IO;
using System.Data;
using Microsoft.SharePoint;
using Microsoft.Office.Server.Search.Administration;
```

Create a function to write out usage information to the console window.

static void Usage()
{
   Console.WriteLine("Export Crawl History to CSV File Sample");
   Console.WriteLine("Usage: CrawlHistorySample.exe SiteURL OutputPath OutputName");
   Console.WriteLine(@"Example: CrawlHistorySample.exe http://MySearchServer c:\logfiles\ CrawlHistoryLog1");
}

In the Main function of the console application, add the following code to open a try block.
```
try
{
```
Add code to check the number of items in the args[] parameter; this number must be equal to 3. If it does not, then call the Usage function defined in step 7.
```
if (args.Length != 3)
{
   Usage();
   return;
}
```
Retrieve the values specified in the args[] parameter, to be used to specify the search center site URL, the output path, and name for the .csv file.
```
string strURL = args[0];
string csvPath = args[1] + "\\" + args[2] + ".csv";
```

Add the following code to retrieve the SearchContext.

SearchContext context;
using (SPSite site = new SPSite(strURL))
{
   context = SearchContext.GetContext(site);
}

Retrieve the crawl history into a System.Data.DataTable object by using the following code.

CrawlHistory history = new CrawlHistory(context);
DataTable table = new DataTable();
table = history.GetCrawlHistory();

Add the following code to initialize the System.IO.StreamWriter object.

using (StreamWriter writer = new StreamWriter(csvPath)) 
{
/*
Code to iterate through DataTable rows, and write
out the values to the .csv file
See steps 14 through 17.
*/

Retrieve the number of columns, as follows.
```
int colCount = table.Columns.Count;
```

Write out the column names to the file stream, with a comma following each name, except for the last column name, and then write a new line to the file stream, as follows.

for (int i = 0; i < colCount; i++)
{
   writer.Write(table.Columns[i].ToString());
   if (i < colCount - 1)
   {
      writer.Write(",");
   }
}
writer.Write(writer.NewLine);

Loop through the collection of rows, and write out each column value to the file stream, with a comma following each value. Write out a new line to the file stream at the end of the row before moving to the next row.

foreach (DataRow row in table.Rows)
{
   for (int i = 0; i < colCount; i++)
   {
      writer.Write(row[i].ToString());
      if (i < colCount - 1)
      {
         writer.Write(",");
      }
   }
   writer.Write(writer.NewLine);
}

Flush the contents of the file stream, close it, and then close the using statement block and the try block, as follows.
```
      writer.Flush();
      writer.Close();
   }
}
```

Add the following code for the catch block.

catch (Exception ex)
{
   Console.WriteLine(ex.ToString());
   Usage();
}

Example

Following is the complete code for the CrawlHistorySample console application class.

using System;
using System.Collections.Generic;
using System.Text;
using System.IO;
using System.Data;
using Microsoft.Office.Server.Search.Administration;
using Microsoft.SharePoint;

namespace CrawlHistorySample
{
    class Program
    {
        static void Main(string[] args)
        {
            try
            {
                if (args.Length != 3)
                {
                    Usage();
                    return;
                }
                string strURL = args[0];
                string csvPath = args[1] + "\\" + args[2] + ".csv";

                SearchContext context;
                using (SPSite site = new SPSite(strURL))
                {
                    context = SearchContext.GetContext(site);
                }
                CrawlHistory history = new CrawlHistory(context);
                DataTable table = new DataTable();
                table = history.GetCrawlHistory();

                using (StreamWriter writer = new StreamWriter(csvPath, false))
                {
                    int colCount = table.Columns.Count;
                    for (int i = 0; i < colCount; i++)
                    {
                        writer.Write(table.Columns[i].ToString());
                        if (i < colCount - 1)
                        {
                            writer.Write(",");
                        }
                    }
                    writer.Write(writer.NewLine);
                    foreach (DataRow row in table.Rows)
                    {
                        for (int i = 0; i < colCount; i++)
                        {
                            writer.Write(row[i].ToString());
                            if (i < colCount - 1)
                            {
                                writer.Write(",");
                            }
                        }
                        writer.Write(writer.NewLine);
                    }
                    writer.Flush();
                    writer.Close();
                }
            }
            catch (Exception ex)
            {
                Console.WriteLine(ex.ToString());
                Usage();
            }
        }

        static void Usage()
        {
            Console.WriteLine("Export Crawl History to CSV File Sample");
            Console.WriteLine("Usage: CrawlHistorySample.exe SiteURL OutputPath OutputName");
            Console.WriteLine(@"Example: CrawlHistorySample.exe http://MySearchServer c:\logfiles\ CrawlHistoryLog1");
        }
    }
}

Share via

How to: Programmatically Export the Crawl History to a CSV File

To export the crawl history to a .csv file from a console application

Example

See Also

Other Resources

Additional resources