This is a site, in which all of us, global user, can find the exact translation of a localized string in Microsoft Products, to an error we get. Afterwards – just bing it and you are on the right path to resolve it.

Enjoy, Rona

To start with, I learned that MediaWiki is one of Wikimedia projects (which really confused me and I kept saying “Wikime … no, Mediawi… no, Wikime… no, WikiWhatever!” whenever I wanted to reference the content source), it has an API that and even a .net wrapper (named DotNetWikiBot Framework). Of-course you can also use the API directly or even export the whole content of the MediaWiki as XML, and index it or even import it to SharePoint …

Anyways, you may ask, why would you build a search connector to index HTML content source instead of using the out-of-the-box (OOB) capabilities? Well, I’ll tell you, this specific MediaWiki had some enhancements that the OOB HTML PH is not familiar with, and anyway, what can be more fun than debugging a Search Connector? – you’ll never know until you try it.

So, back to the starting point: we learnt the content source and its interaction capabilities and decided to build 2 connectors: one that will index the exported XML and another one to index based on the API. The first one will be used for FULL INDEX and the later will be used for INCREMENTAL. Why use both methods? Well, this is a long story and maybe I’ll get into the alternatives and their pros and cons later on. Right now we’ll stick to the technical aspects.

Being familiar with Protocol Handlers and IFilters development and very non-familiar with the MediaWiki – I decided to start with a couple of POCs that apart from proving the possibilities will also produce stand alone versions of the index connectors. Those standalone versions will later be used for troubleshooting, if needed, and any other manual operation required during the component lifetime. So that’s what we did.

During the POC we encountered our first challenge: The API responded well when used as a query string in the browser, but failed miserably when invoked from code (whether directly or via the above mentioned Bot or any other way we attempted). The reason was the language of the MediaWiki. It is set in LocalSettings.php and we failed since it was set to “he” (that’s Hebrew). As soon as we set it to English (“en”) – the API responded well. But we wanted the users to use a Hebrew formatted (right to left) MediaWiki, so we set a MIRROR of the Hebrew MediaWili, which was in English, and used the MIRROR for the API. The MIRROR does not duplicate the content, it only enable setting different settings. We overcame this challenge.

Next we reviewed the exported XML and learnt that MediaWiki is quite forgiving and allows some not that valid XML, a couple of missing closing tags and so on. That would be an issue when trying to parse this xml, so we decided to use the water hose reader and mainly pray regularly. We apparently prayed well.

However, during the POC we learnt that reading every item using the above method will take 138 days for a single FULL INDEX. We considered uploading it to SQL or splitting the file but eventually decided against all those and managed to reduce the index time to … 4 hours with incremental index of a couple of minutes. How? Well, stay tuned to the coming POSTs and I’ll tell you, along with architecture, debugging and troubleshooting tips, known issues and references to official and other great resources in the internet.

Hope you enjoy and find it useful, Rona

• CPsplitter splits the exported content publishing files into sub folders and updates their manifests.
• CPsplitter can be downloaded from a link at the bottom of this post.

Enhanced overview

When exporting content from SharePoint using the Content Migration API - the exported content can be packaged and compressed in CAB files.

To faster Content Migration EXPORT - you can choose to avoid content compression.

This choice has another side effect: the content is exported directly to your chosen target location, and you avoid the use of the system chosen intermediate storage folder.

 So you happily run your faster process and export a couple of thousands of files to your desired folder, until you suddenly notice that the operation is gracefully  s l o w i n g   d o w n  as more and more files are added to this folder . . .

 This is where CPsplitter comes into the picture: it waits for the manifest to be released from the EXPORT process, and moves the manifest, along with all its files, to a sub folder, letting the EXPORT process keep running at full speed ahead.

"The manifest?" you may ask, "What is that?"

·         Well, every exported piece of content is listed in an XML file that is named 'manifest.xml', which is later used in the IMPORT operation.

·         There can be multiple manifest files for a single EXPORT, and you can roughly control the number of files listed in each manifest by setting the 'File Max Size' in the EXPORT application.

"The EXPORT application?" you ask ...

·         Well, there are multiple tools and means to export content from SharePoint 2007, each is best suited for certain scenarios. Some are part of the product, like STSADM and the Central Administration Content Deployment process. Others are highly customized applications that were developed by our partners, and there are even a couple of freely downloadable samples (like the SharePoint Content Deployment Wizard from CodePlex, the Content Publishing tool that accompanies my Content Migration API in Action" post and others) that you can use for some scenarios.

SO ... assuming you would like to give it a try - here is a simple sample walkthrough:

1.       Prepare

1.1.    Make sure you can install and run the applications on the SharePoint server (in a TEST environment!)

1.2.    Select the SharePoint 2007 site you want to export

1.3.    Download the Content Publishing application and unzip

1.4.    Download the CPsplitter application (see link by the end of the article) and unzip

2.       Run the Content Publishing application and in the EXPORT pane ...

2.1.    Set 'Export file location' to be c:\kuku

2.2.    Unselect 'File Compression'

2.3.    Type you site URL in the 'Site URL'

2.4.    Set 'File Max Size' to be 1 (meaning: 1 MB)

2.5.    Click EXPORT to get it going and acknowledge you wish to run it J

3.       While the EXPORT is running ...

3.1.    Look at c:\kuku and see the files getting in

3.2.    Run CPsplitter

3.3.    Click 'Demo' to set the properties

3.4.    Click 'Split Files into Folders'

3.5.    Monitor CPsplitter logging and c:\kuku

So far so good for export …

But now you want to IMPORT, and the manifests are not in the root folder, and the files are scattered all over …

No worries – you’ve got CPsplitter!

Run CPsplitter, make sure MANIFEST.XSLT is in place and the TARGET FOLDER is correct and click UPDATE MANIFESTS.

 “What will it do?” you ask …

·         Here is what it will (hopefully) do: every manifest, from every sub folder of the TARGET FOLDER, will be updated to refer to the files in the sub folder, and will be copied to the root TARGET FOLDER, which is used for the IMPORT of Content Publishing.  

CPsplitter.zip

If you do - you may want to use the CMP Explorer, which is attached by the end of this article.

Alternatively, you could use any of your favorite XML editors :)

Background

Content migration EXPORT produces either a single CMP file or a folder with multiple XML files.

To specify the desired output type - use the FILECOMPRESSION EXPORT settings.

To extract files from a CMP file - copy it and rename its extension to CAB.

 

How To?

Once you have the files in a folder, you are ready to use the CMP Explorer.

Simply type the folder name and click to inspect some of the content of the XML files.

More information

Content migration output XML files are described in details in the WSS SDK, and their schemas are at C:\Program Files\Common Files\Microsoft Shared\web server extensions\12\TEMPLATE\XML\Deployment*.xsd .

Here are excerpts from the WSS SDK:

• Manifest.xml is the primary file used by content migration.
• Requirements.xml contains information that is used as a preliminary check before any import takes place.
• ExportSettings.xml is used to verify the logic of the export and ensure that what is expected in the export is included in the package.
• RootObjectMap.xml defines the top-level object to import.
• SystemData.xml contains all the default objects that are installed on a server that is extended with Windows SharePoint Services. This information is used primarily when the export/import operation is retaining GUIDs between the source and destination locations. This file contains the schema version. If this does not match the version at the destination, the import fails.
• UserGroup.xml contains all the user and group information from the source Web site.
• LookupListMap.xml maintains a simple lookup list that records SharePoint list item (list item to list item) references.
• ViewFormsList.xml maintains a list of Web Parts and tracks whether each is a view or form.

Disclaimer
This posting is provided "AS IS" with no warranties. Use of included utilities are subject to the terms specified here.

Have a great CMPing,
Ronalus

CMPexplorer.zip

No need to deploy on the server. Using SharePoint's web services it can run remotely.

Lists.zip

Above is from the article How to: Customize Deployment for Disconnected Scenarios , which also provides a complete code sample for using the content migration API.

Starting from this code sample, I walked through the API and got to a little application, which can be downloaded from the link below. Do note that as the application uses the SharePoint object model, it should run on the SharePoint 2007 server. Make sure to thoroughly test in test environment, as it is still work in progress...

When running the application, take a look at the session log on the right side of the screen. It lists the properties used in the deployment and may help you develop your own tailor made tools.

Tip: for incremental deployment you need 4 settings:

1. ExportChangeToken field should contain the token from a previous export
2. ExportMethod should be ExportChanges
3. ExportObjectType should be either site or web
4. Import RetainChangeIdentity should be selected

To make things easy, the little application (AKA CP) places the ExportChangeToken for you in the correct field upon an Export completion. You can also save the token, along with a descriptive statement, in an XML file, for future use.

[Update: Oct. 2, 2007] As CP takes its first steps, new ideas emerge. Two of them were already implemented:

1. A batch version (for you, Sharon)
2. An MSBuild task (to comply with your great development, Neeraj)

Both CPbatch and CPtask are also in the below downloadable ZIP.

To run them - use a settings XML file, which can be generated by "Save Preferences" in CP (the interactive version).

For other samples see Migrate a Web site from one location to another  and the WSS SDK.

Cool usage of STSADM for content migration can be found in Jackie's blog: SharePoint Content Migration Object Model and Content Migration Packages .

Disclaimer
This posting is provided "AS IS" with no warranties. Use of included utilities are subject to the terms specified here.

Enjoy content migration,

Ronalus

CP.zip

Although highly improved in 2007, it maintains backward compatible with 2003 syntax.

As STSADM is a command line utility, and the command-line is somewhat limited compared to Windows GUI, especially when it comes to non-English usage, there is a room for GUIs like stsadmWin (STSADM for Windows).

How stsadmWin works?

stsadmWin reads a list of commands from an XML file (namely: operations.xml), and displays a combo with the operations, denoting mandatory and optional parameters, and linking to the utility’s help.

The above download includes an XML file with SharePoint 2003 operations.

An XML file for MOSS 2007 RTM is attached by the end of this post (in stsadmWin.zip ).

Help information

stsadmWin displays help for the selected operation in the 2003 version.

This feature is currently not available in the 2007 version.

To get more information on STSADM 2007 see:

 * Stsadm command-line tool (Office SharePoint Server)

 * Stsadm command-line tool (Windows SharePoint Services)

 * Complete reference of all STSADM operations (with parameters) in MOSS 2007

 * Run STSADM /?   -or-   STSADM –help <OPERATION>

To get a list of stsadm commands grouped by function - see Ian Morrish's WSS Demo .
The list is updated to SP1 and even indicates if a command is unique  to stsadm or if there is a Central Admin/SSP equivalent.

stsadmWin extensibility

Unlike the 2003 version, STSADM in 2007 is extensible. Meaning: you may get newer commands (operations) available as you add components to your environment.

To generate an XML file that suites your environment – you can use the GetOperations utility (also attached by the end of this post - in stsadmWin.zip).

Note: it is advisable to backup existing XML before generating an updated version …

A note on the generated XML:

 * A couple of operations may be generated with wrong syntax.

 * The ones I currently know about are: osearch and profilechangelog.

 * An updated syntax will be available in operationsUpdate.xml, which is referenced below.

 * You will only need this update when you re-generate operations.xml. In this case - you have to update operations.xml with the content of operationsUpdate.xml .

 * The operations.xml files that are referenced below - already include the correct syntax.