CTS - Content Transformation Services 

IMS - Interction Management Services

How to build visual interfaces and shape how people interact with information with FAST Search Search 2010 for Internet Sites

Thread operator is the vital component for running different set of flows in parallel. So CTS flow with parallel execution of flows can be built as shown above, please note that this is just one of the samples. So what you have in the flow is

1. Common starting point, in the above flow it is just a database reader, but in other flows it can be any other starting point like file reader, range generator or any other type of reader available out of the box for CTS components. After common starting point then you can branch the flow execution to as many number of flows that you want to execute in parallel.
2. Next in each branch if you want to start parallel flow execution with delay of sometime then you can use the timer operator, so that main thread of execution does not spins another thread for flow execution. For Each timer operator in each of the branch you can increase the delay time with equal intervals like 5 sec, 10 sec, 15 sec, 15 sec…….
3. In each branch then you can put the thread operator. The Thread operator inserts an extra thread in the evaluation at the point it is located. Use of the Thread operator leads to increased throughput of the flow on multi-core servers. The operator can be used to increase the performance for a chain of CPU-intensive operators and for flows with multiple branches.
4. Once the additional thread is spin off by the thread operator, then all the further execution for the CTS flows is responsible by the spin off thread, in this case we are using flow runner operator. So in this case additional spin off thread is responsible for the flow execution configured by the flow runner operator. Once the flow execution is completed by the flow runner operator which is been executed in thread context of additional thread which is spin off, flow execution returns to the main execution thread.
5. Please not that flow executing in parallel will execute on different content engine node as available in round robin fashion.

As stated above, suppose you have sequence of flows, you complete the flow execution for Flow Runner 1 and then you cannot execute next flow executed by flow runner 2 unless you get a green signal from the database table or field. For this requirement for polling what can be done is, push another flow runner in between the flow runner 1 and 2. This intermediate flow runner would be doing the activity of database polling. The intermediate flow can be implemented in following manner. The flow would be combination for while loop, timer and database reader.

As part of the CTS flow for polling start with While operator, put the valid condition for the while operator for e.g. “IsNullOrEmpty(SystemName)” where SystemName is the flow variable declared in the mapper and assigned with empty string value. After the while operator put timer operator, for allowing it periodic repetition of particular activity, in the above case it is going to be polling the  database with particular query. Configure the time value with which you want to poll the database. After the time operator, put a look up join operator, which will do the look up for the particular table or field in a database. The look up join will use the database reader to get the data from the polling table in database.  After the look up join put a mapper operator which will assign the value from the query to the condition variable, in the above case “SystemName”. “SystemName” will have valid value only if the query from the database reader operator returns valid value, else it would be empty. So the execution for the above flow will continue until the database reader return valid value. In case if you want to stop the DB polling explicitly after certain number of polling action, then you can use run code operator to check the number of polling done and then assign the value to the flow variable “SystemName”, so that the flow execution comes out of the while loop.

Above is one of the options on database polling using content transformations flows, there could be other alternatives as well.

Here is the one of approach for exception handling.

For all the operators in the CTS flows wherever, there is chance that error can occur, add an error edge to the operator like shown below.

Then point the error edge operator to a common flow that will do the common exception handling and notification through flow runner. In the above case we point the error edge to a common flow through flow runner; the error edge flow runner will execute the common flow. This common flow will do the common exception handling. Below can be one of the sample exceptions handling flow. In the common flow you use the run code operator to do all the exception logging that can be down to various sources like event logs, FSIA log files etc.

Even after logging the exception challenge remains how this can be notified to various monitoring. Typically monitoring tools are port based so what they do is monitor certain set of ports on a particular server.  So for notification you can create an exe which leverages certain port and ask the monitoring tools to monitor that port. So whenever exception is raised, you can terminate exe which is leveraging particular port. On the exe termination, monitoring tools do not receive the response and then they raise the red flag or an alert.

Instead of leveraging or building an exe, you can leverage IIS services as well, any ways IIS is pre-requisite for FSIA deployment. So you can have default web site with port (*.80) been stopped when the exception is been raised. Then you can have monitoring tools monitoring the default web site port. So in the exception handling flow you can execute the script which will stop the default web site or any web site configured in IIS for monitoring of flows. Below is sample, in the exception handling flow, in the run code operator, below code would launch as command prompt exe which would then execute the “StopFlowHostingEngineSite.bat” batch file which has script for stopping particular web site on an IIS.

string command = @"D:\Temp\StopFlowHostingEngineSite.bat";
// create the ProcessStartInfo using "cmd" as the program to be run,
// and "/c " as the parameters.
// Incidentally, /c tells cmd that we want it to execute the command that follows,
// and then exit.
System.Diagnostics.ProcessStartInfo procStartInfo = new System.Diagnostics.ProcessStartInfo("cmd", "/c " + command);

// The following commands are needed to redirect the standard output.
// This means that it will be redirected to the Process.StandardOutput StreamReader.
procStartInfo.RedirectStandardOutput = true;
procStartInfo.UseShellExecute = false;
// Do not create the black window.
procStartInfo.CreateNoWindow = false;
// Now we create a process, assign its ProcessStartInfo and start it
System.Diagnostics.Process proc = new System.Diagnostics.Process();
proc.StartInfo = procStartInfo;
proc.Start();
System.Threading.Thread.Sleep(5000);
proc.Kill();

StopFlowHostingEngineSite.bat will contain the following script

%windir%\system32\inetsrv\appcmd stop site /site.name:"Default Web Site"

Please note that this is one of the approaches for exception handling and notification to external monitoring tools, there can be other ways as well. Depending on you requirements, please make appropriate selection.

When querying the query server with valid FQL query and with all fields in index profile and valid index profile been uploaded, if you get following error “packfql:Query Error:Unknown root scope: fieldname”, then resolution for the same is to do the below steps

It is quite possible that your QRServer was not properly update with the new index profile. Recommendation would be to try the following and see if any step resolves it:

1. Restart the QRServer
2. Remove/Backup the %FASTSEARCH%\var\qrserver\webcluster\15100\cache_cs directory and restart the QRServer
3. Refresh views: view-admin –m refresh –a

Could not connect to Config Server at host FQDN for Server port 16005

 Resolution

Unchecking IPv6 option in network property does not work. You have to completely disable by adding “DisabledComponents” DWORD entry with value 0xFF under HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip6 \Parameters\

 Even if is 64 Bit Windows Server 2008 or Windows Server 2008 R2, please ensure that you are creating 32 bit DWORD value as mentioned above. Irrespective of 32 bit or 64 bit, you have to create 32 bit DWORD value as mentioned above.

C:\Program Files\Windows Installer XML v3\bin>dark.exe <MSI File Path> <Path for Wix File (*.wxs)>

1            Introduction

.NET supports lot of API’s for doing instrumentation with help of trace, TraceSource classes etc, but there is nothing on the client side front in web apps to do the instrumentation. With so many applications been build using Ajax there is no way to instrument the java script code. In fact ASP.Net Ajax framework also has nothing to support this out of the box. This gives the approach to instrument the client side java script code

Note: This is currently in development phase. Will upload the code when it is fully developed and tested

Ajax instrumentation framework gives the option to instrument the client side java script code. This framework comprises of ASP.Net script controls and client-script libraries. Client side java script code invokes the Ajax instrumentation java script API’s to instrument the code.

1.        .Net 2.0

2.       Asp .Net Ajax Framework.

3.       IIS.

Figure 3‑1: Ajax Instrumentation

Ajax instrumentation can be categorized in to following components

1.       Server Side Components

2.       Client Side Components

3.       Ajax Web Service / WCF Service

This would be typically an ASP.Net 2.0 Web app using Ajax Instrumentation. Block diagram is been shown below

Figure 3‑2: Server Side Components for Web Apps.

It is script control which adds the Ajax trace listeners as per the config file. There would be separate config file for Ajax instrumentation. Please see the configuration section for details.

Ajax trace listener is a component which when added to the trace manager listener collection will emit the java script so as to add the listener at client side. It is been inherited from Base Listener class.

All the web apps that need to use the Ajax instrumentation need to inherit their web page from Web Page Base class. This class basically overrides the pre-render and Init events to inject the trace manager script control which in turn will injects the trace listener. Trace manager script control has overridden pre-render method, which basically gets the list of the listeners added to trace manager; each listener has Create Script property which returns the script for registering the listener at client side. Trace managers just iterates through all the listeners in the collection gets the script associated with it and registers it using the script manager.

This can also be stated as Client Side asynchronous communication layer.

Figure 3‑3: Client Side Components (Asynchronous Communication Layer)

This comprises of the java script code associated with web app. For invoking the Ajax instrumentation API’s, java script code associated with it needs to be injected. This needs to be done manually by the developer. Developer will adding java script code for the API’s and will be passing appropriate parameters to it.

Java script code which invokes the Ajax instrumentation API’s

This is a client side java script class. The entire set of java script API’s that need to be invoked to do java script diagnostics will be in this class. This class will also be holding the listeners collections, which will be initialized by the scripts generated by the trace manager control. The trace manager control will be injected in to the page by Web page base class. When the trace manager Ajax instrumentation API is been invoked internally it iterates through all the listener collection and calls the respective API’s of that Ajax listener class.

This is again a java script class. API’s of this class will be only be invoked if the listener is been added in the trace manager using configuration. API’s of this class will invoke the Ajax trace web service / WCF service and pass the trace data to it. This class will be basically queuing up all the tracing requests, will be passing all the queued data after configured time interval. Web Service will be invoked by the trace listener class after specific interval.

This also can be categorized as server side component for Asynchronous communication layer.

Figure 3‑4: Server Side Asynchronous Communication Layer

This service will be invoked by the java script Ajax instrumentation API’s. This service then passes the trace data that it has received to the Sym Instrumentation.

 .Net has some in built standard listeners

1.       TextWriterTraceListener- Writes the trace output to the xml, csv, text files. There are couple of variants of TextWriterTraceListener

a.       XmlWriterTraceListener

b.      DelimitedListTraceListener

Above 2 trace listeners are derived from the TextWriterTraceListener and are used for writing the trace information to XML files and CSV files respectively.

2.       EventLogTraceListener – It is used to route tracing messages to the Windows event log

3.       DefaultTraceListener - This class is used when we want to see the tracing output in the Visual Studio Environment. If there is requirement to capture Trace messages outside of Visual Studio .NET, then some other Trace Listeners must be used.

4.       ConsoleTraceListener – Writes the trace information to the console output.

.NET framework provides the flexibility of writing custom Trace Listeners. In order to implement custom Trace Listener, it must be inherited from TraceListener class. Typically a custom trace listener is used where trace data needs to be saved in some kind of custom data source, now number of custom data sources may vary, for each custom data source a separate listener needs to be written. There might be requirement to log the same trace data in to different data source. So in that case .Net provider model in combination with custom trace listener comes in handy. Below is high level diagram of how to use same custom trace listeners accross diffrent data sources.

In this case Custom trace listener will be generic trace listener and will not be agnostic to any data source. Instead of creating separate listener, different set of providers for each data source will be created; common functionality across the provider will be encapsulated in base provider class. Another advantage with this approach is that if trace data needs to be logged in to the additional data source, then we just need to write another provider specific to that data source and plug it in the config file.

Advantage of using the .Net provider model

1.       The main advantage by using the provider model is that, Custom Trace Listener becomes a generic trace listener; it can be configured to trace the data in any of the data source by writing the provider for that data source. If the provider model is not been used then Custom Trace Listener becomes specific to custom data source, and if the trace information needs to be traced in some other data source then new custom trace listener needs to be written right from scratch.

2.       If additional tracing information needs to be passed, then provider model can provide single insertion point, but if it is not been used then it needs to replicated to all the custom listeners.

3.       No need to explicitly instantiate classes. The .NET framework will automatically manage class instantiation, including re-using classes that have already been instantiated. This improves the memory management of your application.

4.       Switching data sources is much easier. Changing the data source of trace data is a simple as replacing your existing concrete (implementer) class with a new concrete (implementer) class and inheriting from your provider class. Code in the custom trace listener remains unchanged, thereby reducing effort required to switch data sources and the amount of regression testing required thereafter.

ProviderModelAndCutomTraceListener.jpeg

Below code snippet contains the code of XMLSerializable dictionary.

using System;

using System.Collections.Generic;

using System.Text;

using System.Xml.Serialization;

using System.Runtime.Serialization;

namespace CustomType

{

    /// <summary>

    /// Xml serializable dictionary object.

    /// </summary>

    /// <typeparam name="TKey">Key type</typeparam>

    /// <typeparam name="TValue">Object type</typeparam>

    [SerializableAttribute]

    public class XmlSerializableDictionary<TKey, TValue> : Dictionary<TKey, TValue>, IXmlSerializable

    {

        #region Constructors.

        /// <summary>

        /// Initializes a new instance of the <see cref="XmlSerializableDictionary&lt;TKey, TValue&gt;"/> class.

        /// </summary>

        public XmlSerializableDictionary()

            : base()

        {

            // default constructor.

        }

        /// <summary>

        /// Initializes a new instance of the <see cref="XmlSerializableDictionary&lt;TKey, TValue&gt;"/> class.

        /// </summary>

        /// <param name="info">A <see cref="T:System.Runtime.Serialization.SerializationInfo"></see> object containing the  

        information required to serialize the <see cref="T:System.Collections.Generic.Dictionary`2"></see>.</param>

        /// <param name="context">A <see cref="T:System.Runtime.Serialization.StreamingContext"></see> structure

        containing the source and destination of the serialized stream associated with the

        <see  cref="T:System.Collections.Generic.Dictionary`2"></see>.</param>

        protected XmlSerializableDictionary(SerializationInfo info, StreamingContext context)

            : base(info, context)

        {

        }

        #endregion

        private const string ITEM = "item";

        private const string KEY = "key";

        private const string VALUE = "value";

        #endregion

        #region IXmlSerializable Members

        /// <summary>

        /// This property is reserved, apply the <see cref="T:System.Xml.Serialization.XmlSchemaProviderAttribute"></see> to

        the class instead.

        /// </summary>

        /// <returns>

        /// An <see cref="T:System.Xml.Schema.XmlSchema"></see> that describes the XML representation of the object that is

        produced by the <see cref="M:System.Xml.Serialization.IXmlSerializable.WriteXml(System.Xml.XmlWriter)"></see> method

        and consumed by the <see cref="M:System.Xml.Serialization.IXmlSerializable.ReadXml(System.Xml.XmlReader)"></see>

        method.

        /// </returns>

        public System.Xml.Schema.XmlSchema GetSchema()

        {

            return null;

        }

        /// <summary>

        /// Generates an object from its XML representation.

        /// </summary>

        /// <param name="reader">The <see cref="T:System.Xml.XmlReader"></see> stream from which the object is

        deserialized.</param>

        public void ReadXml(System.Xml.XmlReader reader)

        {

            XmlSerializer keySerializer = new XmlSerializer(typeof(TKey));

            XmlSerializer valueSerializer = new XmlSerializer(typeof(TValue));

            bool wasEmpty = reader.IsEmptyElement;

            reader.Read();

            if (wasEmpty)

                return;

            while (reader.NodeType != System.Xml.XmlNodeType.EndElement)

            {

                reader.ReadStartElement(ITEM);

                reader.ReadStartElement(KEY);

                TKey key = (TKey)keySerializer.Deserialize(reader);

                reader.ReadEndElement();

                reader.ReadStartElement(VALUE);

                TValue value = (TValue)valueSerializer.Deserialize(reader);

                reader.ReadEndElement();

                this.Add(key, value);

                reader.ReadEndElement();

                reader.MoveToContent();

            }

            reader.ReadEndElement();

        }

        /// <summary>

        /// Converts an object into its XML representation.

        /// </summary>

        /// <param name="writer">The <see cref="T:System.Xml.XmlWriter"></see> stream to which the object is

        serialized.</param>

        public void WriteXml(System.Xml.XmlWriter writer)

        {

            XmlSerializer keySerializer = new XmlSerializer(typeof(TKey));

            XmlSerializer valueSerializer = new XmlSerializer(typeof(TValue));

            foreach (TKey key in this.Keys)

            {

                writer.WriteStartElement(ITEM);

                writer.WriteStartElement(KEY);

                keySerializer.Serialize(writer, key);

                writer.WriteEndElement();

                writer.WriteStartElement(VALUE);

                TValue value = this[key];

                valueSerializer.Serialize(writer, value);

                writer.WriteEndElement();

                writer.WriteEndElement();

            }

        }

        #endregion

   }

}