They would have stopped and restarted the Osearch but still the same behavior.

Here is one of the steps which can help us to come out of the situation.

On the SharePoint server, open the command prompt and go the bin location

Use stsadm -o sync -listolddatabases <days>

listolddatabases

A valid number of days, such as "2"

Lists content databases that have not been synchronized since n days. The presumption is that they have been deleted or moved.

If the command returns the list of databases that means that those databases have not been synchronized.

Now use stsadm -o sync -deleteolddatabases <days>

deleteolddatabases

A valid number of days

Performs the same operation as the listolddatabases parameter, except it deletes old records corresponding to these databases. It does not delete the databases themselves.

Restart the incremental crawl and check the behavior.

The psconfig upgrade command line was failing immediately and hence no upgrade logs was generated.

Below was the stack trace:

-----------------------------------------------------------------------------------------------------------------

Exception: System.ArgumentException: An item with the same key has already been added.

   at System.ThrowHelper.ThrowArgumentException(ExceptionResource resource)

   at System.Collections.Generic.Dictionary`2.Insert(TKey key, TValue value, Boolean add)

   at System.Collections.Generic.Dictionary`2.System.Collections.IDictionary.Add(Object key, Object value)

   at Microsoft.SharePoint.Administration.SPAutoSerializingObject.DeserializeBasicObject(XmlElement xmlValue)

   at Microsoft.SharePoint.Administration.SPAutoSerializingObject.SetStateXml(XmlNodeList childNodes)

   at Microsoft.SharePoint.Administration.SPAutoSerializingObject.SetStateXml(XmlDocument value)

   at Microsoft.SharePoint.Administration.SPPersistedObject.Initialize(SPConfigurationDatabase configurationDatabase, Guid id, Guid parentId, String name, SPObjectStatus status, Int64 version, XmlDocument state)

   at Microsoft.SharePoint.Administration.SPConfigurationDatabase.GetObject(Guid id, Guid parentId, Guid type, String name, SPObjectStatus status, Byte[] versionBuffer, String xml)

   at Microsoft.SharePoint.Administration.SPConfigurationDatabase.GetObject(SqlDataReader dr)

   at Microsoft.SharePoint.Administration.SPConfigurationDatabase.RefreshCache(Int64 currentVersionOverride, List`1& newObjects, List`1& deletedObjects, Int64& newestObjectVersion)

   at Microsoft.SharePoint.Administration.SPConfigurationDatabase.RefreshCache()

   at Microsoft.SharePoint.PostSetupConfiguration.Farm.RefreshCacheAndUpdate(TaskBase task)

   at Microsoft.SharePoint.PostSetupConfiguration.UpgradeBootstrapTask.Run()

   at Microsoft.SharePoint.PostSetupConfiguration.TaskThread.ExecuteTask()

--------------------------------------------------------------------------------------------------------------------

Since the upgrade log was not getting generated, hence reviewed the ULS and the event logs.

Noticed below error many a times in the ULS logs

The SPPersistedObject with Name ABC_7d94e326-919e-4aa1-9577-b1549c028a81, Id bf7a2181-d667-4da7-ab96-896ed5bcf638, Parent e6ee5401-e263-40d5-a3b1-ab4a90ee20bb failed to initialize with the following error: System.ArgumentException: An item with the same key has already been added.     at System.ThrowHelper.ThrowArgumentException(ExceptionResource resource)     at System.Collections.Generic.Dictionary`2.Insert(TKey key, TValue value, Boolean add)     at System.Collections.Generic.Dictionary`2.System.Collections.IDictionary.Add(Object key, Object value)     at Microsoft.SharePoint.Administration.SPAutoSerializingObject.DeserializeBasicObject(XmlElement xmlValue)     at Microsoft.SharePoint.Administration.SPAutoSerializingObject.SetStateXml(XmlNodeList childNodes)     at Microsoft.SharePoin...

------------------------------------------------------------------------------------------------------------------------

Below were the errors in the Event Log:

Event Type:            Error

Event Source:                       Office SharePoint Server

Event Category:                   Office Server Shared Services

Event ID:        6480

Date:                                      10/9/2009

Time:                                      7:29:46 PM

User:                                      N/A

Computer:   ############

Description:

An exception occurred while executing the alternate url synchronizer job.

Reason: An item with the same key has already been added.

Techinal Support Details:

System.ArgumentException: An item with the same key has already been added.

   at System.ThrowHelper.ThrowArgumentException(ExceptionResource resource)

   at System.Collections.Generic.Dictionary`2.Insert(TKey key, TValue value, Boolean add)

   at System.Collections.Generic.Dictionary`2.System.Collections.IDictionary.Add(Object key, Object value)

   at Microsoft.SharePoint.Administration.SPAutoSerializingObject.DeserializeBasicObject(XmlElement xmlValue)

   at Microsoft.SharePoint.Administration.SPAutoSerializingObject.SetStateXml(XmlNodeList childNodes)

   at Microsoft.SharePoint.Administration.SPAutoSerializingObject.SetStateXml(XmlDocument value)

   at Microsoft.SharePoint.Administration.SPPersistedObject.Initialize(SPConfigurationDatabase configurationDatabase, Guid id, Guid parentId, String name, SPObjectStatus status, Int64 version, XmlDocument state)

   at Microsoft.SharePoint.Administration.SPConfigurationDatabase.GetObject(Guid id, Guid parentId, Guid type, String name, SPObjectStatus status, Byte[] versionBuffer, String xml)

   at Microsoft.SharePoint.Administration.SPConfigurationDatabase.GetObject(SqlDataReader dr)

   at Microsoft.SharePoint.Administration.SPConfigurationDatabase.RefreshCache(Int64 currentVersionOverride, List`1& newObjects, List`1& deletedObjects, Int64& newestObjectVersion)

   at Microsoft.SharePoint.Administration.SPConfigurationDatabase.RefreshCache()

   at Microsoft.SharePoint.Administration.SPConfigurationDatabase.StoreObject(SPPersistedObject obj, Boolean storeClassIfNecessary, Boolean ensure)

   at Microsoft.SharePoint.Administration.SPConfigurationDatabase.PutObject(SPPersistedObject obj, Boolean

*****************************************************************************************

Event Type:            Error

Event Source:                       Windows SharePoint Services 3

Event Category:                   Timer

Event ID:        6398

Date:                                      10/9/2009

Time:                                      7:29:45 PM

User:                                      N/A

Computer:    ############

Description:

The Execute method of job definition Microsoft.SharePoint.Administration.SPConfigurationRefreshJobDefinition (ID 31cc6fc1-2aef-4fe6-b248-e594f6e57961) threw an exception. More information is included below.

An item with the same key has already been added.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

*****************************************************************************************

Event Type:            Error

Event Source:                       Windows SharePoint Services 3

Event Category:                   Timer

Event ID:        6398

Date:                                      10/9/2009

Time:                                      7:29:46 PM

User:                                      N/A

Computer:    ###############

Description:

The Execute method of job definition Microsoft.Office.Server.Administration.AlternateUrlSynchronizerJob (ID a60272bc-65da-4322-b9cb-50bdb5f73b88) threw an exception. More information is included below.

An item with the same key has already been added.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

****************************************************************************************

Ran the following query:

SELECT Id, ClassId, ParentId, Name, Status, Version, Properties FROM Objects

From the results of the above query, searched for the “ABC_7d94e326-919e-4aa1-9577-b1549c028a81” and found an entry for the same.

Since the event log was also showing errors related to “URL” and “AlternateUrlSynchronizerJob” , checked properties column of the Object.

There was an entry for “Microsoft.SharePoint.Administration.SPAlternateUrlCollection”

Ran the same above query on a Out of the Box test server farm and found that “Microsoft.SharePoint.Administration.SPAlternateUrlCollection” is available inside the Objects table under the config DB as many times the count of SharePoint Web applications.

For example if we have 4 web applications, we should see 4 entries of “Microsoft.SharePoint.Administration.SPAlternateUrlCollection” under the Objects table.

But at customers end there was an extra entry with the name “ABC_7d94e326-919e-4aa1-9577-b1549c028a81”.

This seemed to be an orphan entry.

Hence we used

“stsadm.exe -o deleteconfigurationobject –id”

to delete the orphan web application ID.

After deleting the orphan entry, the psconfig progressed from Step 2 and started to upgrade the databases.

When we use a single web application for SSP and Mysites then the content database will contain both SSP admin site and the personal sites.

Later they realize that it not as per the best practices to use the same web application for SSP and Mysites and now how to separate Mysites from the SSP web application.

Here are the steps which will assist us to separate the SSP and Mysite into different web applications:

1. Create a new web application for mysite.

Create a Mysite Site Host template site collection for the web application

Go to Managed Path under Application Management and add the "Personal"  as Wildcard inclusion.

Enable the Self Service Site Management for the web application under the Central admin.

2. Modify the My Site settings under the SSP admin page to point the new mysite web application.

In the above steps we have taken care of creating a new web application for mysite and updating the reference in the mysite settings

But what about the personal sites which are still stored in the SSP content database.

If you run a query as given in below Figure against the SSP content database it will return all the sites including the personal sites.

Our next task would be to move these personal sites from SSP content database to the mysite web application content database.

1. Take a SQL backup of SSP content database

2. Create a list of sites in the sites.txt file using the redirector operation ">", use the following syntax:

stsadm -o enumsites -url http://<servername> –databasename SSP_ContentDB > C:\sites.txt

3. Open the Sites.txt and make sure that it contains the complete list of all the sites.

4. Remove the ssp/admin site and the root site from the sites.txt file.

5. Make sure that the sites.txt contains only the personal sites.

6. Remove the content database from the Mysite web application

7. Add the mysite content database to the SSP web application

8. Check if SSP web application contains 2 content databases.

9. Move the site collections specified in the sites.xml file from SSP content database to MySite Content database, use the following syntax:

stsadm -o mergecontentdbs -url http://<servername> -sourcedatabasename SSP_Content -destinationdatabasename MySite_Content -operation 3 -filename c:\sites.xml

Thats it!

We have successfully achieved our requirement.

While troubleshooting such issues we can use below scenarios:

1. A user creates an item and assigns to self

2. A user creates an item and assigns to a different user

We need to check in which scenario the email is not sent.

If the user mentions that the email is not sent in scenario 1 then here is the possible explanation.

The logical reason for the above is just because there is no need of an email when the user knows that there is a change in the item as he has himself done those changes.

If the user mentions that the email is not sent in scenario 2 then :

In such scenarios we need check if option "Require content approval for submitted items” under Versioning settings link of the list settings is enabled.

If that option is enabled then we need to "Approve" the item. Once item is approved, immediately an email is sent to the person to whom the task is assigned . But that email is sent as a change email.

This behavior seems to be by design

Issue
-----------------------------------------

The Query Server under Search Administration page always shows the below 2 status

Server Status - Error
Propagation Status - Waiting for Initialization (MOSS-XYZ-WFE1)

Sometimes we also get error message in the Application log
Event Type: Error
Event Source: Office SharePoint Server
Event Category: Office Server Shared Services
Event ID: 6482
Date: DD/MM/YYYY
Time: hh:mm:ss
User: N/A
Computer: Server Name
Description:
Application Server Administration job failed for service instance Microsoft.Office.Server.Search.Administration.SearchServiceInstance (GUID).
Reason: The underlying connection was closed: An unexpected error occurred on a send.

If we receive the above error which is a good hint, then we can try the below article which contains the solution
http://support.microsoft.com/?id=962928

But what if there are no errors and still we notice the below?
Server Status - Error
Propagation Status - Waiting for Initialization (MOSS-Stage-WFE1)

Possible resolution
===============
For Issue -> Server Status - Error
Try the steps given in article http://support.microsoft.com/?id=962928

For Issue -> Propagation Status - Waiting for Initialization (MOSS-Stage-WFE1)

Try to repair MOSS installation on the problematic server.
Ran PSConfig wizard on the server.

This may resolve the issue completely.

When we create a new site under the top level site collection the Content Query Webpart fails to show the data and instead displays :

"There is a problem with the query that the web part is using. Check the configuration of the web part and try again..."

The Site collection had more than 200 sites under the top level site.

Each site has multiple document libraries. We found that the issue happens only when we add more document libraries in the new site.

If we reduced the sites from the site collection we were able to add more document library in the new site and the Content Query Webpart still worked.

Hence we suspected that we might be reaching content query webpart limits.

http://technet.microsoft.com/en-us/library/cc263061.aspx.

Content Query Web Part

The Content Query Web Part utilizes the Windows SharePoint Services cross-list querying mechanism to retrieve content from a SharePoint site collection. If the Web Part is configured to issue a query that involves a large number of lists, the cross-list query mechanism may raise an exception.

By default, cross-list queries have a list limit of 1,000. This means that if you configure the Content Query Web Part with a query that includes more than 1,000 lists, the cross-list query will not complete, and the Web Part will not show any content. The reason for this throttling is to avoid overburdening SQL Server 2005.The more lists the cross-list query includes, the longer it takes for the database server to return the content the query is asking for. For very large numbers of lists, this could cause the database server to disproportionately process cross-list queries at the expense of other requests.

If your requirements involve querying for more than 1,000 lists, you can increase the list limit if the database load that the operations require is acceptable. You can do this by adding a MaxListLimit attribute to the ListsOverride property of the Web Part. For example, if you wanted to raise the list limit to 2000, you would set the ListsOverride property as:
 <Lists ServerTemplate="850" MaxListLimit="2000">

This Table contains all the Job Titles.

http://technet.microsoft.com/en-us/library/cc678870.aspx 

"proc_StartTimerRunningJob" is the stored procedure responsible for creating the Job Titles inside the TimerRunningJobs Table.

"proc_DeleteAllTimerLocksAndRunningJobs" is the stored procedure responsible for deleting all the Job Titles from the TimerRunningJobs Table.

Sometimes I have noticed that "Shared Services Timer Job" is not displayed for one of the servers in the SharePoint farm inside Timer Job Status Page.

Below are the different steps which may help in resolving the above issue:

1. Disable and Enable the Shared Services Timer Job from the Timer Job Definition Page.

2. Disconnect and reconnect the problematic Server from the SharePoint farm.

If the above steps does not resolves the issue then :

3. Create a new SSP.

• The following events are logged in the Application log:

Event Id: 2268
Event Type: Error
Event Source: w3svcwp
Description: Could not load all ISAPI filters for site/service. Therefore startup aborted

Event Id: 2214
Event Type: Error
Event Source: w3svcwp
Description: The HTTP Filter DLL C:\Program Files\Microsoft Forefront Security\SharePoint\FSSPUsernameFilter.dll failed to load. The data is the error.

• The SharePoint Central Administration page is unavailable.

And we have a KB article for the same - KB # 943621

But what if Forefront is not uninstalled and when we browse the SharePoint site we receive HTTP 500 and the following events are logged in the Application log:

Event Id: 2268
Event Type: Error
Event Source: w3svcwp
Description: Could not load all ISAPI filters for site/service. Therefore startup aborted

Event Id: 2214
Event Type: Error
Event Source: w3svcwp
Description: The HTTP Filter DLL C:\Program Files\Microsoft Forefront Security\SharePoint\FSSPUsernameFilter.dll failed to load. The data is the error.

To resolve the above error:

Add the user under which the Application Pool identity for the problematic SharePoint Site runs to the following folders as Full Control:

C:\Program Files (x86)\Microsoft Forefront Security\SharePoint

C:\Program Files (x86)\Microsoft Forefront Security\SharePoint\FSSPUsernameFilter.dll

Also there may be some instances when we browse any SharePoint Site we receive an error:

"Accessor is invalid"

or

In such cases follow the below steps:

Run the below query on problematic SharePoint site Content DB:

SELECT * FROM AllDocs WHERE  (VirusStatus > 0) AND (VirusStatus IS NOT NULL)

If it returns results, which means the VirusStatus column has been changed to a non-zero and non-null value.

Rerun the Antivirus scan and re-query the above SQL.

If issue still exists then you may need to get in touch with your security team for more information.

Note: Do not modify the SharePoint Table manually as it may lead to an unsupported environment.

References:

http://support.microsoft.com/kb/943621/en-us

http://support.microsoft.com/?id=928169

http://support.microsoft.com/kb/841057/en-us

"The folder you entered does not appear to be valid. Please Choose another."

As per the Whitepaper=>  Microsoft SharePoint Services Understanding and Troubleshooting the SharePoint Explorer View

http://www.microsoft.com/Downloads/details.aspx?familyid=C523AC7A-5724-48BE-B973-641E805588F4&displaylang=en