This is a great article and is now a part of TechNet as well http://technet.microsoft.com/en-us/library/cc288609.aspx

But during some instances customers do come up with questions.

So i have tried to put some examples out here.

Internal address - The address used by Proxy to contact SharePoint server

          Note: Ideally the internal address should not be used by users to browse the site, it should be used only by the proxy to contact the server.

          This will avoid lot of confusion that happens when the AAM needs to be changed when the address for browsing the site changes.

External/Public address - Used by users on the internet

Company Network/Private address - Used by users on the private / company network

For any scenario the ISA rule configuration has to be done per "Configuring the reverse proxy server" topic of the link http://technet.microsoft.com/en-us/library/cc288609.aspx

I have been asked many a times that can we alter the AAM after the web application has been extended with a particular value in the URL's. And the answer is yes without any reason for failure. If we are aware of the address to be used at the time of extending the web application, we can use that and the AAM configuration is done automatically. But there will be many instances when the address is not known then or has changed. So modify AAM's are absolutely safe and can be done at any time to cater to our needs.

Also the address used in AAM need not be added to the host header in the IIS Web site. At the same time the host header value of IIS is no replacement for AAM. If you want SharePoint to respond to an address it has to be present in the AAM.

Web site in host header(hh) mode is an exception

Lets consider different scenarios

Scenario 1

Company network address - http://www.contoso.com

External address - http://www.contoso.com

Internal address - http://Sharepointservername

Here on the LAN www.contoso.com should resolve the internal IP address

The above AAM setting is not mandatory. You can use below setting also

Company network address - http://www.contoso.com

External address - http://www.contoso.com

Internal address - http://www.contoso.com

 

And so the AAM would be just one entry for http

Scenario 2

Company network address - http://www.contoso.com

External address - https://www.contoso.com

Internal address - http://Sharepointservername

Scenario 3

Company network address - http://sharepoint.contoso.com

External address - http://www.contoso.com

Internal address - http://Sharepointservername

Scenario4

Company network address - http://sharepoint.contoso.com

External address - https://www.contoso.com

Internal address - http://Sharepointservername

Scenario 5

Company network address - https://sharepoint.contoso.com

External address - https://www.contoso.com

Internal address - http://Sharepointservername

Then this article might be helpful....read on

First lets get a little familiar with the terms used

• First stage recycle bin -  It is nothing but the site's Recycle Bin. Anything a user delete's from the site goes here.

               Users can delete it from here or restore the item back

• Second stage recycle bin - The  site Collection Recycle Bin. If item is deleted from the site recycle bin it lands here.

               Site collection Administrators can delete it from here or restore the item back

Some helpful articles

• New to SharePoint 2007 recycle bin, read View, restore, or delete items in the Recycle Bin of a SharePoint site
• Want to know how to configure it then refer SharePoint Recycle Bin Administration
• An interesting article on how to use the recycle bin
• Protect and restore content by using Recycle Bins and versioning http://technet.microsoft.com/en-us/library/cc288451.aspx

How does deletion work

When you delete an item from a Web site, the item is sent to the site's Recycle Bin. If you click Recycle Bin on the Quick Launch, you can see all of the items that you've deleted from your site. You can either restore or delete the item from the Recycle Bin. When you delete an item from the Recycle Bin, the item is sent to the Site Collection Recycle Bin.

When an item is deleted from the site collection recycle bin (manually or automatically) the item is flushed from the content database. Other wise it remains in the database. Only one flag

 

Few thoughts that crossed my mind 

1. How can one land up with this big an item in the second stage recycle bin?
2. If it is in second stage recycle bin, how did they flush it from first stage?
3. Is this an acceptable behavior?

Here's what i found

1. Lets consider one example: say there is a folder in the document library which has many files uploaded. The folder is deleted which is few GB in size.
2. When we delete items from the site or the first stage recycle bin it remains in the content database. But when we delete it from second stage recycle bin, that is when it is flushed from the content database. So deleting a large file from first stage bin was not a problem.
3. Ideally speaking no, but as of now we need to leave with it.

There are certain things that can be done to avoid this situation

1. Restore the problem library to its original location. Delete the documents few at a time. Flush the second stage recycle bin periodically.

          By default the below command will show the period at which the recycle job will be run
          STSADM -o getproperty -propertyname job-recycle-bin-cleanup -url http://<sitename>

          Run the below command to run the recycle timer job periodically
          STSADM -o setproperty -propertyname job-recycle-bin-cleanup -propertyvalue "<configure a convenient time>" -url http://<site name>

2. Disable Second stage recycle bin from the central administration page
          Go to Central Administration > Application Management > Web Application General Settings
          Set Recycle Bin > Second stage Recycle Bin > OFF

                    Note: This will flush the items in the second stage recycle bin for site collections in that web application. This is a web application level setting.
                    On enabling second stage recycle bin the items will be removed from the recycle bin. The files are deleted from the database so it cannot be retrieved.

3. A custom code based on SharePoint Object Model can be used to flush the large files from the second stage recycle bin.

    Sample code

                SPSite site = new SPSite(<<siteurl>>);
                SPWeb web = site.RootWeb;

                SPRecycleBinQuery q = new SPRecycleBinQuery();

                q.RowLimit = Int32.Parse(<<number of items to be deleted>>);
                q.OrderBy = SPRecycleBinOrderBy.Default;

                SPRecycleBinItemCollection itemColl = web.GetRecycleBinItems(q);
                foreach (SPRecycleBinItem item in itemColl)
                {
                    Guid[] id = new Guid[1];
                    id[0] = item.ID;
                    itemColl.Delete(id);
                }

    Disclaimer:

    The sample code provided is on "AS IS" basis with no warranties, and confers no rights.

Recently while working with a customer, I was asked how the crawl behaves if the dedicated Web Front end (WFE) for crawling becomes unavailable. To answer this question I read through some interesting information about crawl which I thought I should share. This information would be present with you already.

Some situations when the Web Front end can become unavailable - IIS on the  WFE is hung, Application pool for the site is stopped, Network or hardware failure on the server, System rebooted unexpectedly, Server crash etc....

What is the Crawl behavior in a situation "the item has been deleted from the site or the crawl account's access to the item has been revoked"

Full crawl will mark an item for deletion in the content index during crawl. And at the end of the crawl the items are deleted from the content index. There are two separate stored procedures to perform these jobs.

And Incremental Crawl behaves in a similar manner to full crawl.

For more details on crawl refer the links below:

• How to crawl content As the topic states this tells you the basics of crawl configuration. It has further links for detailed configuration
• Crawl content This link talks about when to crawl, how to crawl and scheduling them
• Enterprise Search Protocol Handlers Here you will find the details about the complete indexing system
• Change Log This is a key feature for incremental crawl

Now going back the original question "Incremental crawl behavior when the dedicated server is unavailable"

What I noticed is that if the site is unavailable for crawling, the indexer will try to crawl the site and fail. Also the result for full and incremental crawl is not the same.

If you perform a full crawl when the site is not available, it will delete all the items in the content index. Oops so if the site becomes unavailable just before the scheduled full crawl we are in for trouble as we will end up with an empty content index.

On the other hand the consecutive incremental crawl will keep the items in the content index. But if the incremental crawl fails one hundred consecutive times, the index server removes the affected content from the index. Refer the topic "Reasons to perform a full crawl" @ http://technet.microsoft.com/en-us/library/cc160651.aspx

I tested this behavior with the steps below

1. Created a web application with a publishing site.
2. Created a SSP to crawl only this site.
3. Performed a full crawl
4. Stopped the application pool for the web application and performed the full crawl.
5. Started the application pool and performed the full crawl again
6. Scheduled the incremental crawl to occur every minute
7. Stopped the application pool for the web application

Below listed is the screenshot of the above test:

Note: Click on the images to see larger picture

FULL CRAWL

I) Site

II) Items in index after successful full crawl

III) Site unavailability after the application pool for the site is stopped.

IV) Manually initiate a full crawl. (Here full crawl is not scheduled)

V) Index status after the full crawl

VI) Error in crawl log - "The crawler could not communicate with the server. Check that the server is available and that the firewall access is configured correctly."

INCREMENTAL CRAWL

I) Site

II) Items in index after successful full crawl

III) Site unavailability after the application pool for the site is stopped.

IV) Incremental crawl schedule

V) Error in the crawl log - "The crawler could not communicate with the server. Check that the server is available and that the firewall access is configured correctly."

VI) Content still present in the index after consecutive crawls

VII) Content cleared after 100 consecutive incremental crawls

I hope this answers the question about crawl behavior when WFE setup for crawling becomes unavailable.