FAST ESP 5.x - fsearch maxmemusage clarification

Context : FAST ESP 5.x Architecture

If your fsearch process fails to start with the below error:

Loading attribute vectors will exceed maximum memory usage, set to 1887436800 B (1800.00 MB). fsearch will not load this index. Please reduce index size or number of navigator fields and sortable fields in the index profile

You might have applied the workaround of setting the parameter maxmemusage to 3221225472 in the fsearch.addon.

 

=> I would like to clarify a few things regarding the fsearch maxmemusage parameter.

 

The maxmemusage is not controlling the memory the fsearch process will use. Its main purpose is to prevent an fsearch to load a huge partition index putting a risk its stability.

 

What is the memory limit of the fsearch process?

On Linux 64-bit OS, the fsearch should have most of the 4 GB address space available.

On Windows 64-bit OS, the fsearch also has the full 4 GB address space. This is due to the fact that fsearch alongside fixmlindex have the flag /LARGEADDRESSAWARE on (https://msdn.microsoft.com/en-us/library/windows/desktop/aa366778(v=vs.85).aspx#physical_memory_limits_windows_server_2008_r2 ).

Those are the only two ESP processes with this linkage flag, all the others (including the indexer) cannot address more than 2 GB of memory.

 

What is a huge partition index for a fsearch process?

The responsibility of creating a partition index falls to the indexer which has plenty of mechanisms to limit the "size" of partition index (docsDistributionMaxMB is one of them).

 

Upon startup the fsearch process

1. loads in memory all navigators’ indexes

The size of the navigation indexes depends on the number of documents the partition index has + the number of navigators defined in your index profile.

[2013-02-01 08:41:16.819] VERBOSE fsearch Number of documents indexed: 16

[2013-02-01 08:41:16.823] VERBOSE fsearch Active attributevector memory usage is 4521 B (0.00 MB)

 

2. Evaluate the max memory cache for the main partition indexes.

[2013-02-01 08:41:17.404] VERBOSE fsearch Maximum cache memory usage: 822083575 B (784.00 MB)

The global cache size is defined in rtsearchrc.xml parameter maxMBCacheSize and is distributed across all fsearch depending on how many documents their corresponding partition index has.

The sum of #1 + #2 is the value tested against the maxmemusage (default 1800Mb).

[2013-02-01 08:41:17.404] VERBOSE fsearch Total attributevector + cache memory usage is 822088096 B (784.00 MB)

Please note that the navigation indexes are actually loaded, the index cache is actually a Maximum cache usage (potential usage).

 

In my example I have 4521 + 822083575 = 822088096 bytes. With only 16 documents indexed, you can imagine that 784 Mb is a max memory usage not the actual one.

 

Also you can easily understand that the fsearch process memory usage is not limited to the navigation indexes and indexes cache alone, it also has to allocate memory to cache search results amongst other things.

 

Thus allowing your fsearch process to go up to 3221225472 bytes (3 Gb) could harm your search experience (fsearch crash when reaching the max mem of 4 GB).

Yes it will solve the error "will not load this index" but IMHO it should be considered as a temporary solution.

 

For long-term solution, the usual recommendation in those cases is :

 

1. To review your document distribution. Things like adding a new partition or column should be considered.

2. To review your index profile. Make sure all defined navigators are actually used in the query side.

 

Hope this helps.