FAST ESP download query logs throws error "Premature end of file"
Issue :
You might have observed that many times executing the command "logtransformer\bin\launch.bat downloader" in FAST ESP 5.3 to download the query logs will simply get stuck or throws an error "Premature end of file" without downloading any query logs.
Detailed error stack:
E:\esp\components\logtransformer\bin>launch.bat downloader
[ jacorb.home unset! Will use '.' ]
[ File .\jacorb.properties for configuration jacorb not found ]
[ configuration jacorb loaded from classpath]
In CSInfoInt ctor
Log4J configured from: file:/E:/esp/etc/log4j.properties
Log file downloader 5.1.7 build 5.3.4.6
Started download.
[Fatal Error] :1:1: Premature end of file.
Finished downloading files.
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
sorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.commons.launcher.ChildMain.run(ChildMain.java:228)
Caused by: com.thoughtworks.xstream.io.StreamException: Premature end of file.
at com.thoughtworks.xstream.io.xml.DomDriver.createReader(Unknown Source
)
at com.thoughtworks.xstream.XStream.fromXML(Unknown Source)
at com.fastsearch.esp.logtransformer.datastore.meta.dao.DaoSupport.loadO
bjectFromXml(DaoSupport.java:114)
at com.fastsearch.esp.logtransformer.datastore.meta.dao.MetaDataDAOImpl.
getLogFileProducingSource(MetaDataDAOImpl.java:127)
at com.fastsearch.esp.logtransformer.datastore.FdsLogStoreManagerImpl.ge
tOrCreateLogfileProducingSource(FdsLogStoreManagerImpl.java:368)
at com.fastsearch.esp.logtransformer.harvest.download.DownLoader.fetchDa
taSource(DownLoader.java:290)
at com.fastsearch.esp.logtransformer.harvest.download.DownLoader.updateL
istOfActiveQueryServers(DownLoader.java:227)
at com.fastsearch.esp.logtransformer.harvest.download.DownLoader.runBatc
h(DownLoader.java:104)
at com.fastsearch.esp.logtransformer.harvest.download.DownLoader.main(Do
wnLoader.java:316)
... 5 more
Cause :
This issue occurs if there are any corruption of the file
$FASTSEARCH/components/logtransformer/data/datastore/datasources.xml
Resolution :
1. Stop the logtransformer
a. nctrl stop logtransformer
b. Please verify that the logtransformer process is no longer running in Task Manager. If necessary please kill the process.
2. Truncate the query statistics table:
a. %FASTSEARCH%\rdbms\bin\psql -p16070 -dvespa -Ufast (password is fast)
b. truncate queryStatistics; truncate periodProperty;
c. \q
3. Take a backup and then delete the below files
a) Check that all META files in
$FASTSEARCH/components/logtransformer/data/download have a size greater than zero, and remove any that has 0 KB.
b) Delete $FASTSEARCH/components/logtransformer/data/datastore/datasources.xml
c) Delete $FASTSEARCH/components/logtransformer/data/datastore/logmerger-statistics.xml
d) Delete $FASTSEARCH/components/logtransformer/data/datastore/querylog/crateindex.xml
4. Start the logtransformer
a. nctrl start logtransformer
The downloader should start downloading the querylogs within 15 minutes. The query report generator should start running at 5 past the hour. If for some reason this doesn’t happen you can run the commands manually:
cd %FASTSEARCH%\components\logtransformer\bin
launch.bat downloader
launch.bat logmerger
Once those commands are finished, run the following:
cd %FASTSEARCH%\esp4j\bin
launch.cmd queryreportgenerator -s "2014-09-............" -e "2014-10............." -D -L
Set the start date to the date you want the statistics to first be generated for and the end date to the current time.
Published by - Prasad Joshi