Log Parser - How to parse IIS Logs with and without X-Forwarded-For

Question

I have IIS logs dating back to 2013. About a month ago, we switched to add X-Forwarded-For data to the IIS logs and now the files have the "_x.log" extension. I am writing these logs to a SQL Server table.

I am able to parse the previous, non-X-Forwarded-For logs using -i:IISW3C and can use -iCheckPoint for these logs.

The problem is, when it gets to the "_x.log" files, it generates parsing errors.

I am able to parse the "_x.log" files using -i:W3C", but I cant use -icheckpoint with "W3C" it seems.

I am not concerned about the non-"_x.log" files as they are historical and I can put them in their own folder and run "-i:IISW3c" on them and have their own SQL table, but the "_x.log" files, I need to use a checkpoint in order to schedule a batch file to run in real time and update my database table that includes X-Forwarded-For

Is there a workaround for this or something I am missing?

Answer 1

Unfortunately, it appears that there is no way to use the -icheckpoint parameter with the "W3C" input format in the current version of LogParser. According to the LogParser documentation, the -icheckpoint parameter is only supported for "IISW3C" and "NCSA" input formats.

One potential workaround would be to parse the "_x.log" files using the "W3C" input format without the -icheckpoint parameter, and then manually track the progress of the parsing and resume parsing from where it left off in case of an interruption. This would require some custom scripting and error handling, but it could be a viable option if you cannot find a way to use -icheckpoint with the "W3C" input format.

Another option would be to switch to a different log parsing tool that supports both X-Forwarded-For data and checkpointing for the W3C input format. Some potential alternatives to LogParser include AWStats, Webalizer, and GoAccess. However, switching to a different tool may require some changes to your parsing and database insertion process.