LogParser To The Rescue

Warning: This doesn’t seem to work with very large IIS logfiles, I tried with a 750mb file which didn’t error but was unreadable with a disk full error. My mistake, it does work, it’s TextPad that can’t handle it.

Microsoft LogParser may be the answer to our AVG logfile spam woes, I’m been fiddling with it and have come up with a quite simple way pre-processing the logs with logparser to remove the offending spam. You can put the query inline within the logparser commandline, but it’s easier to stick it in a file once it gets a bit longer. So, I have this in my file

select * into c:\logs\ex%log%out.log
from c:\logs\ex%log%.log
where not (cs(User-Agent)=’Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.1;1813)
or cs(User-Agent)=’Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.1;+SV1)’
and cs(Cookie) is null
and cs(Referer) is null)

Which you run with

C:\Program Files\Log Parser 2.2>LogParser -i:iisw3c -o:w3c file:c:\logs\avgspam.

Which yields something like

Elements processed: 209607
Elements output: 151434
Execution time: 8.47 seconds

This shows that logparser has removed ~58K rows from our log and created a new log that we can feed to our stats program.

I’ve tested this with a couple of logs from different clients now and it seems to to work.

Let me know if you have improvements or have come up with a different work around for this.

AVG better not start using more UserAgent strings though as this could get very messy.

If you’re new to logparser (as I was), this guide has some good (IIS centric) examples – Analysing IIS logs with LogParser

Update: I’ve now worked out how to use logparser with Awstats to filter out AVG spam. If you’re using Awstats this is a very quick fix.

add to del.icio.us :: Bookmark Post in Technorati :: Add to Blinkslist :: add to furl :: Digg it :: add to ma.gnolia :: Stumble It! :: add to simpy :: seed the vine :: :: :: TailRank :: post to facebook :: Bookmark on Google :: Add to Netscape :: Share on Yahoo :: Add this to Live

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: