Share via

More Thoughts on Measuring Anti-Phishing Accuracy

Some of you may have seen stories comparing IE7’s anti-phishing accuracy with our competitors, citing different studies than the one I blogged about earlier that showed IE7’s Phishing Filter had the best overall accuracy. Paul Robichaux, from 3Sharp (the company that ran the study I cited), provides his initial take on this other study here.

Tony Chor
Group Program Manager

Comments

  • Anonymous
    November 14, 2006
    you guys said you are the better, but the firefox team said that they are the better one. who is more believable. see below: http://www.mozilla.org/security/phishing-test.html

  • Anonymous
    November 14, 2006
    The comments are equally interesting: http://www.haloscan.com/comments/paulr/2545/

  • Anonymous
    November 14, 2006
    It's great to see a competition between IE and FF about which product has a better phishing filter. Consumers will benefit from this as theses filters will become better and better. IE7 really is a lot better than IE6. I specially like the new printing engine, which finally prints out what I want. Great work!

  • Anonymous
    November 15, 2006
    It is not good that the Mozilla team has not checked for false positives in their tests or used any other product that did better in earlier tests then their product. It is much much more easy to catch phishing url's if you risk false positives.

  • Anonymous
    November 15, 2006
    I've been having a very close look at the raw data (unfortunately spending way too much time putting the data into chronological order) and concentrating on only those URLs where one browser detected the phish, and the other didn't - I've ignored all URLs where both detected, or both didn't. Overall things are pretty steady when it comes to hit/miss percentages between the two browsers, but I note that there are three days with extraordinary spikes in IE7 detection failures - almost as if the phishing filter was down or otherwise having issues - on 29 October, 3 November and 6 November.  The same spikes did not occur with FF.  I also note that from 3 October (when PhishTank's FF plug in was released) there was an almost complete drop-away in failures from FF (in fact, from 3 to 6 November inclusive, over which time 185 URLs were tested, FF only failed 3 times).   Considering PhishTank provided the URLs for testing, the marked drop off in FF failures from the day of their FF plug-in release is suspicious, to say the least. If we take away the three days mentioned above then the miss score drop from IE243:FF128 to IE107:FF102. I can't get past the fear that by not making sure that test URLs had not been submitted to Google during or before testing, that the tests are at best skewed, and at worst fatally flawed.  True, the URLS were downloaded every hour, and tested within 15 minutes, but that still allows up to 1 hour and 15 minutes between a phish being submitted to PhishTank and making it to the list - plenty of time for the PhishTank reporters, who in all likelihood use FF as their Web browser, to also submit the same URLs to Google via FF2's inbuilt Google based service. I'll be putting up a PDF of my checks to http://www.msmvps.com/spywaresucks in the next 24 hours or so. I was going to send it live tonight, but I see that FF says that there were 243 instances where Firefox blocked but IE did not, and that there were 117 instances where IE blocked but Firefox did not,  but my count makes it 243 and 128.  Because of this discrepancy I want to double check my numbers before going live.

  • Anonymous
    November 15, 2006
    > It is not good that the Mozilla team has not checked for false positives in their tests As far as I remember, both IE and the Google Toolbar had 0 false positives in the MS study, so I wouldn't say that's critical here. > plenty of time for the PhishTank reporters, who in all likelihood use FF as their Web browser How do you know that?

  • Anonymous
    November 15, 2006
    @Dao >>plenty of time for the PhishTank reportes, who in all likelihood use FF as their Web browser >How do you know that? Balance of probabilities.  PhishTank provides a FF add-in (and a Greasemonkey add-in for Opera), but nothing for IE.  To be honest, I'd never heard of the service before the study although I see there is a spike in publicity in the past week.

  • Anonymous
    November 15, 2006
    > PhishTank provides a FF add-in (and a Greasemonkey add-in for Opera) Well, go ahead and write one for IE, I guess they'll welcome you. Personally, I wouldn't know how to do that. For Opera and Firefox, you have to know JavaScript only. That could be a deciding factor for developing add-ons, but it doesn't mean that the whole community uses Firefox (heh, why would they support Opera? :). Note that you don't need an add-on in order to report phishing sites, but to automatically check against the list.

  • Anonymous
    November 15, 2006
    The comment has been removed

  • Anonymous
    November 15, 2006
    http://www.53.com.portal.ehasbee.jp/startproc.id/ gets blocked by FF, but not IE. I've reported it using the IE feature, let's see how long it takes to block this site.

  • Anonymous
    November 15, 2006
    @BidAl FF2 doesn't block http://www.53.com.portal.ehasbee.jp/startproc.id/

  • Anonymous
    November 15, 2006
    Lex Now FF2 Does block that site.

  • Anonymous
    November 16, 2006
    goose has lost his/her magic. Sounds like Fduch now.

  • Anonymous
    November 16, 2006
    Don't beat me to tell you do how you sound yourself.

  • Anonymous
    November 17, 2006
    ...I think that compertion between IE and FF will only benefit the consumer in the end. And that is a good thing Marc Liron - Microsoft MVP http://www.updatexp.com/ie7-issues.html

  • Anonymous
    November 21, 2006
    The comment has been removed

  • Anonymous
    November 26, 2006
    I turned mine off. Kept slowing down searching. I  do not feel like I have missed anything.