One of the most difficult aspects of web application security scanners is understanding how to evaluate them. Obviously the false positive false negative ratios are important, but it’s often difficult to measure, as it depends on the web application in question. However, Larry Suto came up with a very interesting concept on how to do unbiased measurements of web application scanners. One of the most important measurements is to understand how well the spider portion of the scanner works.
Think about it - if the scanner can’t actually reach a certain percentage of the application how is it going to find vulnerabilities in what it can’t reach? That’s the premise that Larry was working on. So he took three scanners, NTObjective’s NTOSpider, IBM/Watchfire’s AppScan and HP/SPI Dynamics’ WebInspect and ran them against several different vulnerable applications. He then measured them by looking at the number of links crawled, the coverage of the applications (using Fortify’s Tracer) and then he measured the false positive and false negative ratios.
NTObjective’s spider came in first with AppScan and WebInspect a distant second and third respectively - both in terms of coverage and false positives/negatives. Personally, I’m really more interested in the coverage though, because writing signatures is relatively easy once you have a good scanner to utilize them. Anyway, I believe this is one of the first truly unbiased ways to measure and compare a web application scanner’s performance. Larry’s web application scanner coverage paper and statistics can be found here.