Cenzic 232 Patent
Paid Advertising
web application security lab

Web Application Scanning Depth Statistics

One of the most difficult aspects of web application security scanners is understanding how to evaluate them. Obviously the false positive false negative ratios are important, but it’s often difficult to measure, as it depends on the web application in question. However, Larry Suto came up with a very interesting concept on how to do unbiased measurements of web application scanners. One of the most important measurements is to understand how well the spider portion of the scanner works.

Think about it - if the scanner can’t actually reach a certain percentage of the application how is it going to find vulnerabilities in what it can’t reach? That’s the premise that Larry was working on. So he took three scanners, NTObjective’s NTOSpider, IBM/Watchfire’s AppScan and HP/SPI Dynamics’ WebInspect and ran them against several different vulnerable applications. He then measured them by looking at the number of links crawled, the coverage of the applications (using Fortify’s Tracer) and then he measured the false positive and false negative ratios.

NTObjective’s spider came in first with AppScan and WebInspect a distant second and third respectively - both in terms of coverage and false positives/negatives. Personally, I’m really more interested in the coverage though, because writing signatures is relatively easy once you have a good scanner to utilize them. Anyway, I believe this is one of the first truly unbiased ways to measure and compare a web application scanner’s performance. Larry’s web application scanner coverage paper and statistics can be found here.

36 Responses to “Web Application Scanning Depth Statistics”

  1. Jim Manico Says:

    In terms of coverage, can’t you easily teach these tools about all application endpoints fairly easily? One could conjecture that NTObjective’s NTOSpider has more default spider coverage, but AppScan and WebInspect are more capable of finding deeper vulnerabilities due to the maturity of the tools.

  2. Jim Manico Says:

    Just to clarify, if you take the link/coverage bonus away, I wonder if the vulnerability statistics would be so radically in favor of the NTOSpider.

  3. Drastic J Says:

    All web app scanners provide a tool for manual crawling (technically they are MITM proxies and scanners capture requests and responses). This obviously can crawl what you can crawl manually which means total support for Javascript, Flash, Java or any other 3rd party or challenged stuff for automated crawlers.

    Thus coverage can simply be perfect by manual support but when it comes to vulnerabilities it’s not so easy to write signatures for vulnerabilities. There are lots of issues that you need to consider like avoiding false positives and detecting strange XSS, different responses and conditions etc. So it’s not just regex foo :) (but a bit)

  4. Jim Manico Says:

    “That’s a possibility that the playing field could be leveled and that’s why I think code coverage is important…I mainly used the out of the box configuration for each scanner…its not clear to me yet how to configure a scanner to reach areas of the application that it was not able to cover initially…I think your point is important though and it should be followed up on for another round of study…it will be interesting to see how the vendors respond to that point.” - Larry Suto (Study Author)

    > If you level the playing field and teach each tool total app coverage,
    > does NTOSpiders’ vulnerability assessment still come way up on top?
    >
    > –
    > Best Regards,
    > Jim Manico

    PS: Thanks for giving me the OK to post this, Larry.

  5. Ory Says:

    I wonder why the results are so different than the ones published by Joran Wiens in his Web Application Scanner Rolling Review (http://www.networkcomputing.com/showArticle.jhtml?articleID=202200386).

    According to the whitepaper, the scanners were run using the “default mode”. This probably explains some of the differences - Not all applications are the same, and scanners require some configuration changes in order to work properly on different types of applications.

    -Ory

  6. fazed Says:

    @Ory - I guess this would lead onto anouther way
    of testing the scanners,
    how easy it would be for the average system admin
    to run it against their application and receive the best
    results. most sys admins would leave it in the default mode.

  7. Ory Says:

    @fazed - I don’t think that the “natural” target audience for web application scanners are sys admins (although I won’t dictate who should buy such tools).

    In order to achieve good results, web application scanners should be used in conjunction with manual security assessment, which requires close acquaintance with the web application and its different functionalities. While these tools automate the tedious job of assessing a web application for vulnerabilities, I wouldn’t recommend the “point & shoot” methodology for performing such assessments. Bottom line, these tools require a bit (seriously, not too much) of configuration tweaking, in some scenarios.

  8. Robert Says:

    My 2cents on this review.
    http://www.cgisecurity.com/2007/10/12

    I agree with Ory default configuration is a pretty inaccurate/misleading baseline.

    Note: I *used* to work for SPI.

  9. Adam Muntner Says:

    This test didn’t measure anything productive. These are complex, expensive applications. They aren’t meant to be used in default configurations. No one uses them that way and expects to get a Useful result.. A more realistic test would be to have someone knowledgeable with the tools configure them.

    Metrics are great - so long as they measure something meaningful. Otherwise it’s just GIGO.

  10. Richard Says:

    Right or wrong, the average person buying these may not be security professionals. I consult and find developers, sysops, and QA buying.

    These folks, and sometimes junior security people think you can just point and shoot. The only thing they can compart it to are network scanners. Unfortunately, that is how many of those get used, too.

    This means false positives that the operator doesn’t know how to validate. In the worst case, you get non-programmers looking at AppSec bugs and having no idea where to go or even what it all means beyond whatever description is produced by the AppScanner.

    Like it or not, the default crawler is important because many of these buyers/users don’t know what they are looking at, they probably don’t understand the entirety of their own application and most have no AppSec training.

    As a consultant, I have to deal with that reality and it flies up the OSI model to layer 9 (the political layer..just after L8 the fiscal layer) ;) really quickly. No, not ideal. Maybe not even right. Just an inconvenient truth.

  11. Sirw2p Says:

    Yeah, i think that it´s a good item to talk about. Many companys jsut uses this applications (web-scaners) for auditing their applications and nothing more, and they dont know that this isnt sufficient.

    Cases of false positive are very frequent using this tools..

  12. Kris Drent Says:

    Before you can debate the value of this kind of “metric” you would first need to answer the question of whether using these products as “point and shoot” tools should be considered as appropriate use. I don’t believe it is, and most of the appsec world would probably agree.
    As Ory and Adam already mentioned, the value of these tools are not necessarily realized in their “unconfigured” state. Applications are all very customized in everything from their navigation mechanisms, their session management, their logical requirements for passing authentication challenges, etc. It shouldn’t be surprising that running a tool that has not been configured to work well against applications with such individual requirements would result in lack-luster performance.
    While some vendors propagate this point-and-click nonsense with their sales and marketing tactics, most have been very direct that these are specialized tools that need to be configured for the target application framework and instance of application and they offer training and certification on the tool. They even offer training and certification on the tool.
    Certainly those of us in the AppSec industry know very clearly the limitations of these tools, and we know that they are severely limited without configuration. How many times have I seen a client point Web Inspect or App Scan at an app, and come up with a relatively clean report. Then I ask if they provided credentials or a way for the tool to log into the app, and stay logged in. The response is usually “Well, no…” The scan banged on the front door of the app for 2 mins and quit. This is not the scanner’s fault…
    So we should be clear about what is considered as “appropriate use” of these tools before choosing performance metrics. And we can’t damn the tool for not performing when we haven’t used the functionality it has, but hasn’t been configured to use.
    Finally, I’ve seen each of these tools have significant trouble identifying obvious vulnerabilities in various circumstances, even when the crawling went well. So, IMO, there is still a lot to be concerned with beyond the reach of a crawl.
    In the end, these are good tools. But we need to be certain we’re using the tools as their supposed to be used. And in the today’s modern web app world, there are way too many complexities to boil a reliable application-layer vulnerability scan down to a click…

  13. Larry Suto Says:

    I thought that I would wait until the end of the initial flurry to create a single response to some of the issues that have been raised:

    1) “can’t you easily teach these tools about all application endpoints fairly easily?”

    After some further thought, the answer is yes and no. It is fairly easy to enter links into an application. Here are the problems:
    a) applications have hundreds or thousands of links. There is no list of these lying around for security personnel to enter them. This is one of the reasons that you buy an automated tool.
    b) in many cases, you have to enter form data to get to another page. While all scanners have this capability, some do it better than others.

    c) hand crawling a website is an extremely tedious and time consuming process. While consultants may be comfortable spending hours doing this (and charging for it) security teams simply do not have the manpower to do it.

    2) “I wonder why the results are so different than the ones published by Joran Wiens in his Web Application Scanner Rolling Review”

    The results are completely consistent with those results; Appscan beat WebInspect in that study and it beat WebInspect in mine. The difference was that I evaluated NTOSpider as well and it consistantly out performed Appscan and WebInspect in my tests

    There are also significant methodological differences between the studies:
    a) Jordan only looked to test AJAX functionality for Ajax 1.0 applications

    b) Jordan did not methodically look at coverage but rather spot tested certain links.

    3) “In order to achieve good results, web application scanners should be used in conjunction with manual security assessment, which requires close acquaintance with the web application and its different functionalities. While these tools automate the tedious job of assessing a web application for vulnerabilities, I wouldn’t recommend the “point & shoot” methodology for performing such assessments.”

    Comments from current and former employees of the companies that didnt so as well, are admitting that their scanners are not designed to be run in isolation and should only be run by experts. While I would not disagree with that statement based on the results, I might take issue with his generalization to all scanners. While I did not exhaustively test every link, I did a review of the applications and I did not find any major functionality that NTOSpider missed. NTOSpider did the best job with just “point and shoot.”

    Automated tools should be able to crawl a site and test it for the vulnerabilities that it tests for. These tools should do the best they can on their own, and in my tests one scanner did better than the others. Now its certainly true that manual pen testing is also part of an in depth security review, and I am in no way stating otherwise.

    If an enterprise has resources to do a manual penetration test, manual penetration should complement automated tools, not serve as a crutch for their failure. Not all penetration tests are done with manual in addition to automated tools. Manual testing is expensive and far slower than automated testing. While a manual test by an expert will enhance security, this is not always possible. Many tests are done with only automated tools and they should be as accurate as possible.

    4) “one could conjecture that NTObjective’s NTOSpider has more default spider coverage, but AppScan and WebInspect are more capable of finding deeper vulnerabilities due to the maturity of the tools.”

    I find this comment to be most interesting. I tested three tools, one’s performance exceeded the other two, not only in coverage, but also in false positives - All of the tools allow users to customize the scan configuration to get better penetration. NTOSpider did better without special configuration. While we can certainly investigate whether this claim is true, I would think that it is presumptive to assume that AppScan and WebInspect are superior in this regard based on the evidence to date. Definitely more scans can be done and sure more advanced tuning can be performed in additional tests…but Spider has displayed to me a promising trend for good future performance regradless of the test criteria.

    As a final comment I want to reiterate - to RSnake’s point- the value of using code injection techniques to measure coverage of scanners…I think there is lot of value in actually having visibility into the actual apis the scanner invokes as well as the parameters and return values it receives…I think there is a lot of room for research in this area….there are dark areas in applications that no scanners reached…it is important to determine if there are vulnerablities in these areas and why the scanners are not reaching the areas

  14. Shagghie Says:

    Adam-
    I disagree…. I think there is too much dichotomy in this thread of comments in general, however. X% of users WILL use a default scan mode. But that is not the point. The point is that NTO Spider obviously has a better out of box configuration, and may also have better capabilities as well.
    As we all know, any web crawler’s true value is in it’s ability to crawl PAST areas of a site that other crawlers choke on. If Spider did that, then more power to those guys!

    So, Adam, when you posted…
    “No one uses them that way and expects to get a Useful result..”
    … it made me say to myself…
    “well wait, it looks to me that the results NTO got were MORE than USEFUL, imo!”

    For customers like the DoD, etc., I can’t imagine a better scenario than a tool that did some of the configuring and ‘intelligence’ for the organization that is guaranteed NOT to have expertise on board nine times out of ten.

    Anywho.. just thought I’d help us all out and regain some perspective…

  15. Laurens Greven Says:

    I have personally seen more positive falses than false positives. It wrong to rely on computers to take care for computers, a trained human eye is way harder to fool than any web-app.

  16. ntp Says:

    I thought that I would wait until the end of the initial flurry to create a single response to some of the issues that have been raised:

    while not in agreement with you, hopefully i can provide the necessary counterpoint without coming across like a jerk (which i don’t really intend to do)

    1) “can’t you easily teach these tools about all application endpoints fairly easily?”

    After some further thought, the answer is yes and no. It is fairly

    the answer is yes.

    easy to enter links into an application. Here are the problems:
    a) applications have hundreds or thousands of links. This is one of the reasons that you buy an automated tool.

    why would i spend $30k when i already have find(1) and locate(1)/updatedb? one could also use a professional crawler (usually cheap - and in the case of crawler.archive.org and httrack - free and open-source) to augment their test results.

    There is no list of these lying around for security personnel to enter them.

    unless the target website maintains a cms. or has done an assessment like this before. or has had a sqa team get this list. or the developers built a list. or marketing built a list. or IT built a list for software asset management. or you have 25 minutes and filesystem access. or a few hours and a proficient crawler.

    b) in many cases, you have to enter form data to get to another page. While all scanners have this capability, some do it better than others.

    one word: xpath. does your scanner support it? does your scanner include a domain-specific language to simplify scripting form data, etc? developers and sqa use xpath and DSL’s to work with their applications when testing. security testing is no different.

    c) hand crawling a website is an extremely tedious and time consuming process. While consultants may be comfortable spending hours doing this (and charging for it) security teams simply do not have the manpower to do it.

    internal security teams waste all of their time configuring and rebooting firewalls, tweaking security appliances, and updating their av. so get an application security team and give them the resources to do manual crawls using customizable web application security scanners.

    2) “I wonder why the results are so different than the ones published by Joran Wiens in his Web Application Scanner Rolling Review”

    my answer: because neither of your results match reality?

    The results are completely consistent with those results; Appscan beat WebInspect in that study and it beat WebInspect in mine. The difference was that I evaluated NTOSpider as well and it consistantly out performed Appscan and WebInspect in my tests

    if you had read the rolling reviews - you would note that appscan got the worst review of the top three, and probably rightfully so. NTOSpider? you must seriously be kidding, right? this is some sort of sick joke being played on the web application security community?

    There are also significant methodological differences between the studies:
    a) Jordan only looked to test AJAX functionality for Ajax 1.0 applications

    as opposed to?

    b) Jordan did not methodically look at coverage but rather spot tested certain links.

    you use this word, “coverage” a lot. i’m pretty sure you have no idea what it means. plus - you go between “surface coverage”, “application coverage”, and “code coverage” in your paper - as if they are the same things.

    do you understand how tracer works? do you understand how web application security scanners work? do you understand how automated software testing works?

    next set of questions: do we want to look at coverage vs. spot checks? which is a better test? what kind of coverage do we want to look at - surface, code, etc? what exactly are we measuring it and why? can a spot check test be a better test in some cases?

    3) “In order to achieve good results, web application scanners should be used in conjunction with manual security assessment

    automated software testing is anything but fully automated. arian evans recently used the words, “completely automated” on the webappsec mailing-list. this entire argument has brought up points about configured web application security scanners vs. “default-mode”.

    knowing this, i can tell you that manual assessments are now a thing of the past… these automated tools are capable of guiding and helping any web application vulnerability assessor who doesn’t feel like building his/her own equivalents by writing a crawler, scraper/parsing-engine, and dealing with all of the problems associated with quality and application security tools already in existence. so, in other words, these expert tools help experts (but they don’t do their job for them).

    Comments from current and former employees of the companies that didnt so as well, are admitting that their scanners are not designed to be run in isolation and should only be run by experts.

    oh you mean anybody at WhiteHat, Cenzic, IBM/Watchfire, and HP/SPI who actually has run their own tool before? i’m not talking about the sales guys or CTO’s. let’s name some names here - Arian Evans, Ory Segal, Billy Hoffman, Caleb Sima, a probably a few more [extremely technical] guys.

    so the people who built these tools and invented this industry have no idea what they are talking about? that’s good to know! thanks!

    assume that we need to build web application security scanners for the masses (KMFDM). can we at least build them in a way that teaches/educates up? i bet over 95% of people who say, “i’m a total idiot about web applications, i never want to learn anything about web applications - however - i need to measure the security of my web applications with a point-and-click default-mode interface and get a report” are all looking for one thing: compliance.

    yes, compliance. so let’s build scanners just for the compliance dorks, or at least separate the attacks that do compliance and pretty reports into one wizard of the tool, while the rest of tool can do it’s real stuff in expert mode (where the wizard sits between the chair and the keyboard) and submit security-related defects directly to an issue tracking system with a priority of ($).

    oh wait, some scanner vendors are already doing these things.

    everybody else wants to learn and become and expert, so that we don’t have this growing problem. that’s what owasp, lunch-n-learns, and cheap hacker con/events are all about! that’s what books, articles (print or web), and irc/forums/blogs/wikis are all about… cheap/free instructional capital.

    While I would not disagree with that statement based on the results, I might take issue with his generalization to all scanners. While I did not exhaustively test every link, I did a review of the applications and I did not find any major functionality that NTOSpider missed. NTOSpider did the best job with just “point and shoot.”

    let’s leave the pointing-and-clicking to pen-testers who deal with ballpoints and ink.

    btw - can you actually define what functionality you checked for? actually i’d like to see the entire data-driven set in a spreadsheet.

    Automated tools should be able to crawl a site and test it for the vulnerabilities that it tests for. These tools should do the best they can on their own, and in my tests one scanner did better than the others.

    in “your tests”. i’d prefer to see the results from the fortifysoftware people who wrote Tracer.

    4) “one could conjecture that NTObjective’s NTOSpider has more default spider coverage, but AppScan and WebInspect are more capable of finding deeper vulnerabilities due to the maturity of the tools.”

    I find this comment to be most interesting. I tested three tools, one’s performance exceeded the other two, not only in coverage, but also in false positives

    your false positives claim is also something i find lacking. did you just check to see if it was a 404, or did you explore the individual tests themselves and try to turn them into an exploitable aka true positive? the phrase, “false positive” is out of style… i prefer “non-exploitable” vs. “exploitable”.

    also - either you, i, or both of us have no idea what you mean by “coverage” in the sentence above.

    All of the tools allow users to customize the scan configuration to get better penetration

    yes, but do they support xpath?

    NTOSpider did better without special configuration. While we can certainly investigate whether this claim is true, I would think that it is presumptive to assume that AppScan and WebInspect are superior in this regard based on the evidence to date

    “based on the evidence to date”? so you’re referring only to your own test information (and possibly also that german paper)? all i have to say is - wait for the flood of information to get out…

    Definitely more scans can be done and sure more advanced tuning can be performed in additional tests…but Spider has displayed to me a promising trend for good future performance regradless of the test criteria

    seriously, man - you have to back up statements like this with statistics and facts that make sense. or at least some sort of technical jargon explaining why.

    e.g. because NTOSpider does depth-first search and contains an algorithm that does a pseudo-pass-run breath-first search to maintain a more accurate crawl… blahblahblahblahblah

    when you use the word, “trend”, that means that you have metrics to “trend” with. all it appears you have is a bunch of bad numbers.

    As a final comment I want to reiterate - to RSnake’s point- the value of using code injection techniques to measure coverage of scanners…I think there is lot of value in actually having visibility into the actual apis the scanner invokes as well as the parameters and return values it receives…I think there is a lot of room for research in this area….there are dark areas in applications that no scanners reached…it is important to determine if there are vulnerablities in these areas and why the scanners are not reaching the areas

    there also has been discussion about your paper (and how Tracer works by measuring code coverage) on the fuzzing-l, securitymetrics-l, and dailydave mailing-lists. the consensus is that code coverage is great only up to around 10% per tool, in which case it hardly matters other than that you hit the important cluster of code with “good tests”.

    it’s not the crawler or the default-mode, or how much code it covers - but it always goes back to the data-driven security tests (built into the tool) and the tester (driving the tool).

    unfortunately, the only areas of research I think are important here are to add fault-injection tracking and intelligent fault detection capabilities to our automated web application security fault-injection testing tools. of course, more protocol support and scripting support - as well as quality/quantity in data-driven tests (but these don’t change how the tools operate themselves).

    fault-injection tracking will rely on code coverage (and thus the bytecode injection pending lack of source code for java, managed runtime like the microsoft CLR, or even python/ruby/etc bytecode) - but it wouldn’t be just to get “visibility”. it would actually use the coverage to guide the tests, so a hit in a certain area would cause the tool to start testing around those areas of code. for examples of fuzzer trackers, see PaiMei or BinNavi.

    fault-injection tracking won’t really increase the number of vulnerabilities found (it may reduce false positives, though), but rather the time-between-finding (Jeremiah Grossman calls this “TTH, or Time to Hack”). this is because each tool will find all the bugs in one 10% code coverage cluster and usually no more… so once you’ve found them through dynamic analysis - you’re done. it’s some sort of natural limitation.

    for intelligent fault detection, you’ll want to add support for things like fault-injection stepping, which would take into account the order and operation of faults. beSTORM has done some of the best work for fuzzer intelligent fault detection… and they are working on fault-injection for web applications as well. beSTORM is smart in that they are moving beyond the typical scanner-supported protocols (HTTP and [drumroll]… SSL) and into more interesting areas such as proxying the file/SQL side (imagine SQLiX running against the equivalent of JDBCSpy… or DirBuster or DFF-Scanner running against the equivalent of FileMon). intelligent fault detection might be able to increase the capabilities of our dynamic analysis approach to finding false negatives, but to what degree it is uncertain. due to the nature of web applications, they don’t have the ability to do stack unwinding or advanced binary instrumentation… so freezing the application to find off-by-one errors isn’t as easy to do (which is how fat applications can find new weaknesses, or classes of vulnerabilities).

    this is what Brian Chess and Kureha (KJ) are trying to do with Tracer… turn it into an intelligent fault detection tool (particularly for SQL in the way I describe above). Dave Aitel was discussing similar ideas in a recent thread (related to your thread) on the dailydave mailing-list, but he’s looking to make it more towards a fault-injection tracker from what i understand. check out those resources, i’d be interested in hearing more about these subjects as nobody has really touched on any of them until this week.

  17. Adam Muntner Says:

    NTP has an uncanny way of reading my mind. To supplement his comments:

    Larry wrote - “I think there is lot of value in actually having visibility into the actual apis the scanner invokes as well as the parameters and return values it receives…”

    One of the things I really like about Hailstorm (which you didn’t review - and whose defaults are not useful - I sent them about 20 enhancement requests a few weeks ago on this subject) is that all the tests are written in .js. You can read the code to see what it’s doing and not doing. This is very useful for telling why it’s finding or not finding something.

    Another comment - We’re a services organization - we do assessments and teach developers to write secure code. That’s it. The only products we sell are assessment tools we use ourselves and know well. Our customers often want them to use in house. I’m all for it! When we sell a customer a $30k web app assessment tool, we train them to use it. That doesn’t mean, click the installer, reboot, and click start. It means, tuning it to their apps. This doesn’t take long to do on a per app basis. The results are a lot more meaningful, and help them to integrate the results into their SDLC. If you’re going to spend $$$ on a tool, you want to get value from it. Point and click doesn’t get you there.

    What’s just as important is, what happens after the app is tested? I’ve seen reports go to developers who say, OK, so there’s vulnerabilities. What is XSS and SQL injection anyway? How do I fix it? As they say, a fool with a tool is still a fool. If you don’t have the resources or expertise to validate false positives to save the developers time (which is usually at a premium) or don’t have the time to configure the tool properly in the 1st place, then you just wasted money on a tool because you don’t have the process or skillz in place to do anything useful with the results.

    NTP made great points about coverage. Surface area and depth are two different things. Dave Aitel and others made some great comments on the Daily Dave list about your paper btw.

    Your paper doesn’t let us tell what you really tested, or whether you interpreted the results correctly. Based on the paper and comments, not meaning to tase you here bro, my guess is you didn’t. A real research paper would have included appendices of the scanner reports, the code tested, etc. You didn’t provide any method for peer review - we’re supposed to take your word for it.

    “If an enterprise has resources to do a manual penetration test, manual penetration should complement automated tools, not serve as a crutch for their failure. Not all penetration tests are done with manual in addition to automated tools.”

    No automated web vulnerability scanners do ‘penetration tests’ in the sense that automated tools like CANVAS and Impact do at a different layer. They all do ‘vulnerability assessment.’ These words actually mean something and we do people a disservice when we misuse them.

    Furthermore hands on testing isn’t a crutch for the failure of automated tools, the way you mean. All automated tools inherently can and can’t find certain classes of flaws. I wrote about this a while ago on Security Catalyst in a guest blog post - http://www.securitycatalyst.com/2007/06/04/web-app-security-comparing-and-contrasting-black-box-white-box-fault-injection-and-sca/
    Business logic type flaws are an example of what automated scanners don’t find. Not wasting developer time and being able to support and coach them through the remediation process is a good reason to do hands on validation at least - otherwise you don’t know if they’ve really fixed it. Example - XSS fix that handles the webapp scanner test, but not filter evasion techniques. The scanner reports it’s fixed, but the hole is still there. What’s your coverage then?

    We could go on and on about this - but it’s not really a fruitful use of any of our time. I’m kinda surprised RSnake posted the article to be honest, though the discussion that came out of it I think will be very educational to a lot of people, so perhaps that was his intent. I don’t read minds.

    OK, so I’m going to channel NTP now. What he said is, “hopefully i can provide the necessary counterpoint without coming across like a jerk (which i don’t really intend to do)” (yeah right…. you”re our very own webapp Sacco and Vanzetti) - but what he meant to say is - the next article in this series will identify which commonly available hammers from Home Depot are best suited for driving woodscrews.

  18. MustLive Says:

    The coverage of the applications (by web app scanner) is very important measure.

    But, RSnake. As I see your site has UXSS vulnerability:

    XSS:

    http://ha.ckers.org/files/CoverageOfWebAppScanners.pdf#xss=javascript:alert(’xss’)

    http://ha.ckers.org/files/CoverageOfWebAppScanners.pdf#xss=javascript:alert(document.cookie)

    Which is not good. Better to fix it. I understand that you will not be attacked through this hole, but don’t forget about visitors of your site (which can use vulnerable Acrobat plugin).

  19. RSnake Says:

    I don’t really consider that a hole since almost no one who visits this site is vulnerable to it after the huge fiasco last year, but to make everyone feel better I zipped it. Next, when you find a vulnerability in zip that allows you to trick some windows 95 user into getting owned, I’ll take some more corrective measures. But really, is it worth even having this discussion? Yes, PDFs are vulnerable to some small segment of the population (of which the user community of this site doesn’t represent). Does it matter?

  20. Erik Says:

    Since this topic seems to be getting a little more interest again, I recomend that you also read the responses from HP and IBM regarding Suto’s paper.

    Analysis of Larry Suto’s comparative case study (by Jeff Forristal of HP Security Labs)
    http://portal.spidynamics.com/blogs/spilabs/archive/2007/11/12/Analysis-of-Larry-Suto_2700_s-comparative-case-study.aspx

    “Analyzing the Effectiveness and Coverage of Web Application Security Scanners” - Take II (By Ory Segal of Watchfire)
    http://blog.watchfire.com/wfblog/2007/12/analyzing-the-e.html

    Both of these articles add some additional color to this discussion that is worth reading.

  21. RSnake Says:

    @Erik - thanks for the links.

    FYI, for those who are interested in the results, I was told Larry Suto is going to be doing a second round of tests coming up to fix many/all of the issues brought up by those papers. Also he will be attempting to get more scanning vendors into the mix this time around. It will be interesting to see the results.

  22. Fraggeleh Says:

    http://blog.watchfire.com/BetterUntaughtThanIllTaught.pdf

    I cannot believe I spent my time reading that - what a pathetic defense, and the funny thing was that he only ran HIS product, and compared it to Suto’s results, which he stated repeatedly were questionable. When AppScan still turned up with a bunch of false negatives, he had the gall to go through, make changes to the scan and rescan the site…. 1. Suto did not reconfigure his scans for preferable use ie: If one product lacked the functionality to do its fucking job, he didn’t compensate by baby walking it through. 2. If he were running that on a massive company intranet/production site, having to re-scan because your program can’t tell the difference between errors and vulns simply doesn’t cut it - it’s a waste of time, resources and functionality in the product itself. That whole paper reeked of stupidity and bias, wafting from the direction of IBM. Oh and ntp, why begin your counter-point discussion by saying “I hope I don’t come accross as a Jerk”, and then question Larry’s credibility as to his results based on what? Your extensive bias toward more matured products? You scoff at NTOSpider, yet it fucks your pet products in the ass despite retesting from IBM’s side, and then demands the money back because they weren’t a good enough lay. Sorry though, I don’t want to come across as a jerk or anything..

  23. Rant Says:

    A study is only as good as its methodology. It’s as useless as reviewing the performance of 2 graphics cards on gaming PC with a pure vanilla windows installation.

    If you actually cared about gaming performance you would disable all of the bloated services and other junk that is defaulted. The results of the former are wildly inaccurate for a serious security software user.

    If you don’t care about performance of software, why publish a study comparing it?

  24. RSnake Says:

    @Rant - To answer your question I posted it for the same reason I described in this post: http://ha.ckers.org/blog/20080102/larry-sutos-paper-drama/ For people who aren’t interested in reading the whole thing, let me summarize:

    So let me reiterate because I think people really took this whole thing and blew it way way out of proportion. The part of Larry Suto’s paper that I thought was interesting was the concept of looking at how well a spider can crawl a site. He may not have done a great job setting up the test sites, you may question configurations, or the types of sites, or whatever you like - again, I’m not interested in that part… at all. What I am interested in is the concept - which is that if you cannot locate the page the exploit resides on, it doesn’t matter how good your exploitation engine is. Here’s what you should get out of that post, and nothing more: crawling effectiveness matters. How you chose to measure that is your own religion. I’m not saying the inverse isn’t true, EG: if you can’t exploit it, it doesn’t matter how good your crawler is. I’m just saying if you haven’t thought of the crawling depth metric you probably should.

    Enough drama. Don’t like the results? Post your own. Again (not sure how many times I have to say this, but I’ll try to type slowly for those who don’t care about or are incapable of reading all the words on this page and the other posts on the topic), I don’t actually care about the results - it’s the concept that’s interesting to me. Not. The. Results.

  25. none Says:

    Hi guys, i’ve develloped 2 scanners, one oriented audit intranet& web application , the other one is only web application.
    Can find most of vulnerability, the goal of this scanner was 0 false
    positive result.
    I wonder if you can make a website (because it will be unfair if i do it eheh) and then test every scanners on it, including mine.

    I guess it will be cool to compare the results =)

    Regards.

  26. tom brennan Says:

    Can’t wait to see Larry’s 2nd review at the 2008 OWASP APPSEC event in NYC Sept 24th and 25th

    http://www.owasp.org/index.php/OWASP_NYC_AppSec_2008_Conference

  27. DR Says:

    Attackers are well-aware of the valuable information accessible through Web applications, and
    their attempts to get at it are often unwittingly assisted by several important factors.
    Conscientious organizations carefully protect their perimeters with intrusion detection systems
    and firewalls, but these firewalls must keep ports 80 and 443 (SSL) open to conduct online
    business. These ports represent open doors to attackers, who have figured out thousands of
    ways to penetrate Web applications.
    The standard security measures for protecting network traffic, network firewalls and Intrusion
    Prevention Systems (IPS) and Intrusion Detection Systems (IDS), do not offer a solution to
    application level threats. Network firewalls are designed to secure the internal network
    perimeter, leaving organizations vulnerable to various application attacks.
    Intrusion Prevention and Detection Systems (IDS/IPS) do not provide thorough analysis of
    packet contents. Applications without an added layer of protection increase the risk of harmful
    attacks and extreme vulnerabilities.

    Web Application Level Attacks is the Achilles heel. In the past, security breaches occurred at the
    network level of the corporate systems. Today, hackers are manipulating web applications
    inside the corporate firewall. This entry enables them to access sensitive corporate and
    customer data. An experienced hacker can break into most commercial websites with even the
    smallest hole in a company’s website application code. These sophisticated attacks have
    become increasingly threatening to organizations.

    I recommend a service call GamaSec ( www.gamasec.com) remote online web vulnerability-assessment service
    that tests web servers, web-interfaced systems and web-based applications against thousands
    of known vulnerabilities with dynamic testing, and by simulating web-application attacks during
    online scanning. The service identifies security vulnerabilities and produces recommended
    solutions that can fix, or provide a viable workaround to the identified vulnerabilities

    www.gamasec.com

  28. none Says:

    I actually looked into online scanning service market and it seems to be picking up. Question is how valuable are they really ? Gamasec, Beyond Security service, Powerfuzzer online. Maybe it’s time to review them as well.

  29. Scott Says:

    I found this website that does some technology comparisons (well correction) comparision of application fw’s. It looks like you have the ability to select criteria of importance to get a more customized evaluation based on your needs. The url is www.itprocafe.com

  30. Security Expert Says:

    The best VA Solution is definitely ISC Technologies. They actually can bypass WAF - Signature based and network based techniques.

    They bypassed F5, Imperva, Appshield and modsecurity.

    Professional VA + Good WAF = Safe Web-App

    Just asked for first free check, and if you are lucky you will get one.

    Their Website is: www.ISC-Tech.com

  31. userX Says:

    With recent market shifts (IBM buying Ounce and HP partnering with Fortify) It can be fairly safe to assume a shift or inclusion of Static and Dynamic application security testing is occurring. However this still has just added to the need of some ability for users to know what they need to look at when considering to purchase a tool. (open source not a option). What are some of the top abilities users should be looking at when comparing tools? Auto Crawling ability, reports, false postives, attack vectors etc?

  32. Red36 Says:

    Reek and perlcritic are very similar, and yet so very different. ,

  33. ravi Says:

    i am work in captcha work

  34. Tom Brennan Says:

    This sounds like a perfect project for an Accredited University such as those listed at:
    http://www.owasp.org/index.php/Membership#Current_OWASP_Organization_Supporters_.26_Individual_Members

    to provide a research report around with controls and test subjects.. just sayin.

  35. Maximilian Corrientes Says:

    Hi,

    i know this arcticle is quite old, but it would be very nice, if anybody would confirm our results.

    http://labs.german-websecurity.com/en/blog/?p=12

    Usually online scanner won’t be compared with software scanners, but the result is very interesting.

  36. floyd Says:

    Today there is the WIVET project for the exactly same purpose:

    http://code.google.com/p/wivet/