Cenzic 232 Patent
Paid Advertising
web application security lab

Larry Suto’s Paper Drama

If you don’t care about drama, skip this post, there isn’t any new information in it.

Somehow I always end up being the center of controversy, even when I’m really only vaguely interested in the subject matter at hand. This time it comes from the Full-Disclosure mailing list which is known for, among other things disclosing zero-day exploits in applications. My only problem with Full-Disclosure has been the noise, as it’s unmoderated and although humorously belligerent I generally don’t have the time to pay much attention to it anymore. Anyway, I’ve read in a few places now that people are concerned with Larry Suto’s paper on web scanning depth analysis.

First let me put some rumors to bed here. I am not paid by NTO to use their tool. They let me use it for testing purposes because they actually care about making their product better. I have given similar help to three other scanning vendors as well. This shouldn’t come as a surprise to anyone, as I’m part of the NIST.gov SAMATE Web Application Vulnerability Scanner Team and the WASC Web Application Security Scanner Evaluation Criteria Team even though Anurag keeps forgetting to put my name on the site.

NTO did pay for the banner on the top of the page, as did WhiteHat before and others will no doubt in the future. No conspiracy, we needed money to keep the site up and running back when we were completely self funding things and instead of paying everything out of pocket we opened the site up to some banners. I should hope everyone is well aware of it as a banner since it even says it’s a paid advertisement. After getting rid of Google and Omniture banners we decided to sell our own. It’s a nice way to keep it relevant and to reduce the amount of third party JavaScript on the site. If competitors want the space, they are welcome to it. If someone wants me to review their scanner publicly I’ll do that too. We have been asked to do this once before, although it hasn’t gone anywhere yet, I suspect because of the holiday season. No magic, folks. This is how the world works. People ask me to help out or write about something that actually interests me and I’ll do it, time permitting.

Also, Andre Gironda mis-read a quote from me regarding the tools I use for testing. The part he read was that I use NTOSpider. The part he either glossed over or failed to understand were these words:

A better question would be which ones don’t I use!

This is by no means an authoritative list of all the things I use in fact, I’ve written a number of tools that I don’t discuss, and it’s certainly not all the commercial scanners I have access to (most companies I deal with don’t want me to discuss my relationship with them for whatever reason - fair enough). Tin foil hat wearers beware - there is more to me than one or two scanners, and I would hope after the ungodly amount of times I’ve talked about it people would understand that I’m not really all that convinced scanners are all that great in the first place, except for automating some of the time consuming tasks (like crawling, for instance). Aside from a handful of commercial scanners, I have access to and use pretty much everything (and even if I haven’t had access to the scanner, I’ve probably seen the results from the scanners from various clients who sent me the results). None of it, in my mind, stands out as the hands down winner in all categories. Each have their good and bad parts. If people really want me to start doing reviews about why each are good/bad in their own rights, please send me a request to do so with access to whatever scanner you want me to test. I’ll be happy to oblige, time permitting.

Next, let me talk about the actual topic at hand. I was not involved in the technical aspects of how Larry Suto built his test environment. I was aware of the paper as it was being written, as were some of the scanning companies from what I was able to ascertain by talking to a few of them. That said, I didn’t question his methods - and in fact, I wasn’t that interested in them (I said as much in my post actually). I was and continue to be far more interested in the premise, rather than the result. Anything can be fixed, but it’s nice to have a baseline by which scanners can test themselves - whether they chose this particular environment, or another, is outside the scope of what I care about.

So let me reiterate because I think people really took this whole thing and blew it way way out of proportion. The part of Larry Suto’s paper that I thought was interesting was the concept of looking at how well a spider can crawl a site. He may not have done a great job setting up the test sites, you may question configurations, or the types of sites, or whatever you like - again, I’m not interested in that part… at all. What I am interested in is the concept - which is that if you cannot locate the page the exploit resides on, it doesn’t matter how good your exploitation engine is. Here’s what you should get out of that post, and nothing more: crawling effectiveness matters. How you chose to measure that is your own religion. I’m not saying the inverse isn’t true, EG: if you can’t exploit it, it doesn’t matter how good your crawler is. I’m just saying if you haven’t thought of the crawling depth metric you probably should.

Anyway, enough drama already! I’d suggest, for those of you who worry about alien abduction, if you have a problem with me, email me already, and I’ll try to quench the voices in your head. Lastly and most importantly, happy new years to those using the Gregorian calendaring system! :)

6 Responses to “Larry Suto’s Paper Drama”

  1. Ronald van den Heetkamp Says:

    Ah well, who reads newsgroups these days? Who’s got the time to read all the gossip and chatter. Newsgroups are like toilet paper factories for me ;)

    Happy new year also!

  2. Anurag Agarwal Says:

    >>even though Anurag keeps forgetting to put my name on the site.

    Stop busting my chops man!! What have I done to you? I didn’t even send the picture of you flirting with the waitress in the OWASP party to your girlfriend. In case you haven’t seen it, here is one of the many pictures.

    http://bp2.blogger.com/_p4tgkwtZjQ8/R0I0w5RnWFI/AAAAAAAAAKk/tgUxRO5Wqms/s1600-h/DSCN0358.JPG

    Oh i remember, I drove you and samy on the wrong side of the road after getting drunk. LOL. But you have to agree, that was fun. :)

    Your name has been on the site since the day 1 along with other people who are contributing. :)
    You can check it out at (in case you haven’t)

    http://www.myappsecurity.com/wassec/index.php5?title=Volunteers

  3. RSnake Says:

    @Anurag - Since when did my site become myspace? And I was not flirting with her, she was flirting with me. I was trying to eat (hence the food in my hand and mouth in the picture). Sheesh.

  4. Anurag Agarwal Says:

    stop making excuses. we all know what you were doing. Besides, I have witnesses. :)

  5. Ix Says:

    Hmm… sucks that you’re taking so much heat over this, as I personally can not agree with any of their points against you.

    Personally if I spend a lot of money on a scanner I’d expect some level of competent scanning without having to customize it every time, and I’d expect it to be near perfect if I do customize it. Free scanner, sure I’ll customize that, but if I paid for it I expect a level of functionality out of the box worth paying for.

    Looks to me like the people out against you are more afraid that they’ll be worthless if tools become more competent than anything else and that makes me wonder what they’re doing in this business if someone can make a program that can work as well as them.

    I have to agree with Ronald though, been a long time since I’ve seen a newsgroup that hasn’t been overtaken by people that don’t deserve to post on it and just fill it with worthless drama so that they can feel like they’re important. Sure there’s still some good people posting, but they seem to be outnumbered by the drama queens with too much time on their hands.

  6. Jeff Jones Says:

    Do you happen to know the current status of the SAMATE effort? Or the WASSEC project? I’ve been scouring the Internet trying to find something current, but it looks like both projects have not been updated for some time. Thanks for your time and consideration.
    Kind Regards,
    Jeff Jones