Mozilla Says “Ten Fucking Days”
I’ll do a more thorough writeup of the craziness that is Blackhat, but this I thought should go out ahead of all the other stuff. I don’t have a lot of time so I’ll try to make this story short. Two days ago after Jeremiah and my talk (you can get the slides off of the WhiteHat site) a number of people from Mozilla came up and said they wanted to talk more about the issues we were finding and other suggestions we might have (I’m going to write this part up more thoroughly later in a separate post as well). We were also invited to the Mozilla “milk and cookies pajama party” which is pretty much exactly as it sounds.
We showed up, and nearly immediately I was surrounded by the bulk of the Mozilla QA and security team that was attending Blackhat. They asked me lots of questions, and gave me lots of info. It was a pretty equitable trade of information. Clearly, they acknowledge that they need help from the community but they also feel confident that once things come to their attention it’s simply a matter of days to close their holes. They said the recent rollouts were actually slower than they would have liked them to be, even though they were only a week and a half apart. Further, they said that they could roll out any critical patches within 10 days. Not one to let challenges go untested I called BS.
At this point Mike Shaver threw down the gauntlet. He gave me his business card with a hand written note on it, laying his claim on the line. The claim being - with responsible disclosure Mozilla can patch and deploy any critical severity holes within “Ten Fucking Days”:
I told him I would post his card - and he didn’t flinch. No, he wasn’t drunk. He’s serious. I’ve always been a fan of Mozilla and Firefox however this is a pretty bold claim for a company of any shape or size. I shopped the business card around to some various people while I was at the Microsoft party the next day to get people’s reaction. The consensus was that it was funny, very difficult to achieve and in one case, one of the head guys of security at Amazon simply doubted that the patches would be of sufficient quality. I’m not going to comment on my personal feelings on this matter except to say that I’d love to see Mozilla back up their promise.
August 3rd, 2007 at 10:27 am
What gets measured gets improved … of course, 10d > 0d.
August 3rd, 2007 at 11:27 am
They’ll pull it off.
August 3rd, 2007 at 11:45 am
This is almost meaningless…
Yes, I believe Mozilla could usually come up with a good, tested fix and release in 10 days. And they’ll do their best to come up with a fix as soon as possible.
However, bugs can be arbitrarily difficult to diagnose and fix. When you combine this with the issue being in code that is only understood by one or two people (some areas of the layout engine seem to be like this), and combine this with the fact that sometimes people happen to be unavailable to fix things due to vacations etc., it becomes clear that you simply cannot always roll a fix within an arbitrary, short time period.
Rather than get fixated at ten fucking days I’ll be confident knowing they’ll roll a fix as quickly as they can, which is usually very quickly indeed.
August 3rd, 2007 at 3:49 pm
Rome wasn’t build in one day, but heck, firefox isn’t rome. And Mozilla has ten whole days. I don’t know, put 20 geeks in front of a computer for ten days and just watch them go.
It could be done, depends on difficulty of the issues to fix though. I’d say give them ten days, if they patched, hail for mozilla, if they didn’t, laugh out loud, ask for cookies (and your money back).
You should’ve made a bet.
August 3rd, 2007 at 5:27 pm
and Bill say’s “one fucking service pack”
August 3rd, 2007 at 6:24 pm
I think ten years isn’t enough, look at the history. I don’t believe in “fixes” nor “patches” cause some where you’ll open up another hole. It’s hilarious, really. It’s okay to claim all that alleged FF security stuff, but I think Mozilla does a terrible job. I’m not ready to give into MSIE7.0, but I am close, because they have a few darn good features.
August 3rd, 2007 at 6:40 pm
Really hope that they can produce something ‘new’
August 3rd, 2007 at 8:00 pm
A release is much more than a guy banging out a patch and testing it. There’s a lot of time spent building and sanity-checking all the pieces that go into a release: 44 languages times 3 platforms is 132 packages to build and test, plus another 132 binary diff update packages and another 132 “full” update packages. And then all the update.rdf files served by aus.mozilla.org telling clients the update is available and where to get it.
Sure, lots of that is automated, but it still takes a lot of time and you never quite trust the process not to glitch somewhere.
For the Firefox 2.0.0.6 release there are 517 individual files hashed in http://releases.mozilla.org/pub/mozilla.org/firefox/releases/2.0.0.6/MD5SUMS
The equivalent Thunderbird file has another 417 (fewer supported languages)
August 3rd, 2007 at 9:50 pm
I think it can be done. I’m not a Mozilla fan boy but they seem to have the best patch system in the world going right now. I don’t doubt that they can pull of miracles under file.
Still, we won’t know until it is put to the test. Hopefully that kind of need won’t happen soon.
August 3rd, 2007 at 10:21 pm
two words - MEMORY LEAK
Maybe it isn’t a security problem but after 2+ years I think the 10 days has passed
August 3rd, 2007 at 10:26 pm
Two words - memory leak
August 4th, 2007 at 12:13 am
Meh. I’ve posted critical security bugs to Mozilla before, (like ones that will post information to the wrong sites) with exploits, just to have them downgrade them to non-critical, and not fix them for almost a year. Easy way to weasel out of them.
Anyway, when Mozilla has idiots like Darin Fisher who implement things like the PING attribute in the A tag that makes it easy for websites to track you against your will, and that *not a single user on the planet wants* it’s obvious that Mozilla doesn’t care for your privacy or security. Only when this PING tag is removed will I have any respect for them again.
August 4th, 2007 at 12:18 am
What was Mike Shaver’s promise? That Mozilla will fix security bugs you find within 10 days? That Mozilla will fix security bugs within 10 days, regardless of who reports it, if asked to do so by the reporter?
August 4th, 2007 at 1:09 am
“one fucking service pack”…in “half a fucking year.”
August 4th, 2007 at 2:09 am
Todays Tech August 4, 2007
New Wi-Fi network proves critical in Minneapolis bridge disaster A new Wi-Fi network in Minneapolis — only partially completed and just two months old — is nonetheless giving the city critical help in responding to this week’s collaps…
August 4th, 2007 at 3:01 am
I think the thing of note is the “responsible disclosure” this will rule out a lot of exploits found, as quite a few are just coded into a POC and then released into the wild and the vendor only knowing after the fact.
It will be interesting though when their claim becomes tested. I think ten days is a decent timeframe to aspire to but not realistic in all cases.
August 4th, 2007 at 4:39 am
Don’t know bout u - but I feel they are just inviting trouble with this claim…
August 4th, 2007 at 7:07 am
I’d believe it. The Firefox team has been very responsive to serious security holes, and has probably already done this 10 days thing several times. You could easily check by looking at the date a serious security issue was reported versus the date where there was a release with the fix. It wouldn’t always be 10 days, but it might be in some cases.
-Max
August 4th, 2007 at 7:53 am
It’s easy for me to kicm ‘em about it, I understand that before anyone else is going to say it, but if a moron like me can still make firefox stall on a simple thing like this:
function {
location . href = “uri”;
function call
}
I think that is something to be ashamed of, really.
August 4th, 2007 at 8:03 am
But, it goes way offtopic.
It’s about Mozilla fixing stuff in ten days, and they can do. Like we saw in the latetst update.
August 4th, 2007 at 8:12 am
is it called core dump, huh ???
-IT-
August 4th, 2007 at 8:22 am
Firefox used to be the shiznit. . . for a while. The last few “upgrades” have been wonky. I agree about the mem leak. Come on guys!
I’ll be back in “ten fucking days”. LOL //:
August 4th, 2007 at 8:31 am
Of course they can do it.
August 4th, 2007 at 9:06 am
2 words: memory leak
3 words: never had it
August 4th, 2007 at 9:18 am
When I did an analysis last year of how long IE and Firefox were vulnerable to unpatched exploits, it turned out that IE was open to known flaws for 284 days in 2006. Mozilla, on the other hand, had just nine days of exposure.
See:
http://blog.washingtonpost.com/securityfix/2007/01/internet_explorer_unsafe_for_2.html
August 4th, 2007 at 9:46 am
Oh wow you think your fuzzing tools will really pump out a bug so dangerous it can’t be fixed?
You’re not a real coder, you’re a fucking PHP HACK. Go run your metasploit some more times, I’m sure you can get that remote shellcode working.
August 4th, 2007 at 12:23 pm
@Deat by Fire - I’m not sure if you’re speaking to me or someone else, or why you have hostility, but nothing I find uses any automated tools. I’m not trying to beat Mozilla up over anything. What I find aren’t even really considered vulns (in the traditional sense). what I work on is primarily logic flaws and (mis-)use cases which is an interesting problem. The reason it’s interesting is that I don’t need traditional vulns (buffer over flows for example since you mentioned meta-sploit) to do what I do. The Google Desktop exploit I found, for instance, had nothing to do with anything an automated tool could find - it had to do with strange interactions between client side and server side and man in the middle code and forced browsing.
One of the weird things that came up after the talk is “Whats next?” Both Jeremiah and I are having a hard time even working with the browser guys in the same way more exploit authors do - because what we are finding is sort of systemic to the way browsers naturally work. Also, we’ve broken so much of the cross domain policy mantra that it’s not important to find vulns because it’s already so broken there is really less of a need.
The issues Jeremiah and I work on aren’t (for the most part) something you can simply do an additional string compare for and be on your way. They generally require re-thinks of how the architecture of some part of the system works. That’s why 10 days is a weird thing for us - while it may be a great line in the sand for traditional exploit writers who use the fuzzers and pre-canned exploit generation tools you’re talking about, it really can’t apply to most of the problems I work on.
So to answer your question - it’s not a matter of “so dangerous it can’t be fixed” it’s a matter of “so fundamental to how things work it’s a pain to fix”. Danger is all relative - you’re not going to own a box with the issues we find. However, you could write worms that infect millions of user’s webmail for instance.
As an easy example since it came up this week, the CSS history hack that Jeremiah found almost a year ago and that I re-wrote to not require JavaScript is still unfixed. The JavaScript version of that hack probably will be fixed soon according to Mike - but that’s not exactly ten days. But to his credit it’s not a “critical” vulnerability, so they are under less of an obligatory time crunch to fix - consumers just don’t care as much about those issues even though they do have a lot of power when combined with other attacks. I hope that’s more clear.
Simply put: for me, while 10 days is admirable - it’s almost completely non-applicable to everything Jeremiah and I have found. Not to say that Mozilla doesn’t have plans to fix them and not to say they don’t see a need to, they’re just harder to fix because it hurts the consumer almost as much as the exploit. Minimizing that impact is critical, and adds lots of time to the patches.
August 4th, 2007 at 2:13 pm
You should darken more his bunisess card…it’s hacked
August 4th, 2007 at 2:14 pm
– removed –
August 4th, 2007 at 4:11 pm
That sounds incredibly unprofessional if he mean’t it in the contact you are portraying.
Davide, nice image hack
August 4th, 2007 at 4:45 pm
Either way, I give them lots of respect for the attempt - even saying it shows they’re dedicated.
I won’t be one to eat them alive if a few of them take eleven. That’s still pretty damned impressive in my book.
August 5th, 2007 at 1:17 am
I have to totally agree with Chris D here. Mem leaks (which make me tear my hair out daily) and other suggestions aside - they have my respect for having the cojones to make this claim. The number “10″ is merely symbolic in my book.
August 5th, 2007 at 3:13 am
@David - nice one - I removed the link but very clever. Would you mind emailing me how you did it? I love that kind of stuff. I had to remove it because I’m trying to be nice to Mike. He didn’t say I had to white out the text, but I didn’t want him to get 10,000 calls on his cell either.
August 5th, 2007 at 10:40 am
Ahahah… great!
Mozilla Rox!
August 5th, 2007 at 5:07 pm
I’m not sure if David did it this way, but my guess is that it’s form the EXIF data. Check out this exif viewer.
August 5th, 2007 at 8:30 pm
That’s exactly right. Pretty cool hack actually. Probably very useful for more sensitive applications. I appreciate the link! He also emailed me offline with the same info.
August 6th, 2007 at 12:15 am
@RSnake: last time I checked your image still contained its EXIF thumbnail, while Jeremiah’s didn’t.
I friendly blogged about this difference tonight, and also attached a quick recipe for (batch) stripping away EXIF/IPTC metadata from JPEGs before publishing them: http://hackademix.net/2007/08/05/two-faces-same-card/
Just in case you really didn’t do this on purpose, and David didn’t already email you this info as well
August 6th, 2007 at 1:01 am
What info didn’t I email to him? About how to remove thumbnail?
I explained him to read this article http://exploit.blogosfere.it/2007/07/quando-le-immagini-non-mostrano-verita.html
or http://no.spam.ee/~tonu/exif/ for better understanding but I did’t receive answer to continue. But is the image with exif metadata still here?
August 6th, 2007 at 8:11 am
I know this is going to seem the thickest thing ever posted here, but what exactly is memory leak and why on earth do people view it as such a big problem?
August 6th, 2007 at 8:33 am
Excellent stuff Giorgio, I figure I can do the same in photoshop which I normally use to view watermarks & other metadata
August 6th, 2007 at 8:39 am
It seems Photoshop does this automatically when you save for the web inside photoshop. It’s my favorite app, so that is nice to know
August 6th, 2007 at 10:55 am
(I thought I commented here on Friday, but I was working from my Blackberry, which is not especially web-friendly. Bleh.)
Glad you enjoyed the party, Robert. To clarify, I was making a personal commitment, not a Mozilla one, that you could redeem that card if there was a vulnerability that you believed needed to be turned around in 10 days. I didn’t consider at the time that it would be taken as a Mozilla policy statement — even *I* don’t make new policy announcements at late-night parties in Vegas
— but it seems to have been read that way, which I can understand in hindsight. I’m sure I’ll be answering for my potty mouth and apparent lack of clarity for a while…
August 6th, 2007 at 11:41 am
@Giorgio - good writeup! But Jeremiah has suffered the same flaw - I’m not sure what you mean: http://regex.info/exif.cgi?dummy=on&url=http%3A%2F%2Fbp3.blogger.com%2F_JdybrokZBAk%2FRrY4PBXLuzI%2FAAAAAAAAAkY%2FRgil8CwLI8M%2Fs1600%2Ften-fucking-days.jpg
I think you were looking at the wrong URL. Anyway - I’ve cleaned mine up - no, I wasn’t trying to be evil. As I said in the post I was in a big hurry as I wrote it trying to get back to DefCon. Why does everyone think I’m out to screw Mozilla? Alas - browser companies are my friend!
@Mike - yes, it was really nice meeting you and I owe you a few emails and follow up posts. Honestly, I think while not policy, it shows confidence in your ability to do the right thing. It may have been perceived badly by few, but I think most of the people who matter get it. I saw Window’s response as well: http://blog.mozilla.com/security/2007/08/06/mike-shaver-ten-days-and-expletives/
I know full well it isn’t Mozilla’s actual policy - but if nothing else, it shows people you and your team take this very seriously. In my mind this is nothing but a kick in the butt in the right direction. I wouldn’t have removed the expletive for anything.
There probably is still an interesting follow-up conversation which I think we talked about briefly. That being how do we define open source when bugs are hidden from view? To be more clear: is it better to allow people to try to patch themselves or to allow or obfuscate the issues until a patch is ready for everyone - thereby delaying the patching for those who may be able to do it themselves? Clearly I can reverse engineer fixes, but that essentially just changes the problem from everyone knowing the problem to just a select few determined people. It’s a tough problem, and not one I expect to get “answered” but it is an interesting discussion.
August 6th, 2007 at 1:51 pm
@RSnake:
My fault, I didn’t notice the picture inlined in Jeremiah’s post was just an incidentally washed up (by Photoshop’s “Save for Web”?) thumbnail:
http://regex.info/exif.cgi?dummy=on&url=http://bp3.blogger.com/_JdybrokZBAk/RrY4PBXLuzI/AAAAAAAAAkY/Rgil8CwLI8M/s200/ten-fucking-days.jpg
As you noticed, it is linked to your revealing original
@Ronald:
Yes, Photoshop’s “Save for web” can strip out some metadata (it’s probably what happened with Jeremiah’s thumbnail), but every time you use that function you encode the JPEG again, losing quality.
IrfanView method is lossless, allows batch processing and is affordable even for the not so rich hackers
August 6th, 2007 at 2:30 pm
Cool stuff, I didn’t know that. And I’m into graphis design all my life
As far as the discussion, I can see the humor in the business card. If I would work for Mozilla, I probably would do the same at that particular moment. Let’s not wash out the circumstances it was presented
IMHO, I still think Mozilla can do a better job. When they rely on types like us, and highly skilled programmers and hackers I really think I may question that. Most bugs are things that could have been anticipated on, and would have been found with better testing & fuzzing. Not only that, I think certain security design principles are just denied only to comply with usability and the ease of using Firefox.
And why all the quick media releases, like sitting in the front seat before microsoft does? Is Google pushing you guys? why the competition anyways, in the end it’s about the end users and not about a battle of intellects. Everyone with a few braincells likes Firefox more then MSIE and that will never change, so relax
August 6th, 2007 at 3:01 pm
Interesting methodology, however.
August 7th, 2007 at 1:32 am
It seems Shaver pledge hasn’t even lasted 10 days.
Mozilla: 10-day patch guarantee ‘not our policy’
http://www.infoworld.com/article/07/08/06/Mozilla-10-day-patch-guarantee-not-our-policy_1.html
“The open source browser maker was forced to issue a statement Monday, retracting a pledge attributed to the company’s director of ecosystem development, Mike Schaver, to fix any critical security bugs in the browser within “Ten ****ing Days.”
August 7th, 2007 at 4:24 am
As he said “it is not and never was Firefox’s policy”, It was his commitment to fix any bug that RSnake thought needed urgent attention.
RSnake and Jeremiah should be chuffed, not many security professionals (even the other major corps) would get an offer like that.
It’s great to see a more solid working relationship between vendor and consultant - rather than them thinking we are the all on the “evil” dark side, rather than what most of us really do.
Walk a line on the grey side of life.
August 7th, 2007 at 10:38 am
A few more fucking days, then?
August 7th, 2007 at 1:13 pm
He couldn’t be drunk, wasn’t the Mozilla party dry?
Personally, I think Shaver might have gone a little too far in attempting to prove a worthy point. Mozilla is serious about fixing security issues and would like to work with researchers on these issues.
Unlike other vendors, they are not going to throw out blanket security responses (looking at you Apple) and hope to improve Firefox with the help of researchers that report vulnerabilities.
Ten days or not, it is a good stance to work toward.
August 7th, 2007 at 4:08 pm
fortunately you were allowed to publish a scan of the card on your blog. somehow it’s a funny contemporary document.
August 7th, 2007 at 6:55 pm
Hey, I could fix almost any bug in ten fucking days, but if the bug was anything more than a typo, I would worry that I had introduced something worse.
Because if I had failed to test on a wide enough array of test systems, I would have increased the risk that thousands of customers would be calling my support team on the phone first thing tomorrow morning. And then I’d have some ’splainin’ to do.
August 7th, 2007 at 6:58 pm
This is incredibly unprofessional.
August 8th, 2007 at 12:59 am
If only they’d get around to fixing the memory leak I might actually start using their bloated heap of junk again…
August 8th, 2007 at 6:32 am
Lets see what happens and keep us updated.
August 8th, 2007 at 8:16 am
When you get to be old and gray and have been jumped on repeatedly you tend to back away from making objective claims. It also starts to sink in that that the reward for actually doing something is frequently outweighed by the pain and aggravation you get for for doing it.
Then you go and work with ISO …
My personal thanks to the folks who encouraged more carefully considered remarks, and to the folks who find the holes and to the folks that fix them, 10 days or not.
And to “Corporate America”, thanks for almost nothing.
August 14th, 2007 at 11:30 pm
I think it will easily get done as he is saying. This is Mozilla mind you and not Microsoft!
Atleast even if they get the job done in 11 days or 12 days, doesn’t matter. They are faster than the rest at the least.
August 16th, 2007 at 2:06 pm
This is FUCKING AWESOME!
i like it director of an company goes in this ways.
i would like to have this “Ten Fucking Days” Card.
This rocks so much. Mozilla rocks. TEN FUCKING DAYS.
(P.s.: I can make a girl pregnant within THEN FUCKING DAYS!)