Cenzic 232 Patent
Paid Advertising
web application security lab

Releases.mozilla.org SSL and Manual Update Fail

I did a presentation at the DefCon Comedy Jam about how users manually validate updates for Firefox was just a total mess from a security perspective. It had a lot to do with the fact that they are using round robin DNS and relying on the good will of a lot of dispirit sites to do their hosting for them. Well, I’ve been wondering more and more about how I may or may not be able to download these releases and verify them manually in a secure way. So I did a few checks and here’s the IP space I found and the status of what their SSL/TLS ports were when checked yesterday:

63.245.208.152 (live but with SSL/TLS mismatch)
128.61.111.9 (connection refused)
129.101.198.59 (connection refused)
131.188.12.212 (connection refused)
149.20.20.5 (connection refused)
156.56.247.196 (connection refused)
202.177.202.154 (connection refused)
204.152.184.196 (connection refused)
204.246.0.136 (connection refused)
216.165.129.141 (connection refused)
64.50.236.214 (connection refused)
64.50.236.52 (connection refused)
129.101.198.59 (operation timed out)
155.98.64.83 (operation timed out)

So only 1 out of 14 even have SSL enabled. Okay… well today, I took a little spin on SSLLabs and I found that the one site that does have SSL enabled (63.245.208.152) has a SSL/TLS mismatch error for videos.mozilla.org. I mean… seriously! If your browser goes to https://releases.mozilla.org this is sort of what’s happening under the hood:

$ telnet releases.mozilla.org 443
Trying 202.177.202.154…
telnet: connect to address 202.177.202.154: Connection refused
Trying 204.152.184.196…
telnet: connect to address 204.152.184.196: Connection refused
Trying 204.246.0.136…
telnet: connect to address 204.246.0.136: Connection refused
Trying 216.165.129.141…
telnet: connect to address 216.165.129.141: Connection refused
Trying 64.50.236.214…
telnet: connect to address 64.50.236.214: Connection refused
Trying 128.61.111.9…
telnet: connect to address 128.61.111.9: Connection refused
Trying 129.101.198.59…
telnet: connect to address 129.101.198.59: Connection refused
Trying 131.188.12.212…
telnet: connect to address 131.188.12.212: Connection refused
Trying 149.20.20.5…
telnet: connect to address 149.20.20.5: Connection refused
Trying 155.98.64.83…
telnet: connect to address 155.98.64.83: Operation timed out
Trying 156.56.247.196…
telnet: connect to address 156.56.247.196: Connection refused
Trying 63.245.208.152…
Connected to releases.geo.mozilla.com.
Escape character is ‘^]’.
^]
telnet> quit

Yes, after a minute of trying your browser will eventually find the one HTTPS server - or it won’t (sometimes it just gives up). So then in poking around within my Mozilla config I saw a reference to http://en-US.www.mozilla.com/en-US/firefox/3.5.7/releasenotes/. So I switched to SSL/TLS with this link, because I like being secure, and I get a SSL/TLS warning as well. These are the kinds of things that make Firefox incredibly unsafe to manually download and verify the binaries if you are in a hostile environment. And I’m just scratching the surface compared to my presentation. How many of those 3rd party sites do you think can be exploited?

39 Responses to “Releases.mozilla.org SSL and Manual Update Fail”

  1. Wladimir Palant Says:

    Robert, while SSL is nice to have getting together a mirror network supporting SSL is a real pain (I am learning that the hard way right now). And with the download amounts of Mozilla this is hardly possible at all. Which is exactly why all releases and updates are signed…

  2. RSnake Says:

    Yes, and now you have to trust that their signature is valid and that the checksum matches… the MD5s are served up over HTTP. Sorry, this is just a horrible design. Hard or not, it’s bad security.

  3. bob fa Says:

    I am not convinced. I am under the impression that updates are signed and the information is sent over https / verifiable therefore a https connection for the file to download is not required.

  4. RSnake Says:

    @bob fa - I inadvertently deleted your last post - please re-post if you would with the link.

  5. RSnake Says:

    @bob fa - I think you’re also missing some context. Sometimes the update mechanism fails, and I’ve had to do it by hand (and in hostile environments). In which case, I can’t rely on anything in the browser to save me. That’s what the whole premise of the speech was.

  6. Skuld Says:

    Aren’t the executables code signed though?

  7. Mike Shaver Says:

    I’m not sure what the update fail is: the hashes for the updates are served over HTTPS, so there’s an intact chain of trust. Can you elaborate on how “the update system for Firefox [is] just a total mess”? It doesn’t look like you’re describing what our update system actually does, in this post.

    If the update mechanism is failing, please do report a bug (or just email me if you can’t be bothered with bugzilla).

    For installing from scratch rather than updating, our Windows binaries are signed, which should give you what you need; our Mac installers are still not, unfortunately, so you have to use the PGP signatures. Linux distributors deal in signed packages as well.

  8. RSnake Says:

    @Mike - please talk with Dan Veditz - he has more context, and the full powerpoint I did at DefCon that does a much more thorough job of explaining everything than I could do here. This post is meant to compliment that speech, but if you haven’t seen it, it’s not going to make much sense.

  9. Mike Shaver Says:

    Then what *is* the point of this post? Just to indicate that some of our sites aren’t set up for SSL? You seem to be implying that there is some security issue with our update system (really, you’re coming right out and saying it!), so I think it’s reasonable to ask for you to explain the reasoning behind that position.

    Can you provide the Powerpoint? I would like to understand how https://releases.mozilla.org is relevant in some way to the Firefox update system, I admit!

  10. Gavin Sharp Says:

    releases.mozilla.org isn’t used for client updates at all. The update connection is made over HTTPS to aus2.mozilla.org, which provides a SHA-256 hash of the update file and a link to Bouncer, our mirror management app (download.mozilla.org). Bouncer redirects to one of our mirrors (plain HTTP). The updater downloads the update file and checks the integrity of the update using the hash obtained via SSL. Please stop spreading misinformation about Firefox’s update security :)

  11. RSnake Says:

    The point of this post is to further demonstrate that when (not if) the update mechanism fails (because it most certainly has in my case), Mozilla makes it painful to download the bits and then verify that they are in fact secure. It’s exactly the point that releases.mozilla.org isn’t SSL enabled that is the problem since that’s (apparently) where the MD5 and SHA1 hashes are stored (sometimes).

    I uploaded the PowerPoint for you here: http://ha.ckers.org/files/ff.ppt

  12. RSnake Says:

    @Gavin - where are you SHA1 and MD5s stored for the public? If they have a point at all they should be over SSL, correct? Please look at the powerpoint or talk to Dan. It’s not mis-information, you are just missing context.

  13. Mike Shaver Says:

    Thanks — it was “make Firefox incredibly unsafe to download *and update*” (emphasis mine) and the title of the post that made me believe that the examples were relevant to the way the update system is structured.

    I don’t think that checking the signature on the Windows installer is incredibly difficult, by “verify signatures without being able to provide your own tools” standards. Do you have an example of a better practice here for Windows installation?

    releases.mozilla.org being https wouldn’t give you the assurance you want, since that’s a mirror network including commodity CDN. We could provide the hashes for the installers on something linked from our download pages via SSL, but if you’re on Windows you really just want to check the binary signature anyway, no?

  14. Mike Shaver Says:

    (I think that when you say “the update system” in sentence 1, and “these releases” in sentence 3, people are going to believe that you’re talking about the update system still; I did, and I think Gavin did as well. If you just remove the references to the update system from your post, I suspect it’s a lot clearer what you’re talking about — unassisted downloads starting from our web site — and what you’re not — the application update system.)

  15. RSnake Says:

    @Mike - agreed, and I did change it to be more clear.

    I guess my biggest problem here is that the initial download is over HTTP and then if the update mechanism fails, a user may try to download it again by hand (as I had to several times). Other (less sophisticated users) are going to have a really tough time in a hostile environment because signatures mean nothing. SSL is their primary recourse - albeit a bad one, given what gets allowed into the cert stores these days, but that’s a whole other (huge) issue.

  16. Gavin Sharp Says:

    Here’s an example: https://aus2.mozilla.org/update/3/Firefox/3.5.7/20091221151141/Darwin_Universal-gcc3/en-US/release/Darwin%209.8.0/default/default/update.xml?force=1

    (I was wrong about SHA-256, looks like it’s actually SHA512)

  17. Mike Shaver Says:

    Do you mean that the signatures on the Windows binaries mean nothing? I think that’s the most useful thing for the user, actually, because it’s about what’s actually on their system about to be installed, rather than part of a transitive-trust chain they have to reason about from CA to browser to them.

  18. Gavin Sharp Says:

    Sorry, I hadn’t seen the latest comments before posting a second time. Thanks for updating the post. It still references “updates” without specifying that you mean “manually downloaded and installed updates”, so it’s still somewhat misleading, I think.

  19. RSnake Says:

    @Mike - possibly… I’m not convinced that ignoring SSL is wise though. Self signed executable that claim to be from “Mozilla corp” are getting more common - was even part of Jabra’s part of the Unmasking You preso. Albeit it was a Java app but that kind of thing would fool almost anyone who wasn’t paying attention (even a lot of security people). At least with SSL you have to either downgrade (which at this point the way your sites are built isn’t necessary) or you have to find another issue with the SSL chain or SSL itself. Not that I like SSL mind you - I think it’s got a lot of failings. But it’s all we’ve got until something like DNSSEC + some sort of browser add-on comes into play.

    @Gavin I searched and didn’t see the word “updates” once in my post. Which part is still unclear? Regarding the link - where did you find that? It it something that’s linked to that a user might actually find? I tried directory transversal and got a mixed bag of XML so it’s not like that’s a site someone would normally visit or something…?

  20. RSnake Says:

    @Gavin - ah, I think you meant the title - I modified that as well.

  21. Mike Shaver Says:

    I don’t think that the Windows setup will present the name claimed by the cert if it couldn’t verify it — does Java really do that? Wild.

    I agree that a complete SSL chain helps (modulo mirror/CDN concerns), but I don’t think that anyone is going to actually verify that chain for their download, since they need to make sure that there’s nothing dumb like an https -> http -> https redirect sequence that renders the whole thing useless. For people who are being careful, I still maintain that the signed Windows binaries are the strongest (easiest to be confident in) thing we can offer to people who are being cautious.

    I think Gavin means “make Firefox incredibly unsafe to download and update” in the last section.

  22. Gavin Sharp Says:

    The link I posted was for our update system and isn’t typically exposed to users - I was still under the impression you were talking about our updater rather than manual downloads, so feel free to ignore it.

    I think it’s irresponsible to publish a post whose title is roughly “[mozilla] SSL and update fail” that’s actually only about manual downloads. The vast majority of our users use our secure update system to get their updates, so mentioning “updates” at all (as in “Firefox [is] incredibly unsafe to download and update”) is misleading at best.

    As for solving the security of manual downloads, I’m not sure exactly what you think we should do. Just providing hashes over SSL is easy, of course, but also not very useful - the number of downloaders who would manually verify a hash is insignificant (and those people could typically figure out how to otherwise get software securely anyways). We could expose a full-https manual download option, but that’s places a large burden on us (can’t use the mirror network) and doesn’t help people who visit “http://mozilla.com” and get MiTMed before they even reach our “Download over SSL” button. Might be worth doing anyways, I guess.

  23. RSnake Says:

    @Mike - the funny thing is that is exactly what I was doing. I was monitoring all the traffic trying to find both a safe place to download it from (SSL/TLS enabled website) as well as a hash to verify that the download I received was in fact at least valid. But neither of those things really matter if they’re being downloaded from some random web server that you don’t control as they could be compromised. So, to your point, SSL is useless in that environment. So what I propose is that you (I can already feel the arguments coming) should not use third parties for any part of your binary downloads. The whole thing should be owned and operated by Mozilla. Self signed does present a warning for both Java and .exe but in Java it says the name. In either case it says the filename, and both are extremely easy for people to click through. Even a fairly skilled user is not safe in that environment where they can be downgraded - because they can’t even manually type in https on most of the site and get a secured version without SSL/TLS errors. The https://en-us.www.mozilla.com/en-US/firefox/3.5.7/releasenotes/ is a great example.

    @Gavin - it’s changed. Sorry if that was confusing. Again, this all makes a lot more sense if you’d actually attended the presentation.

  24. Daniel Veditz Says:

    Our SHA1HASH files _are_ available securely (and I’ve given you the link in the past) but they’re supremely useless to the vast majority of our users. Here’s one:
    https://ftp.mozilla.org/pub/mozilla.org/firefox/releases/3.6/SHA1SUMS

    The installers are also signed with detached PGP signatures, which should be just as good for anyone capable of checking sha1 hashes.

  25. RSnake Says:

    @Daniel - agreed - you HAVE given them to me in the past, but that’s just me. There are millions of users out there who aren’t me. The PGP sigs on the site were out of date when I checked before DefCon. I haven’t checked since then. But they weren’t signed by anyone at Mozilla anyway, so that wasn’t much help.

  26. Daniel Veditz Says:

    Oh, and for people paranoid about whether they’ve actually reached the real Mozilla site, it, too, it available through SSL

    https://www.mozilla.com/

  27. RSnake Says:

    @Daniel did you actually click on any of those links from https://www.mozilla.org/ ? They all take you to HTTP sites…

  28. Mike Shaver Says:

    I agree that people *can* click through it, but I think you’ve shifted from “in a hostile environment, it is very hard to verify” to “many people won’t bother to verify…and probably don’t know that all their environments are hostile anyway”. It’s easy to click through SSL-mismatch errors in many browsers too, so we’re back in the same boat.

    I think that someone who wants to verify the binary can do that reliably on Windows, and unfortunately not on Linux or Mac very well today (there’s an open bug, which I will poke).

    I agree that we should (always, always) be looking for ways to reduce both the number of places that are vulnerable to subversion, and the amount that users have to go out of their way to verify that they’re getting the right thing. There are a number of things we can do towards that end, and we’re working on some of them already.

  29. Mike Shaver Says:

    RSnake: the SHA1SUMS are delivered via SSL, which is what you’re saying we should do, no? If they were signed, they wouldn’t need to be over SSL, I think we can all agree, but that brings us back to the fact that the Windows binaries *are* signed already!

    I think it’s a little disingenuous to say “well, *I* have them, but others don’t” when you’ve built a post around randomly tweaking URLs. But then, if your post had just been “Mozilla should publicize better how to securely get a hash for their downloads”, I guess it wouldn’t have been quite as exciting to read. :)

  30. RSnake Says:

    @Mike - I concur - I did change the topic slightly. But that’s real life - you could easily be presented with both issues at the same time. One user might not click through the SSL warning the other might not click through the executable warning. I think they’re both issues. To your second point, I actually lost Daniel’s email and then I went searching for it myself. I spent about half a day looking for it. I searched everywhere I could - all over your site, and the Internet (lots of those screenshots actually). It’s like you intentionally hid it. If I can’t find it without happening to know you guys and having a personal relationship with Daniel, that’s not exactly a success story. But that’s not the only place you show hashes either. You can see in the screenshots there are definitely places you show it that aren’t SSL enabled. Either it’s a security mechanism or it’s not. If it is, it should be over SSL, if it’s not, then why bother having it?

    @Daniel - I just checked PGP and it appears it’s still not signed by anyone @Mozilla.org : http://pgp.mit.edu:11371/pks/lookup?op=vindex&search=0×74474499812347DD

  31. RSnake Says:

    @Mike - if you look at http://ftp.mozilla.org/pub/mozilla.org/firefox/ and click on “releases” it takes you to “releases.mozilla.org” so it would extremely unlikely that anyone would be able to find your hashes on a) a protocol you don’t advertise or link to and b) on a site that links you off to other sites when you click on the only important link out of the three available links. And if you look at http://www.mozilla.org/community/mirrors.html it really doesn’t give the user the clue that https://ftp even exists.

  32. RSnake Says:

    To me this is scary… http://ftp.mozilla.org/pub/mozilla.org/firefox/releases/3.6/SHA1SUMS automatically immediately redirects you to http://releases.mozilla.org/pub/mozilla.org/firefox/releases/3.6/SHA1SUMS not https://ftp.mozilla.org/pub/mozilla.org/firefox/releases/3.6/SHA1SUMS ….

  33. Daniel Veditz Says:

    Yes, anything http://ftp.mozilla.org/pub/mozilla.org/firefox/releases/ redirects to releases.mozilla.org. But that’s OK because the SHA1SUMS file is signed, too.

    But if you’re talking about the windows installers then the signature on those binaries should be all that matters. Why is that not good enough? Sure, maybe noobs don’t know what to expect or could be fooled by something signed by someone else, but I assure you those people aren’t going to be saved by secure delivery of a SHA1SUMS file.

  34. RSnake Says:

    @Daniel - true but signed by a key that I can’t validate beyond that it happens to be served up over HTTP via MIT’s PGP server and signed by a bunch of people who have nothing (at least not obviously) to do with Mozilla.

    I don’t know… if that really seems good enough to you, then you’re right, you shouldn’t change it. But you guys seem to be making two arguments.

    1) Signed windows installers are the only thing that matters. If that’s true get rid of all SSL everywhere on the site. It’s useless and slows things down. I could get on board with this position because at least it’s defensible. It just doesn’t make sense with your disjointed use of SSL.

    2) You need to sign things with a key that is not necessarily clear if it’s even yours, and you should have hashes of all binaries that are questionably legitimate. But you shouldn’t have to deliver them over SSL in all cases just some - and only when people happen to know you or happen across some obscure link somewhere. If this is your position I have a hard time understanding how you could believe this is truly secure.

    You could shut me up right now by deciding that SSL is worthless and moving away from it completely for all components and instead doing your own self signing on every component (including XUL and JS files), or using SSL everywhere. I don’t see that there’s really a middle ground. Would you guys disagree?

  35. Daniel Veditz Says:

    You’re right, the releases key could look dodgy. Fixed: it’s now signed by the key published at the Google #1 result for both “Mozilla security key” and “Mozilla PGP key”. And that page is available over https if you manually change the scheme.

  36. Skuld Says:

    When referring to the signed windows binaries, I think they are talking about the code-signing signatures not PGP signatures, i.e. this: http://img208.imageshack.us/img208/9305/image1gn.jpg
    So the statement about keys being served over HTTP from MIT’s server doesn’t really apply to these, although it does still apply to those for apple and linux, which they have already referred to. And if your making the argument that people will just ignore when it isn’t valid, they will also likely ignore when the ssl certificate is invalid on the site, so what would it matter either way? The only people who these issues is going to affect are people knowledgeable about security in the first place and they should know enough to understand trust.

  37. Tom T. Says:

    *****************************************
    # Daniel Veditz Says:
    February 4th, 2010 at 2:55 pm

    Oh, and for people paranoid about whether they’ve actually reached the real Mozilla site, it, too, it available through SSL

    https://www.mozilla.com/

    # RSnake Says:
    February 4th, 2010 at 2:56 pm

    @Daniel did you actually click on any of those links from https://www.mozilla.org/ ? They all take you to HTTP sites…

    # Mike Shaver Says:
    February 4th, 2010 at 2:57 pm

    I agree that people *can* click through it…..
    *********************************************
    @ Danilel Veditz and Mike Shaver:

    “Can” click through it? You can’t do anything else.

    Let’s say I’m an average home user. I go to https://www.mozilla.org using IE (6), because I’ve heard Firefox is safer, and I want to be sure to get it securely. The Firefox link points directly to www.firefox.com. End of HTTPS.

    OK, I go there with Fx 2.0.0.20. I just got back from Mars, and I’ve missed a few versions. Again, https://www.mozilla.org. Ah! A padlock! I view it: a slight, but insignificant, name mismatch, and an aging RC4-128 vs AES-256, but I’m happy! So I click “Firefox”, and am taken to http://www.mozilla.com/en-US/firefox/upgrade.html. End of HTTPS.

    Clicking “Download now” (Windows) offers to save a d/l from (on various tries), Netherlands, Germany, and some other place I didn’t immediately recognize. None of these mirrors were secure. (I’m in the US.)

    One mopre try, far beyond Average User-land: From http://www.mozilla.com/en-US/firefox/upgrade.html, I just inserted an “s” after “http” and hit “enter”. Aha! Success! Same halfway-decent certificate! I’m on the right Firefox sub-page! So… Click “Download”, and again, http, this time a mozilla.org site itself.

    I have to back Robert up on this one. Perhaps “most of your users use the secure update system”, but there really does not seem to be any way to get the installer directly through a https connection.

    What’s weird is that add-ons are served securely through AMO. https://addons.mozilla.org/en-US/firefox/. Add-on delivery is secure, but not the browser itself? I think the browser installer should automatically be d/l’d securely from any browser, anywhere, without any tech knowledge at all. Eliminate all the non-secure ways and default to a secure download, as you do for add-ons.

    I think Robert was right to bring this to the attention of whomever. I don’t want to check signed Windows binaries; I certainly know *how* to check hash checksums, but I don’t want to have to; and most users don’t know how to do either. Let all d/l be secure for John and Jane Average — automatically.

  38. madddddddddd Says:

    even though the signatures are passed over HTTP, the client could still check multiple sources, then the only attack vector would be comprising all the hosts in the mirror pool simultaneously.

  39. JL Says:

    When i read your article i was stumped by both it’s contents and the matter this FI-NA-LY was adressed by a security professional. This almost seemed like taboo.

    At least now i no longer have to feel dumbfounded.

    IMHO the whole browser-extension is a crime in itself, in a hostile environment that is. Each and every one of these is a gaping hole.

    There’s no guarantee to what it does, how well the code was written and people are able to install it, just like that. Why bother to go and ‘intrude’ on a site, just write a plugin.

    Well, i said more the this article’s about, mostly since i did not want to get into yet another SSL is barely used properly and hey it’s broken after all talk. Mostly as i lack proper technical knownledge to persue meaningfull talk.