Webappsec Crisis Management
Last night I was listening to the radio regarding the latest debacle with Countrywide and how they have hired a PR firm to deal with the crisis of everyone thinking they alone are responsible for the real estate crunch. That’s the boring part, the interesting part, I thought, was the three steps towards dealing with a major crisis. How this relates to webappsec and getting hacked? We’ll see. But here are the steps (modified slightly to make sense with webappsec):
First, explain what happened. Tell the consumers what happened, why it happened, and if there is any way to shift blame now is the time to do it. This is the first critical step though, because it will no doubt come up later in an investigation, so you can’t lie, but you can spin. This is where you explain as many details about how you were hacked as possible. And if you don’t know the answer, here is where you say the authorities are looking at it, or there is an ongoing investigation to uncover the details.
Second, attempt to make the consumer whole. That is, if the consumer lost something during the crisis (whether it be a loved one, or a password) attempt to compensate them, or fix the issue. Sometimes this can involve patch management, and sometimes it involves settlement of a class action lawsuit.
Third, take visible steps to assure the consumer that it will never happen again. How is a consumer ever going to trust you again if you can’t prove to them that you have fixed the technology issue that allowed the hole to exist in the first place? And I’m not just talking about that example of that specific hole, but all future holes like it, for the rest of time. This is the hardest step, because it allows for some public scrutiny of the company’s ability to protect the consumer. Maybe it’s building a robot to scan the corporate systems for exploits, but that implies that the robot is as good at finding holes as hackers are. Although hackers aren’t fooled by that, perhaps enough consumers are to make this a viable solution to the third step of web application security crisis management.
Now as a case study, let’s look at something that just happened with Google - the cross site scripting exploit in Google’s Mini Search Appliance. After 12 days, they admitted to the problem (perhaps a little long given you could just look at it and see the problem). That met the first phase of crisis management. The second phase - making the consumer whole - was by delivering a patch to the customers. Granted, that makes the consumer do more work (hidden costs) to use the product, but I think we as a tech industry are used to that kind of thing and although we forget to calculate it into our cost models, we bite the bullet and eat the additional expense.
But what about the last one? What has Google done to convince the consumer that they will never have this problem again? From what I can tell, nothing. Not to pick on Google specifically here, but isn’t that our biggest fear? Sure, we know they can fix the problems once they know about it, but what about the ones they don’t know about? They have had a hard time lately, and have been unable to prove their ability to write secure code. They have essentially failed the last step of webappsec crisis management. Again, not to pick on them, because there are lots of companies who have done worse, but it’s a good case study in why each and every step is vital for public perception.
October 4th, 2007 at 10:16 am
It’s impossible to convince (or “prove”even) to many people that you will never have a problem again. Software is too complex, and built by humans. However, you can show that you are doing your best to avoid such issues in the future.
This is where Microsoft gets a big round of applause from me. They fessed up to their “problem”, and instituted some big changes to try and fix it. Is their software bug free? Not likley, nor will it ever be, but they showed a commitment to doing the best effort they could, and for me, that make a big difference.
Let’s look at Google now. Not to rag on at them, but their software is full of holes, and I don’t see them doing *anything* to assuade users that they are making any sort of “best effort” not to make the same mistakes again. There’s no talk of SDLC out of them, no “secure API’s”, nothing that tell me it’s not anything but likley they will do the same again.
I used to really like Google, but this continued attitute (it’s not just this case, but a lot of similar examples), and the cavalier way that they seem to treat privacy and data worries me. I, for one, have stopped using them. They are Microsoft in the 90’s waiting for their own version of Slammer/CodeRed to perhaps wake up and smell the coffee.
October 4th, 2007 at 11:12 am
As you mentioned above, taking the proper steps to take care of the problem once it is brought to their attention is the key. Just following the thread of PHPIDS on sla.ckers makes me realize the amazing amounts of attack vectors there are on web applications. Which does make me wonder, why there aren’t more web app security assessment tools out there?
I’m sure those vectors are being incorporated somewhere, most likely in rootkit distributions and other malware.. And like MikeA said, it’s going to take some sort of massive ownage of their servers for companies to implement the proper security steps to ensure not only the security of their systems, but the privacy of their customers.
–thrill
October 4th, 2007 at 4:51 pm
I don’t really think Google have been facing a PR crisis here, its been a bad week, but I’ve yet to see an article saying something along the lines of “Google heading the way of Old Microsoft” or similar, all of them have said something along the lines of “Google is having a bad week in terms of security” or similar. So while we’ve been told they have problems, there has been nothing which I would call a PR crisis, and nothing short of an actual crisis where people start heavily criticising Google, _and they start losing revenue_ is going to make them change, we saw the same thing with MS, the only reason they did something was because companies decided that their track record with security was purely unacceptable and demanded better.
And even there it was a push from _companies_, not individuals, who made it happen. Individuals do not have the buying power to influence those changes, especially considering most consumers see Google as a free service which they want, so are not about to stop using it.
So essentially Google did what it thought was enough to stop the criticism getting any worse, rather than stop it.
October 5th, 2007 at 4:40 pm
Kuza55 - While you may not have seen any articles saying that they are the next Microsoft, I have talked to probably a half dozen reporters who have said exactly those words. So I think they are a hair’s breadth from having exactly that article written about exactly that topic. Also many many articles have been written saying exactly those words: http://search.yahoo.com/search;_ylt=A0geu_MRyQZHbYUBUgVXNyoA?p=%22Google+is+the+next+Microsoft%22&y=Search&fr=FP-tab-web-t and http://search.yahoo.com/search;_ylt=A0geu.ZIywZHTV0AljNXNyoA?p=%22Is+Google+the+next+Microsoft%22&y=Search&fr=FP-tab-web-t
But I think you partially mis-read my post. I didn’t mean that consumers had to be mom and pa sitting at a computer. A consumer can be anyone who “consumes” the product - in their case the consumer is a company who buys their product, and they have very real pull in Google’s success with that product line. The consumer’s consumer is mom and pa though - which may be confusing.
But ultimately Google _is_ facing a huge PR crisis - their popularity in the last 6-12 months has gone through the tubes compared with their rivals. Is that 100% security related? Obviously not, but it is significant. Their privacy woes, their predatory business tactics, and their security problems have all made for a PR problem, additively. You may like them still, but you have to face the fact that they aren’t the darling internet star that they once were. And really, why should they be when their only significant revenue source is being an evil advertising company like Doubleclick whom, by no coincidence, they are attempting to buy?
Ultimately though, I was just trying to present a use-case that we can all visualize that recently occurred - it may not have been an ideal one. A better one may have been the TJMaxx case, actually.
October 5th, 2007 at 8:40 pm
You’re right of course that reporters have been making comparisons to Microsoft, but they have been primarily focused on the fact that Google is getting so damn big, and the anti-trust issues we have with both companies.
I’m just saying that until we start seeing articles saying that Google’s apps have as many holes as windows 95 or something similar, and businesses start taking a stand and saying something along the lines of “It is not acceptable to use Google products (e.g. Gmail, google docs) for corporate affairs”, or even do something to shift users to another search engine, its not going to hurt them and nothing is going to get better.
So while I don’t view Google as this evil fiery nemesis, reminiscent of Microsoft (who I never really had anything against, they just wrote buggy software), I think that unless security is brought to the forefront, and Google starts losing money in a way that can obviously be tied to their poor security, its not going tog et any better.
P.s. What’s so evil about advertising companies? (Yes, I really do want to know….)
October 5th, 2007 at 8:54 pm
Kuza55 - I agree on all accounts. Although I wouldn’t say they are wildly better than MS - especially given that a huge chunk of their apps that we can test have been proven to have holes, and yes, I have heard a number of CISOs say they forbid Google Desktop on corporate environments. Take it however you want.
Ad companies, like most companies, are interested in money. Let’s take one very simple to explain example (there are many of these). In the PPC display-banner world one of the ways ad companies makes money is by telling people, “Create the best looking banner you can so people will click on it.” And they even go so far as to actually reject certain banners if they aren’t attractive enough - saying the sites that display the banners need a certain quality. Because the quality of the banner is “high” it does make more unqualified people click on the banner, as opposed to a really ugly banner that basically makes people think they will have to purchase an item immediately once they click on it - forcing qualified leads and lowering advertiser costs. That is only _one_ example. Don’t even get me started on click fraud. The reason I know this? I used to work for ValueClick, and am on the advisory board for Click Forensics.
One of my favorite stories was where Google presented to a huge group of advertisers and said, “Our click fraud amount is X amount” and one guy from a big company stood up and said, “So you are saying that the click fraud that we detected alone amounts for half of all your click fraud?” *crickets* I guess _someone_ is failing to report the real numbers, aren’t they? Online advertising is an ugly business.
October 6th, 2007 at 11:57 pm
One thing they could do is assure their user’s that they’ll stop making so many feature-driven bad design decisions ex: Picassa and Google Desktop’s “locally” running web server, Picasa’s built-in web browser, Google Docs, etc.
It’s ok to make mistakes, we all do. The key is that, like Microsoft, they must learn from their mistakes and adapt their approach.
November 12th, 2007 at 9:27 am
RSnake, you can add the fourth step for webappsec crisis management. It’s thanking for people who discover the holes.
Because Google didn’t thank me for informing them about those holes (and I wrote them after I informed MI5). And it is common situation.
Every company must not forget to thank people who discover the holes and worked to improve their and their costumers security.
November 12th, 2007 at 9:32 am
@Mustlive - Yes, but in not thanking you, did they somehow make that crisis worse? Are you no longer going to be nice about disclosure or is there some ramifications for them? In my experience that really isn’t much of a problem for companies - you may like them less, but historically you aren’t going to hack them any worse. Also, some would argue that these small holes aren’t a crisis. I was really talking about huge data breaches in my post.