Cenzic 232 Patent
Paid Advertising
web application security lab

The Effect of Snakeoil Security

15 posts left…

I’ve talked about this a few times over the years during various presentations but I wanted to document it here as well. It’s a concept that I’ve been wrestling with for 7+ years and I don’t think I’ve made any headway in convincing anyone, beyond a few head nods. Bad security isn’t just bad because it allows you to be exploited. It’s also a long term cost center. But more interestingly, even the most worthless security tools can be proven to “work” if you look at the numbers. Here’s how.

Let’s say hypothetically that you have only two banks in the entire world: banka.com and bankb.com. Let’s say Snakoil salesman goes up to banka.com and convinces banka.com to try their product. Banka.com is thinking that they are seeing increased fraud (as is the whole industry), and they’re willing to try anything for a few months. Worst case they can always get rid of it if it doesn’t do anything. So they implement Snakeoil into their site. The bad guy takes one look at the Snakeoil and shrugs. Is it worth bothering to figure out how banka.com security works and potentially having to modify their code? Nah, why not just focus on bankb.com double up the fraud, and continue doing the exact same thing they were doing before?

Suddenly banka.com is free of fraud. Snakeoil works, they find! They happily let the Snakeoil salesman use them as a use case. So our Snakeoil salesman goes across the street to bankb.com. Bankb.com has seen a two fold increase in fraud over the last few months (all of banka.com’s fraud plus their own), strangely and they’re desperate to do something about it. Snakeoil salesman is happy to show them how much banka.com has decreased their fraud just by buying their shoddy product. Bankb.com is desperate so they say fine and hand over the cash.

Suddenly the bad guy is presented with a problem. He’s got to find a way around this whole Snakeoil software or he’ll be out of business. So he invests a few hours, finds an easy way around it and voila. Back in business. So the bad guy again diversifies his fraud across both banks again. Banka.com sees an increase in fraud back to the old days, which can’t be correlated to anything having to do with the Snakeoil product. Bankb.com sees their fraud drop immediately after having installed the Snakeoil therefore proving that it works twice if you just look at the numbers.

Meanwhile what has happened? Are the users safer? No, and in fact, in some cases it may even make the users less safe (incidentally, we did manage finally stop AcuTrust as the company is completely gone now). Has this stopped the attacker? Only long enough to work around it. What’s the net effect? The two banks are now spending money on a product that does nothing but they are now convinced that it is saving them from huge amounts of fraud. They have the numbers to back it up - although the numbers are only half the story. Now there’s less money to spend on real security measures. Of course, if you look at it from either bank’s perspective the product did save them and they’ll vehemently disagree that the product doesn’t work, but it also created the problem that it solved in the case of bankb.com (double the fraud).

This goes back to the bear in the woods analogy that I personally hate. The story goes that you don’t have to run faster than the bear, you just have to run faster than the guy next to you. While that’s a funny story, that only works if there are two people and you only encounter one bear. In a true ecosystem you have many many people in the same business, and you have many attackers. If you leave your competitor(s) out to dry that may seem good for you in the short term, but in reality you’re feeding your attacker(s). Ultimately you are allowing the attacker ecosystem to thrive by not reducing the total amount of fraud globally. Yes, this means if you really care about fixing your own problem you have to help your competitors. Think about the bear analogy again. If you feed the guy next to you to the bear, now the bear is satiated. That’s great for a while, and you’re safe. But when the bear is hungry again, guess who he’s going after? You’re much better off working together to kill or scare off the bear in that analogy.

Of course if you’re a short-timer CSO who just wants to have a quick win, guess which option you’ll be going for? Jeremiah had a good insight about why better security is rarely implemented and/or sweeping security changes are rare inside big companies. CSOs are typically only around for a few years. They want to go in, make a big win, and get out before anything big breaks or they get hacked into. After a few years they can no longer blame their predecessor either. They have no incentive to make things right, or go for huge wins. Those wins come with too much risk, and they don’t want their name attached to a fiasco. No, they’re better off doing little to nothing, with a few minor wins that they can put on their resume. It’s a little disheartening, but you can probably tell which CSOs are which by how long they’ve stayed put and by the scale of what they’ve accomplished.

22 Responses to “The Effect of Snakeoil Security”

  1. Steven Says:

    Nice write up.

  2. stucky Says:

    How exactly is this situation with CSO’s any different from the rest of the IT industry? All industry?

    Its disheartening sure but its nothing new.

  3. Kyle Says:

    Great article, thanks for taking the time to write it up.

  4. RSnake Says:

    @stucky - it’s different because CSOs have a much higher risk job than most IT managers.

  5. avinash Says:

    well written…banka, bankb analogy is very appealing.

  6. Brian Honan Says:

    You hit the nail on the head with the bear analogy. Until we all start working together as a community, which includes your competitors and vendors, we are simply delaying the inevitable.

    If we look upon ourselves as a herd and the predators are picking off the weak, what are you going to do once those weaker than you are gone and you are next in the firing line? We need to cooperate better on intelligence, tools and methodologies to beat the bad guys.

  7. Greg Says:

    Nice article, worth a read by all. Good insight to the mentality side of things that is so often overlooked. Security folks need to not just look at numbers or actions…they need context. And to get that context they need to understand the mentality of the attackers. If one attack is down, all well and good but perhaps that is because a more hidden, more rewarding hole is being exploited on your system and you have no numbers for it yet.

    Another analogy I use is our kids. Sometimes you have to think like they do to keep them from doing things you told them not to do, or to protect them. “But you said I couldn’t ride my bike to her house…I WALKED so I didn’t disobey you”

  8. llvllatrix Says:

    Counter examples? I’m thinking of the effort behind patching DNS cache poisoning.

  9. RSnake Says:

    @llvllatrix - I wouldn’t exactly call that a counter example. Yes, a lot of CSOs told their IT dept to apply some patches to their DNS services. Patch management is security 101, and it almost certainly wasn’t actually even something the CSOs had to do themselves - just delegate. It was also all over the news - even normal people heard about it. And btw, there are still a ton of DNS services out there that are still vulnerable.

    More importantly how many of those CSOs has fought for and won split horizon DNS while they were updating DNS? How many of them turned off that pesky zone transfer that they’ve had enabled on their secondary DNS while they were at it? How about making sure they removed that old reference that’s now pointing to the malicious domain squatter’s IP? How about that localhost reference? I still find DNS done wrong more often than right.

    That’s true even in security organizations, like OWASP. No, most organizations did the absolute bare minimum. There may be counter examples that would prove your point, but DNS is probably not one you want to rest your case on.

  10. AppSec Says:

    Ah, the good old prisoner theory problem comes back time after time..

    Everyone has to be in on the solution, or everyone loses.

    The idealist loves that scenario, the realist hates it.

  11. Varun Says:

    Of course the root problem here is how to distinguish between good solutions and bad solutions, isn’t it? Much easier said than done. If you have a good technical team plus a management that supports you and understands the business environment you might be successful but that combination is quite rare.

  12. Jarrod Loidl Says:

    Spot on.

    @stucky - I agree. While the risk profile is higher, the patterns of behaviour are certainly the same.

    @rsnake - what exacerbates the problem is when businesses have a blame culture rather than an educate/nuture culutre. If a CSO is going to be sacked for two bad audit findings or a major security incident, then its no wonder they move on.

    However if the C-level down are willing to accept mistakes and failures will occur, but aim to learn from them, then this will do wonders for staff retention.

    - J.

  13. MM Says:

    I agree that there is a real problem with the sale of snakeoil security solutions. However, I don’t necessarily agree with the root cause you identified here. I think the real issue is that we have designed security strategies backwards. Security is not always a cost center, and in fact doing poor security properly can allow organizations to pass any cost associated to security onto unknowing consumers. In other words security for them is free. From my perspective however, practicing this type of security strategy is unethical and socially irresponsible. But then what do I matter?

    The fact is that the average enterprise simply does not care enough. This isn’t because the CSO is only there for a short time, it is because the CSO’s responsibility is more focused on the enterprise not being fined for violating any particular standard. Therefore, the driving force behind security investment is not security itself but rather compliance. As a direct result there EXISTS a market for security snakeoil.

    This is the first major area of a backwards security strategy. When compliance drives security rather than security driving compliance.

    The second major area is the area of ignorance/indifference. The fact of the matter is that the average business decision maker is totally ignorant/indifferent of/to the real issues and social impact of poor information security. Unfortunately, the market is built to increase this ignorance/indifference as opposed to combat it.

    For example, take a company that has invested a lot of time and effort into a particular product. While the product may have become less and less effective over time the cost associated with redesigning that solution to be more effective in a current environment is much higher than innovating the product to a certain extent and marketing the heck out of it. Put simply, less overhead = increased profits. Therefore, there is money to be made by creating less effective security but making the world feel as though you created the end all solution. What is unfortunate is the fact that there are good vendors out there who are creating innovative products and really combating other vendors who push ignorance. Sadly, though these products often are viewed as “high-cost” products that can only sell to particular markets because their overhead is higher. Unfortunately, the average consumer does not do the necessary research to understand the difference between the two. Furthermore, the consumer who does do the research may fall back onto the first major area I identified anyways (it’s about compliance not security). Either way, the provider of the less secure product is making money hand over fist, while the higher cost solution is fighting for a market.

    The environment of ignorance allows vendors selling snakoil to survive and thrive in the existent snakeoil market.

    The final area of a backwards security strategy that I’ll discuss here is the area of demand. There is a major demand for simplicity in the security market. Unfortunately, security is a constant chess match between threats and defenders. Which at the end of the day is anything but simple. However, the demand for simplicity reduces the criticality of necessary components for ensuring security such as higher network visibility and lower-level endpoint lockdown controls. In addition it allows capabilities that should never be key competitive differentiators to play a major role in security technology adoption. As security technology buyers demand solutions that tell them and show them less, they create less and less effective security strategies. However, the strategy itself will over time cost less and less money to implement. Now some might argue that this time of organization will get hacked and lose more money over time. To that I would simply say, “MAYBE.” The fact is that it would have to be a pretty big hack to justify that for a large enterprise. Moreover, the numbers are not in your favor for justifying that arguement. E.g. if a company spends $15,000/year recovering from malware and saves $30,000/year on a low cost security solution, at the end of the day they still netted $15,000. For the sake of arguement let’s say that the organization does have that rare occasion where they really get nailed and do lose a lot of money…

    In that case the company can publicly redeem itself by citing the fact that they met the criteria demanded by compliance boards and they are leveraging the best security products available (according to what the ethos states based on marketing messages). Thus, the fact that they got hacked, really isn’t their fault.

    So the real problem with snakoil security is that compliance has created a market for it, ignorant and indifference have created a sustainable environment for it, and business decision makers are demanding it.

  14. Ben Says:

    To what extent can an industry counter-act this effect by sponsoring OWASP and other community oriented efforts?

    A dream job would be to work full time developing OWASP training used across the industry. After creating the same guidance for 5 different organizations with different branding and examples, seems like it’d be worth 1/5th the cost to each of them for me to create equivalent public material.

  15. llvllatrix Says:

    @RSnake - I agree. I can’t think of another counter example.

    In my mind, the problem seems to be that both the endorsement and efficacy of fixes are made in the short term. That sort of decision making works for simple software bugs; does it work, yes, no?

    As you point out, it doesn’t for security fixes because the vendor may not have actually addressed the problem; merely deterred the attacker. So how do we fix this problem? Is it a problem with the business model?

  16. digi Says:

    @llvllatrix - you fix the problem by educating the purchasers. There is no quick fix to a security issue… Security is a support function, one that rarely brings revenue, and it is costly. Every process we implement costs. Every piece of equipment we buy to support those processes costs. Even after the price is paid, there is still residual risk.

    1. How much residual risk can you take?

    2. There is no one thing that will fix your issues.

    These two points have to be drilled into anyone that runs an infrastructure of any kind, and sadly, mainline IT managers and C-level execs do not understand this and politically, security guys don’t get their way very often.

    Until you can educate the people that pay the snakeoil salesmen, there will always be snakeoil.

  17. MM Says:

    Hey why did you delete my comment?

  18. Aniket Says:

    Liked the bear analogy. But u cant demean products by saying that it helps in giving better security only till a workaround is found!
    Or is this true?!!

  19. Paul Says:

    Snakeoil???

    Trusteer.com

    Trusteer works with more than 60 leading banks around the world to keep your online bank account safe from online fraudsters. Trusteer Rapport has been downloaded by more than 11 million customers. It picks up where anti-virus and firewalls leave off, preventing new, sophisticated attacks that anti-virus and firewalls are not always updated to protect you from. To download Rapport now, click here

  20. Marshmellow1328 Says:

    Here is my shameless plug to an article I recently wrote about similar problems in the software development industry. It does not talk about application security, but talks about similar problems with focusing on short-term solutions versus long-term solutions.

    http://www.marshmallowswisdom.com/2010/08/25/balancing-today-with-tomorrow/

  21. matt Says:

    I think the same reasoning applies to individuals as well as products.

    There are all kinds of bad professionals, and it can be difficult for laypersons to know. Bad lawyers, doctors, dentists, electricians, plumbers and so on. For most professions you can also judge competency by results, but for computer security this is difficult to do because of effect RSnake points out here and for the lack of solid deliverables. Certification has a role to play, but computer security as a profession changes more quickly than most.

  22. stlsaint Says:

    HA! Nice way of telling it!!