Paid Advertising
web application security lab

Inline or Out of Bounds - Defeating Active Security Devices

I’ve wrestled with the concept of inline devices to stop attacks for a long time. Disclaimer: no names of companies will be used in this post. However, there are legitimate reasons both to have inline devices and not to as well. Let’s look at the cons of inline security protection devices for a moment. The major cons of an out of band device have to do with high availability and throughput. In many networks even a fraction of a second delay is totally unacceptable. Packet loss is a non-starter too so an inline device must be able to handle the throughput that the pipe can generate (potentially gig line speeds). Also, the device cannot create a single point of failure, often requiring double the hardware for inline devices, which makes out of line boxes potentially more cost effective (assuming the cost of the switch, for the span port, or tap isn’t too high). Those things combine to make a pretty complex solve if you are determined to have an inline device.

Now the advantages of inline devices is that they can decelerate SSL traffic without requiring an extra external SSL accelerator and that they can see everything and potentially block anything that they detect as malicious. Let’s look at the difference in how an out of band IPS/WAF type device would have to work. Either it can make firewall rule entries by connecting to the firewall in front of the web application or it uses the China method of RST packets in one or both directions. Now, let’s dissect that scenario for a second.

For XSS this isn’t a terrible solution. RST packets in both directions means that anyone getting duped into clicking a link won’t see the resultant malicious JavaScript return to them. That’s great! Now let’s take SQL injection. There are two types of SQL injection that should be considered. The first is where I want to dump all the results from a database, or set my user account to be something other than what it should be which gives me a different sort of auth cookie - both require something to be sent back to the attacker, and because HTTP is a noisy protocol it’s totally feasible that it could block the packet before it’s sent. The other is where I want to change a password, or drop a table - where the attacker doesn’t need to get any result back. The same is true with remote file includes, or command injection. In each of those cases, the resultant request is not necessarily important for the attacker to “see” the results from the same IP address that they were attacking from. In fact, they could easily be different IPs, on unrelated ports, or at different times of day, etc…

So out of line security devices have a severe dis-advantage when it comes to the latter injection attacks (which can often be far more dangerous than the former attacks that involve reflected results). Attackers have long known how to use different IP addresses when necessary, so I don’t see any reason why they couldn’t do so in this case to evade the device or firewall rule. Not to mention firewall rules can often end up blocking lots of innocent people who happen to be behind the same NAT. So for my money, it looks like inline devices win that round - if throughput isn’t a major concern.

Further, there is another hidden problem here. Let’s say an attacker is unsure if an active security device is in place (either inline or out of line - it doesn’t matter), but doesn’t want to get erroneous results when testing. All the attacker would have to do is intentionally send something they would know should be blocked by any decent signature, and listen for a RST packet (if it is a device that sends one to you) or wait for nothing to be returned, (if it is the kind that sends the RST packet to the server or outright blocks the connection). With a large enough sample size of various default signatures, it’s possible to do actual versioning of the exact devices as well.

Enter the dawn of IPS/WAF fingerprinting. Code samples welcome.

4 Responses to “Inline or Out of Bounds - Defeating Active Security Devices”

  1. ntp Says:

    only gigabit line speeds? what about 10gbe/40gbe?

    honestly, WAF and APIPS/APIDS need to have a solid architecture going in, somewhat like you describe.

    i’ve said it many times. let’s go through this again.

    don’t put a WAF directly in between your website and the Internet. if you’d like, you can separate your traffic asymmetrically so that inbound takes a different path than outbound.

    normally most organizations have two separate groups: the front-end HTML/CSS monkey-coders and the backend software engineers. if the HTML/CSS group can’t get it right and make sure their XHTML is validated, then you’re going to want to fix the fact that you’re not providing strict output encoding. if the backend programmers can’t get it right and make sure that every input is validated server-side (and that no client-side code such as Ajax/JSON contains important enough domain logic that can be easily changed), then you’re going to want to fix the fact that there is no input validation.

    one solution to the first problem is for operators to place an inline WAF or APIPS on the outbound. you don’t need this WAF/APIPS to log anything, just be able to provide fixes for encoding issues. access to this machine should be OOB with OTP SSHv2. the WAF/APIPS shouldn’t have an IP address on either the internal or external network.

    in the case of input validation, it should be carefully avoided to utilize a WAF/APIPS on the inbound for this purpose. however, in order to stray basic problems such as DDoS or brute-force authentication attempts, a rate-based IPS can be used on the inbound as long as packets are sampled during normal operation. similarly as described above, the rate-based IPS should be inline, should be accessed OOB with OTP SSHv2, and the internal and external interfaces should not have IP addresses (i.e. bridge-mode).

    i’m trying to say that input validation can only be solved by fixing the code. if an organization decides to use a WAF/APIPS on the outbound, it’s best to compare the cost of converting to validated XHTML to a facelift of an architecture.

    validated XHTML has additional benefits over WAF’s — just as your backend software engineers or software quality engineers. fixing the code is almost always the right answer in 2008. appliances are so 2001!

    the nice thing about APIDS is that, yeah — you can use a tap or SPAN/RSPAN/ERSPAN/mirror port, as well as psamp, netflow/sflow/jflow, or hell even RMON. i hear that IDS load-balancing devices also exist. who would have known?

  2. inrouted Says:

    An excellent comment. 10gbps is already here, and although i havent seen a mast on the horizon, im sure 40 is not too far away. The product market place is oddly sparse when you get into the 10+ line rate range. Products that actually do that tend to be so prohibitively expensive that few companies can foot the bill to implement it.

    Just about every inline device i have seen is deployed bridged as you note above. There just isnt really any other way to do it correctly. Where I get interested in this topic is when we start talking about IPS virtualization, and being able to have multiple instances of an IPS device serving multiple customers, or company departments. Whether the device uses VLAN translation or just trunks it all the way is still being argued. Translation seems to be a waste of sometimes quite valuable vlan addressing space.

    Detection and Evasion:
    Well, as is usual with the IPS device, we dont necessarily need to DOS the device itself. With most implementations the management, monitoring, and reporting infrastructure has the most exposure outside of the target webpage itself. IDS devices ran into this, where you just had to flood the box with bad traffic from various sources, then you blend in with everyone else and get lost in the volume, IPS boxes have the same achilles heel. While a box might be able to process, halt, and forward 10gbps of traffic, i cannot think of very many companies that can handle the reporting load generated by 10gbps of line traffic.

    Code etc:
    You are right sir, in that exploitation has shifted from the appliance is king mentality to “i sure hope we got our code audited”. Because no IPS device in the world can protect a badly programmed site. As far as load balancing, some of the stuff put out there by companies like Tipping Point and Reflex show some interesting progress in how they deploy to Service provider wide installations.

  3. Drazen Drazic Says:

    I did a survey around my team a year or so ago about whether our testing was impacted by *any* “devices” and the answer was in 90% + cases, no! Sometimes testers assume too much when in reality, the stuff that is there, is not doing anything, reporting anything and if per chance it is, no one is able to understand what the hell it is reporting. Is there much difference when put into practice between in-line and out of line in practice so to speak?

    Sorry for the link…it’s not marketing…but Dec’s presentation at Kiwicon, broadcast on Risky Business takes it back to basics.

  4. Brian S Says:

    Very good post RSnake. A few thoughts:

    1. Very few companies actually do much blocking, if any, with their WAFs (inline or out-of-line) because they are afraid of DOSing themselves.

    2. If a website has vulnerabilities like the examples you describe (ability to change another user’s password or drop a table via SQL Injection), I wouldn’t rely on a WAF to cover for that and sleep well at night. Even the very best WAFs can be pretty hit-and-miss in terms of attack detection, and that’s assuming that the WAF is even in blocking mode for that particular signature/profile violation, which it probably isn’t.

    3. Deploying a WAF out-of-line is a whole lot easier and less risky than in-line. And you get a nice view of your web applications and another way of identifying application security defects. These defects then can and should be brought to the attention of the web development teams and fixed at the source.

    4. Drazen brings up a really important point. Having somebody in charge of looking at the WAF data who actually understands web apps is really important, in my opinion even more important than considering the pros and cons of the various deployment modes and blocking abilities. And the person(s) managing the WAF and analyzing the data from the WAF should probably not be the same people managing the network firewalls and IDS devices - chances are they don’t have the right skillset to manage a WAF effectively (even if they’re really good at the network security stuff).