Half a Million HTTP Headers
Over the last few months I’ve accumulated a half a million HTTP headers. It obviously contains a lot of garbage, but it’s still fun to toy with in aggregate. Of course there are lots of hackers in there - because it’s totally unstructured data. It’s some of the worst organized content I’ve ever seen, but it’s still fun to mess with.
If anyone’s interested, I threw it up on my other blog. A half a million HTTP headers might sound like a lot, but really, a lot of it is duplicated data. For instance search engine spiders, or RSS readers are going to look the same every time. But still, I thought it was interesting, just to look at.



September 11th, 2009 at 10:44 pm
380 HTTP________ and 684 HTTP________________
what are those?
September 13th, 2009 at 6:38 pm
They’re client software that is trying to obscure certain fields by replacing the text with other characters. Here’s what I mean:
HTTP________
HTTP_REFERER
HTTP________________
HTTP_ACCEPT_ENCODING
Clever, but not all that helpful for really hiding the information.
September 14th, 2009 at 2:43 pm
Like client side firewalls do, avast and others. Silly, because they also put their name into it. Like: REFERER BLOCKED BY AVAST. Something like that. Silly, now i know what they use.