It was a long weekend and sorry for not posting, but id and I were able to get a lot done this weekend. We got 1 1/2 of the machines up and running out of the three that I had hoped to get running. One machine is having some issues but at least it’s turned on where it was pretty much non-functional as of yesterday. One of my stupid laptops needs to be sold at auction so that I can get another one (that’s the last time I buy a cheap Dell laptop). Anyway, the lab is doing quite a bit better than it was before. Once we get all the software up and running we should see better performance, and less downtime in general.
Speaking of downtime we experienced about 45 minutes of downtime on Saturday morning. A few people posted about it on the forum or emailed us about it so I thought I should mention it. No we weren’t hacked, it was just a runaway process that wasn’t behaving nicely and on top of that it wasn’t giving off any of the obvious signals to help us diagnose the issue. id came to the rescue and we got that one resolved in just a few minutes. From the time I noticed until the time it was back up was only about 15 minutes, cuz he rocks. Hopefully with the new server we’ll notice issues like this faster with some monitoring that I’ve been meaning to build. I’ve built that kind of software before, but those machines and that code is long gone, so I need to do it from scratch.
In web app sec news, we were able to ban in excess of 700 IPs programmatically from attempting to do bad things to the site. The firewall is being updated in somewhat-real-time of things I find particularly annoying. Sort of a self defending network (not to get myself sued). It’s not that they have any hope of getting in, but I hate seeing that crap in my logs. It’s an ongoing process so the site should experience less load from the morons and as a result you might see a small increase in page load time until our traffic load grows again to compensate for any good we may have done to reduce it.
Lastly, id found a rather annoying and very reproducible bug in my Netgear WGT624 wireless router, which caused it to stop routing packets every time he did it. I’ve seen it do this sort of thing in the past but could never consistently reproduce it. I’d tell you how to do it, but it wouldn’t help you because we could only reproduce it over SSH (requiring his keys and the exact server in question to be communicating with one another), and we didn’t have enough time to dump the packets and see what was causing it. Needless to say there definitely is some sort of error on those wireless routers and maybe the next time he’s over we’ll try to figure out exactly what it is. Until then I’ll just put up with it crashing on me every once in a while - as annoying as I find that.
So anyway, it was a productive weekend even if I don’t have a lot to say about it. Hopefully in the next few days or so after we get the bugs ironed out of the servers I can get back to the testing.