Slowness/Downtime Mystery Solved!

It turns out the guy who runs blbl.org decided it was taking up too much of his CPU, so he shut it down. We still had it listed in Spam Karma 2 on one of the blogs hosted here, and since it was gone completely and not even refusing connections, the connections were taking their normal 75 seconds or so to time out, tying up Apache processes. We allow a maximum of 60 processes, so fewer and fewer were available to service requests until there were none. Note that this would not have happened under normal circumstances because the blacklist is only checked for comment attempts; this was caused by spammers.

The problem was very hard to track down because all I had to go on was an IP address that all the Apache processes were attempting to connect to, and the IP address itself isn't mentioned anywhere. There was no reverse DNS for the IP and no mention in Google. I eventually figured out that I could dump the local DNS server's cache and find which hostname was mapped to the IP I had in the cache, which is how I figured out it was blbl.org.

I really need to start working on the spam solution I've been thinking of, which is basically a collection of tripwires all over the 'net that will update everyone's blacklists in real time. We're generally only getting hit in a short burst by any one IP, so it's absolutely vital if one wants to blacklist by IP to share the IPs as quickly as possible.

Share this