THE PROBLEM
I expect most of you have noticed by now that lately the server has been experiencing brief outages a couple of times a day, for two or three minutes each time.
What's happening is this:
We get a burst of spam (evil spam!) hitting twenty to fifty blogs all at the same time.
The server fires up the necessary twenty-to-fifty CGI sessions to handle the spam.
These make twenty-to-fifty connections to the database.
The server runs out of memory, and starts "swapping". That is, it pushes some of the programs out to disk instead of trying to hold them all in memory. Unfortunately, disks are hundreds of times slower than memory, so this makes the machine grind to a halt.
There's an automatic monitoring process that checks every five minutes to see if the web server is working, and if it doesn't get a response, it restarts it. This also kills all the CGI programs trying to process the spam.
Suddenly everything is working again, although sometimes if the spam flood is particularly bad it will go through another cycle of this.
THE SOLUTION
There a couple of things that can be done to fix this.
1. Buy more memory. If the spam gets bad enough, we will still run out of memory, but it should improve things.
2. Move to MT3, which is more efficient in discarding spam and getting out of the way. We want to move to MT3 for many reasons, so I'll be setting everyone up with an MT3 login and blog so we can start the mass migration.
3. Use something like mod_perl. I haven't tried this, and I'll need to do some reading first, but this should speed things up and reduce the memory requirements for this sort of thing. A PHP application would offer the same benefit; I just need to find one that doesn't suck.
4. Close comments. Closing your old comments should help by preventing a lot of spam in the first place.
5. Hunt the bastards down and strangle them with their own network cables. Check with the police in your area first, as they may prefer to do this themselves.
Pixy:
Turns out there IS a mechanism available for effectively throttling comment and trackback spam on MT 2.6 (The default is useless, as it only throttles by IP, and spammers are way past that now) Teh limits would need to be adjusted for MuNu's massively multi-blog nature, but I think it would still help, aspecially with server load issues?
Burningbird has the info, links to where the solutions come from, and the ready-to-upload update files.
The downside is, the solution requires updating to 2.66 and the newer version of Blacklist - which MuNu desperately needs anyway as the newer Blacklist is VASTLY more useful for manual spam removal, especially on trackbacks and crapflooding.
With the time investment to change over to MT3.1, you may not want to invest any further resources into the existing version, but it would have immediate payoff in dealing with the situation until everybody could be migrated.
Paul
Posted by Light & Dark at January 21, 2005 12:48 AMI vote MT 3 giving us more time to hunt the ******** down and strangle them with their own network cables. Heck with the cops, they'll help.
Posted by Nick Queen at January 21, 2005 12:58 AMThanks Paul. I'll take a look at that this weekend. Updating MT Blacklist shouldn't be hard.
Posted by Pixy Misa at January 21, 2005 01:32 AM(I know I said something like that before, but then I got run over by a bus. Or something.)
Posted by Pixy Misa at January 21, 2005 01:33 AMI vote MT3 (even though I don't keep up my site--It will come back, just a matter of time!).
Is there a way to close off comments on all the old posts with out having to go through each and everyone?
And I agree with Dad/Ted, Cialis has to be banned. I got hit with a good 50 spam comments this morning.
Posted by Mookie at January 21, 2005 01:35 AMPixy - damn busses.
Unfortunately, you can't upgrade the Blacklist without going to 2.66 first. In for a penny, in for a pound.
Paul
Posted by Light & Dark at January 21, 2005 02:05 AMI vote for MT3. If we're upgrading, we might as well move all the way up to the latest one. Pixy, is a tip jar ever going to be available to us to help out on all of this? Or are you just set on being a mensch all the time?
Mookie - you're coming back??? When???
Posted by Daniel at January 21, 2005 02:19 AMI'm trying to come up with a suitable regexp, but so far without success.
Posted by Pixy Misa at January 21, 2005 03:50 AMYeah Daniel, I'm gonna come back, mainly I'm just waiting until I have some time to have my template redone. I'm gonna change the name of the blog to "Down Stage Left" and my idea is to have it look like a play bill with what ever show I'm currently obssessed with listening to as a watermarked back ground. Its gonna take a while to get done because I'm not good with Html/CSS and Dad's been so busy that I don't want to bug him overly much about it.
So yeah, in short, I'm coming back but still don't know when.
Posted by Mookie at January 21, 2005 04:49 PMYes to moving and yes to donating. But NO to banning Ciallis. It's the only thing that I've found that really works as advertis....hang on....do what? Yes he did, he said "ban Ciallis". Oh. He meant ban it via the blacklist spam thingy? Are you sure? Whew. Um..never mind about the Ciallis.
Posted by Howard at January 21, 2005 05:19 PM