December 22, 2004

Life During Wartime

I have made some editorial adjustments to this post after Jay Allen - the author of MT Blacklist - showed up in the comments and corrected me on some points. Thanks Jay.

I mentioned in a comment to Helen's post about MT Blacklist that I couldn't see just what Blacklist was up to because I could no longer view the Activity Log. And the reason for that is that there were over 500,000 entries in the log, which is kept in a SQL table.*

So I dumped the table and cleared it out and restarted everything and found that it looked something like this:

Date IP Address Log Entry
2004.12.22 04:24:03 212.87.234.1 MT-Blacklist comment denial on Hold The Mayo: discounts.8bit.co.uk
2004.12.22 04:24:09 212.87.234.1 MT-Blacklist comment denial on trying to grok: discounts.8bit.co.uk
2004.12.22 04:24:30 212.87.234.1 MT-Blacklist comment denial on Ambient Irony: discounts.8bit.co.uk
2004.12.22 04:25:22 212.87.234.1 MT-Blacklist comment denial on Hold The Mayo: discounts.8bit.co.uk
2004.12.22 04:25:37 212.87.234.1 MT-Blacklist comment denial on Hold The Mayo: discounts.8bit.co.uk
2004.12.22 04:25:49 212.87.234.1 MT-Blacklist comment denial on angelweave: discounts.8bit.co.uk
2004.12.22 04:26:05 212.87.234.1 MT-Blacklist comment denial on Read My Lips - the blog: discounts.8bit.co.uk
2004.12.22 04:26:17 212.87.234.1 MT-Blacklist comment denial on Read My Lips - the blog: bmw.com
2004.12.22 04:26:51 212.87.234.1 MT-Blacklist comment denial on Hoppings of Roxette Bunny: discounts.8bit.co.uk
2004.12.22 04:26:52 212.87.234.1 MT-Blacklist comment denial on trying to grok: discounts.8bit.co.uk
2004.12.22 04:27:54 212.87.234.1 MT-Blacklist comment denial on Rocket Jones: bmw.com
2004.12.22 04:27:56 212.87.234.1 MT-Blacklist comment denial on Publius & Co.: discounts.8bit.co.uk
2004.12.22 04:28:04 212.87.234.1 MT-Blacklist comment denial on Rocket Jones: discounts.8bit.co.uk
2004.12.22 04:28:10 212.87.234.1 MT-Blacklist comment denial on The Nap Room: discounts.8bit.co.uk
2004.12.22 04:28:40 212.87.234.1 MT-Blacklist comment denial on Mookie Riffic: discounts.8bit.co.uk
2004.12.22 04:28:40 212.87.234.1 MT-Blacklist comment denial on Being American in T.O.: discounts.8bit.co.uk
2004.12.22 04:28:52 212.87.234.1 MT-Blacklist comment denial on trying to grok: discounts.8bit.co.uk
2004.12.22 04:28:55 212.87.234.1 MT-Blacklist comment denial on Semi-Intelligent Thoughts: comeback.com
2004.12.22 04:28:58 212.87.234.1 MT-Blacklist comment denial on trying to grok: discounts.8bit.co.uk
2004.12.22 04:29:02 212.87.234.1 MT-Blacklist comment denial on Semi-Intelligent Thoughts: discounts.8bit.co.uk
2004.12.22 04:29:02 212.87.234.1 MT-Blacklist comment denial on Semi-Intelligent Thoughts: comeback.com
2004.12.22 04:29:05 212.87.234.1 MT-Blacklist comment denial on Hold The Mayo: bmw.com
2004.12.22 04:29:13 212.87.234.1 MT-Blacklist comment denial on Semi-Intelligent Thoughts: discounts.8bit.co.uk
2004.12.22 04:29:15 212.87.234.1 MT-Blacklist comment denial on Read My Lips - the blog: discounts.8bit.co.uk
2004.12.22 04:29:47 212.87.234.1 MT-Blacklist comment denial on trying to grok: discounts.8bit.co.uk
2004.12.22 04:29:53 212.87.234.1 MT-Blacklist comment denial on Hoppings of Roxette Bunny: discounts.8bit.co.uk
2004.12.22 04:30:01 212.87.234.1 MT-Blacklist comment denial on Rocket Jones: discounts.8bit.co.uk
2004.12.22 04:30:23 212.87.234.1 MT-Blacklist comment denial on Rocket Jones: discounts.8bit.co.uk
2004.12.22 04:30:25 202.57.35.130 MT-Blacklist comment denial on Snooze Button Dreams: hermosa
2004.12.22 04:30:31 212.87.234.1 MT-Blacklist comment denial on Publius & Co.: discounts.8bit.co.uk
2004.12.22 04:30:47 212.87.234.1 MT-Blacklist comment denial on Being American in T.O.: discounts.8bit.co.uk
2004.12.22 04:30:49 212.87.234.1 MT-Blacklist comment denial on Semi-Intelligent Thoughts: discounts.8bit.co.uk
2004.12.22 04:31:05 212.87.234.1 MT-Blacklist comment denial on angelweave: comeback.com
2004.12.22 04:31:14 212.87.234.1 MT-Blacklist comment denial on The Nap Room: discounts.8bit.co.uk
2004.12.22 04:31:15 212.87.234.1 MT-Blacklist comment denial on Civilization Calls: discounts.8bit.co.uk
2004.12.22 04:31:17 212.87.234.1 MT-Blacklist comment denial on Read My Lips - the blog: discounts.8bit.co.uk
2004.12.22 04:31:26 212.87.234.1 MT-Blacklist comment denial on Rocket Jones: bmw.com
2004.12.22 04:32:07 212.87.234.1 MT-Blacklist comment denial on Rocket Jones: discounts.8bit.co.uk
2004.12.22 04:32:11 212.87.234.1 MT-Blacklist comment denial on Rocket Jones: comeback.com
2004.12.22 04:32:13 212.87.234.1 MT-Blacklist comment denial on Rocket Jones: discounts.8bit.co.uk
2004.12.22 04:32:28 212.87.234.1 MT-Blacklist comment denial on Rocket Jones: discounts.8bit.co.uk
2004.12.22 04:32:33 212.87.234.1 MT-Blacklist comment denial on Frank J. Fan Club: discounts.8bit.co.uk
2004.12.22 04:32:34 68.93.224.4 Search: query for 'poltergeist'
2004.12.22 04:32:40 212.87.234.1 MT-Blacklist comment denial on Mookie Riffic: discounts.8bit.co.uk
2004.12.22 04:32:40 212.87.234.1 MT-Blacklist comment denial on Mookie Riffic: discounts.8bit.co.uk
2004.12.22 04:32:46 212.87.234.1 MT-Blacklist comment denial on Rocket Jones: discounts.8bit.co.uk
2004.12.22 04:33:14 212.87.234.1 MT-Blacklist comment denial on Mookie Riffic: discounts.8bit.co.uk
2004.12.22 04:33:47 212.87.234.1 MT-Blacklist comment denial on Rocket Jones: discounts.8bit.co.uk
2004.12.22 04:33:49 212.87.234.1 MT-Blacklist comment denial on The Nap Room: discounts.8bit.co.uk
2004.12.22 04:33:49 212.87.234.1 MT-Blacklist comment denial on Auterrific: discounts.8bit.co.uk
2004.12.22 04:33:49 68.93.224.4 Search: query for 'poltergeist'
2004.12.22 04:34:08 212.87.234.1 MT-Blacklist comment denial on angelweave: comeback.com
2004.12.22 04:34:19 212.87.234.1 MT-Blacklist comment denial on Rocket Jones: discounts.8bit.co.uk
2004.12.22 04:34:28 212.87.234.1 MT-Blacklist comment denial on angelweave: discounts.8bit.co.uk
2004.12.22 04:35:44 212.87.234.1 MT-Blacklist comment denial on Semi-Intelligent Thoughts: discounts.8bit.co.uk
2004.12.22 04:36:04 212.87.234.1 MT-Blacklist comment denial on Hoppings of Roxette Bunny: discounts.8bit.co.uk
2004.12.22 04:36:11 212.87.234.1 MT-Blacklist comment denial on Semi-Intelligent Thoughts: discounts.8bit.co.uk
2004.12.22 04:36:13 212.87.234.1 MT-Blacklist comment denial on Semi-Intelligent Thoughts: discounts.8bit.co.uk
2004.12.22 04:36:18 212.87.234.1 MT-Blacklist comment denial on Being American in T.O.: comeback.com
2004.12.22 04:36:24 212.87.234.1 MT-Blacklist comment denial on Semi-Intelligent Thoughts: discounts.8bit.co.uk
2004.12.22 04:36:33 212.87.234.1 MT-Blacklist comment denial on Rocket Jones: discounts.8bit.co.uk
2004.12.22 04:36:36 212.87.234.1 MT-Blacklist comment denial on Snooze Button Dreams: discounts.8bit.co.uk
2004.12.22 04:36:51 212.87.234.1 MT-Blacklist comment denial on Ramblings' Journal: discounts.8bit.co.uk
2004.12.22 04:37:02 212.87.234.1 MT-Blacklist comment denial on trying to grok: bmw.com
2004.12.22 04:37:40 212.87.234.1 MT-Blacklist comment denial on Read My Lips - the blog: discounts.8bit.co.uk
2004.12.22 04:37:44 212.87.234.1 MT-Blacklist comment denial on Read My Lips - the blog: discounts.8bit.co.uk
2004.12.22 04:37:44 212.87.234.1 MT-Blacklist comment denial on The Nap Room: discounts.8bit.co.uk
2004.12.22 04:37:52 212.87.234.1 MT-Blacklist comment denial on Read My Lips - the blog: discounts.8bit.co.uk
2004.12.22 04:38:41 212.87.234.1 MT-Blacklist comment denial on Civilization Calls: discounts.8bit.co.uk
2004.12.22 04:38:43 212.87.234.1 MT-Blacklist comment denial on Rocket Jones: discounts.8bit.co.uk
2004.12.22 04:39:17 24.1.70.204 'JohnL' added entry #60439
2004.12.22 04:39:30 212.87.234.1 MT-Blacklist comment denial on Hoppings of Roxette Bunny: discounts.8bit.co.uk
2004.12.22 04:39:42 212.87.234.1 MT-Blacklist comment denial on Read My Lips - the blog: discounts.8bit.co.uk
2004.12.22 04:39:53 212.87.234.1 MT-Blacklist comment denial on angelweave: discounts.8bit.co.uk
2004.12.22 04:39:53 212.87.234.1 MT-Blacklist comment denial on trying to grok: discounts.8bit.co.uk
2004.12.22 04:40:06 212.87.234.1 MT-Blacklist comment denial on The Nap Room: discounts.8bit.co.uk
2004.12.22 04:40:15 212.87.234.1 MT-Blacklist comment denial on angelweave: discounts.8bit.co.uk
2004.12.22 04:40:27 212.87.234.1 MT-Blacklist comment denial on Rocket Jones: discounts.8bit.co.uk
2004.12.22 04:40:27 212.87.234.1 MT-Blacklist comment denial on Hold The Mayo: discounts.8bit.co.uk
2004.12.22 04:40:36 212.87.234.1 MT-Blacklist comment denial on Publius & Co.: discounts.8bit.co.uk
2004.12.22 04:40:42 212.87.234.1 MT-Blacklist comment denial on Rocket Jones: discounts.8bit.co.uk
2004.12.22 04:40:45 68.238.27.186 User 'Physics Geek' added 2 entries to the MT-Blacklist
2004.12.22 04:40:59 212.87.234.1 MT-Blacklist comment denial on The Nap Room: discounts.8bit.co.uk
2004.12.22 04:41:26 212.87.234.1 MT-Blacklist comment denial on Civilization Calls: discounts.8bit.co.uk
2004.12.22 04:41:40 212.87.234.1 MT-Blacklist comment denial on angelweave: discounts.8bit.co.uk
2004.12.22 04:42:00 212.87.234.1 MT-Blacklist comment denial on Mookie Riffic: discounts.8bit.co.uk
2004.12.22 04:42:44 212.87.234.1 MT-Blacklist comment denial on The Nap Room: discounts.8bit.co.uk
2004.12.22 04:42:45 212.87.234.1 MT-Blacklist comment denial on Rocket Jones Skunkworks: discounts.8bit.co.uk
2004.12.22 04:42:49 212.87.234.1 MT-Blacklist comment denial on Hoppings of Roxette Bunny: discounts.8bit.co.uk
2004.12.22 04:43:16 212.87.234.1 MT-Blacklist comment denial on Madfish Willie's Cyber Saloon: discounts.8bit.co.uk
2004.12.22 04:43:17 212.87.234.1 MT-Blacklist comment denial on angelweave: discounts.8bit.co.uk
2004.12.22 04:43:38 212.87.234.1 MT-Blacklist comment denial on angelweave: discounts.8bit.co.uk
2004.12.22 04:43:50 212.87.234.1 MT-Blacklist comment denial on Semi-Intelligent Thoughts: discounts.8bit.co.uk
2004.12.22 04:43:53 212.87.234.1 MT-Blacklist comment denial on Practical Penumbra: discounts.8bit.co.uk
2004.12.22 04:43:56 212.87.234.1 MT-Blacklist comment denial on The Nap Room: comeback.com
2004.12.22 04:44:37 212.87.234.1 MT-Blacklist comment denial on Ilyka Damen: discounts.8bit.co.uk
2004.12.22 04:45:09 68.93.224.4 Ping 'http://blog.mu.nu/cgi/mt-tb.cgi/260' failed: HTTP error: 500 read timeout
2004.12.22 04:45:38 212.87.234.1 MT-Blacklist comment denial on trying to grok: discounts.8bit.co.uk
2004.12.22 04:45:53 212.87.234.1 MT-Blacklist comment denial on Semi-Intelligent Thoughts: discounts.8bit.co.uk
2004.12.22 04:46:05 212.87.234.1 MT-Blacklist comment denial on Rocket Jones: discounts.8bit.co.uk
2004.12.22 04:46:14 212.87.234.1 MT-Blacklist comment denial on Rocket Jones: discounts.8bit.co.uk
2004.12.22 04:46:18 212.87.234.1 MT-Blacklist comment denial on Mookie Riffic: discounts.8bit.co.uk
2004.12.22 04:46:33 212.87.234.1 MT-Blacklist comment denial on Read My Lips - the blog: discounts.8bit.co.uk
2004.12.22 04:46:42 212.87.234.1 MT-Blacklist comment denial on Mookie Riffic: discounts.8bit.co.uk
2004.12.22 04:46:53 212.87.234.1 MT-Blacklist comment denial on The Nap Room: discounts.8bit.co.uk
2004.12.22 04:46:59 212.87.234.1 MT-Blacklist comment denial on Read My Lips - the blog: discounts.8bit.co.uk
2004.12.22 04:47:03 212.87.234.1 MT-Blacklist comment denial on trying to grok: discounts.8bit.co.uk
2004.12.22 04:47:12 212.87.234.1 MT-Blacklist comment denial on Hold The Mayo: discounts.8bit.co.uk
2004.12.22 04:47:17 212.87.234.1 MT-Blacklist comment denial on Ace of Spades HQ: discounts.8bit.co.uk
2004.12.22 04:49:26 134.159.132.4 'breanagh' added entry #60440
2004.12.22 04:56:46 68.217.68.204 'Queenie' added entry #60441
Badness. The apparent gaps in that trail of tears do not represent a pause in the incoming spam but rather my attempts to reconfigure the comments script to block it entirely. This is necessary because even when MT Blacklist blocks a comment, it still uses a lot of resources on the server, and when you're getting flooded with crap, the server basically grinds to a halt. Each comment requires an Apache process, a Perl process and a MySQL database process, and the system was processing 30-40 comments at the same time.

[Jay Allen, the author of MT Blacklist, has questioned me on this in the comments. I should explain that the problem the server suffered today was not a question of CPU utilisation but of memory - with 30 comments being handled at the same time, the system did not have enough memory to cope, and was constantly swapping tasks to and from disk, a condition known as "thrashing". This is a bad thing.]

For those who speak Linux, the load average was up around 70 during all of this. Which made it less than perfectly responsive when I was logged in and trying to fix it.

The point where the spam stops dead is where I got smart and simply added a .htaccess file that completely blocks access to the CGI directory for that particular spammer.

Meanwhile, Movable Type have provided a fix this problem - with the release of MT 3.14. Which means I really need to get moving on the migration.

* This is a pretty silly place to keep a log file, but what the heck.

Posted by Pixy Misa at December 22, 2004 05:05 AM

Comments
#1

"This is necessary because even when MT Blacklist blocks a comment, it still uses a lot of resources on the server, and when you're getting flooded with crap, the server basically grinds to a halt."

Actually, when MT-Blacklist blocks a post, it uses very little resources (although more than an Apache-level solution) and happens very early on in the process (before mt-comments.cgi even GETS the comment). The problem in 3.121 and earlier is that comments sent to moderation use more resources than they should. This, as you said, is remedied in v3.14.

Of course, regardless of whether your comment form is hooked to Movable Type or a simple formmail script, if you're getting 500,000 requests to your site in a short amount of time, you're going to buckle under the load. Low-cost shared servers weren't intended for that sort of load and there is precious little that Movable Type can do about that. That sort of thing is best handled by the Apache config[1].

I would also like to point out that neither Movable Type (a software program) nor Six Apart (the company who makes it) were sleeping. In fact, we were hard at work on making your software better. I only WISH that I could get more sleep...

[1]: http://it.slashdot.org/comments.pl?sid=133226&cid=11127930

Posted by Jay Allen at December 22, 2004 08:30 AM
#2

I would also like to point out that neither Movable Type (a software program) nor Six Apart (the company who makes it) were sleeping. In fact, we were hard at work on making your software better. I only WISH that I could get more sleep...

I apologise for that. I didn't mean it that way, and as a programmer myself I understand and sympathise with you on that semi-mythical stuff called sleep.

Posted by Pixy Misa at December 22, 2004 08:58 AM
#3

Would the installation of something like James Seng's SCode be of any use? I've gotten rather handy with that plugin of late, and it seems to work nicely with MTBlacklist...please let me know if I can be of any help in anti-spam warfare. It's a grudge thing with me. ;)

Posted by Queenie at December 23, 2004 07:49 PM
Live Comment Preview
Post a comment









Remember personal info?