I'll be putting a new trackback system in place this weekend. It's a two-stage process. The original script will simply log the fact that a trackback has been requested and acknowledge it to the sender. Then a second script will come along periodically, read the log of requests, and process them.
How does this help?
Three ways: First, when we get hit by a spam flood, we can get fifteen trackbacks per second. The new script doesn't do any work; it just writes the trackback to a file, which takes a small fraction of a second. This means less work for Fluffy, less chance for trackbacks to get turned off, and so on.
Second, if we get 50 copies of the same trackback, it's certain to be spam, and we can just ignore it without doing any more processing. We don't even need to use the blacklist for this.
Third, there's only one copy of the script that actually applies the trackbacks ever running on the server. Since the server has two processors, no matter how many trackbacks we get, it can't overload the server. It might get behind in its processing, but that's pretty much the worst that can happen.
Posted by Pixy Misa at November 18, 2005 04:06 AM | TrackBackSo how long of a delay will there be before trackbacks appear? I usually check to see if my pings went through and re-post if they didn't.
Posted by Susie at November 18, 2005 03:34 PMOKay, that's a downside.
Problem is, only about 1 in every 500 pings we receive is valid. So I have to do something.
I could have it display a pending pings page so you could see it there.
Posted by Pixy Misa at November 18, 2005 03:48 PMI'm currently running the first step, where it records trackbacks in a file, in parallel with normal processing. I'll probably turn the second step on tomorrow.
I'll start with a short interval, say 30 seconds, and see how it goes.
Posted by Pixy Misa at November 18, 2005 05:57 PMI could do the same for comments too. I just realised that. Duh. I'll see if I can work that up this weekend too. I'd just make the interval much shorter.
Posted by Pixy Misa at November 18, 2005 05:59 PMIt should be better than attempting to locate the spam trackbacks and eliminate them.
Posted by Mad Mikey at November 18, 2005 08:43 PM30 seconds or even a minute should be ok...if you do it for comments, should I put a "please wait, don't repost your comment" message in my comment template? (not that I have all that many comments anymore with my lackadaisical posting!)
Posted by Susie at November 19, 2005 02:01 PMWhat I'd do for comments is slightly different. I'd update the individual entry straight away, but update the main page and the archives in the background. Which is what Blogspot does, and I think what Typepad does as well.
Posted by Pixy Misa at November 19, 2005 02:38 PMOK...I don't understand it, but as long as you do...
Anyway--I just tried to pingslaught my buds and got this:
2005.11.19 15:01:18 71.97.163.55 Ping 'http://blog.mu.nu/cgi/mt-tb.cgi/128058' failed: HTTP error: 500 Internal Server Error
2005.11.19 15:01:18 71.97.163.55 Ping 'http://blog.mu.nu/cgi/mt-tb.cgi/128218' failed: HTTP error: 500 Internal Server Error
2005.11.19 15:01:18 71.97.163.55 Ping 'http://blog.mu.nu/cgi/mt-tb.cgi/129112' failed: HTTP error: 500 Internal Server Error
2005.11.19 15:01:18 71.97.163.55 Ping 'http://blog.mu.nu/cgi/mt-tb.cgi/128555' failed: HTTP error: 500 Internal Server Error
2005.11.19 15:01:18 71.97.163.55 Ping 'http://blog.mu.nu/cgi/mt-tb.cgi/129160' failed: HTTP error: 500 Internal Server Error
2005.11.19 15:01:18 71.97.163.55 Ping 'http://blog.mu.nu/cgi/mt-tb.cgi/129017' failed: HTTP error: 500 Internal Server Error
2005.11.19 15:01:18 71.97.163.55 Ping 'http://blog.mu.nu/cgi/mt-tb.cgi/128625' failed: HTTP error: 500 Internal Server Error
The only ones that went through were yours and a haloscan one! Help!
Posted by Susie at November 19, 2005 03:07 PMUh oh. You upset Fluffy. I've given her a sedative, so try again now.
Posted by Pixy Misa at November 20, 2005 01:25 AMSigh...I just now got home from work, and tried re-pingging to no avail. Fluffy's sedatives must have worn off!
Posted by Susie at November 20, 2005 05:09 AM
I had a slight glitch with the new system and had to back off to the original version. I'll be trying again after I have a nap.
I had a lot of trouble sending even one ping. But the spam manages to get through, dammit!
Posted by Pixy Misa at November 20, 2005 06:20 AMIt's nearly ready to go.
I've added a couple of extra things. For one, intra-munu pings always go through, and bypass MT Blacklist. For another, all rejected pings are logged in a special file, and can be retrieved later.
I'll fire it up tonight. 
I hate Perl, and I hate Movable Type, and I hate CGI, and I hate Apache.
What a pile of crap.
Posted by Pixy Misa at November 22, 2005 08:11 AM