Back On Track

In case you found the notice below a little confusing, here's the news: Trackbacks are back in business!

I've just finished hacking together a triple-bypass for the Movable Type / MT Blacklist system we've been suffering from.

The new system isn't amazingly sophisticated, but it does have some useful features:

• Trackback throttling - more than 5 pings from one place in a minute and they all go in the dumpster.
  
• Fast mini-blacklist - stops them before they even go to MT Blacklist.
  
• MuNu bypass - no throttling or blacklists for intra-munu pings.
  
• Background processing - no waiting ages for pings to go through.
  
• No errors, either. Well, them's the breaks.
  
• Detailed logging. If we lose your ping, we can find it and fix it.
  
• Did I mention fast? Fast fast fast!
  
• Less work for Fluffy. Spam floods are a breeze now!
  
• Which means that trackbacks don't get shut down!
  
• Yay!
  

Downside? Well, pings can take up to a minute to appear. But then, they used to take a minute to post. Also, if your ping fails for some strange reason, you don't get an error. It just says "Okay!" and drops it down the toilet. So if you have any problems, let me know.

Go for it, Susie!

Update: Doesn't seem to work for trackbacks from Wordpress. I'm not sure why; it seems that Wordpress does something funny/different when it sends the ping. Investigating. I don't know how to use Wordpress.

Update: 30,322 trackbacks received by the new system. 238 were passed on to stage three, where they still need to get past MT Blacklist. There's still quite a bit of crap getting past stage two, but it is weeding out 99.5% of the spam while using 0.4% of the resources of the old system. I count that as a win.

I'll look at tweaking stage two further. That part is all my own code and written in Python, and since it's a generic trackback/comment filtering system, I can easily re-use it in Minx, so the effort won't be wasted. But I think it's good enough for now.

I've tried pinging from MT 2.6, MT 3.1 and Wordpress, and they all went through fine. If you are running something else, feel free to give this post a ping and we'll see how it goes. Remember that it can take up to a minute for your trackback to appear.

Update: New feature that will be going in tonight: Tunable throttling. Right now there's a hard limit: 5 pings from any one (non-munu) source in a batch (which represents one minute) and they all get deleted. But if some spammer sends us 10 pings in one minute and 4 the next minute, the 4 pings go through.

With the new system, it holds a grudge. So if you send 10 pings in one minute, the next minute it still counts you as having sent those pings - less a tunable percentage (currently 10%). So in minute one, you send 10 pings, they get dumped. Minute two, it considers you to have already sent 9 pings, so anything else you send gets dumped and bumps the counter up. At 90% grudge level, it would take 9 minutes before it would let them send even one ping through.

Any number less than 1 gets dropped completely, so people sending a ping here, a ping there, shouldn't have any problems. And as before, it doesn't affect intra-munu pings at all.

Update: Oh yeah, I'm going to expand I've expanded the intra-munu thingy to a more general whitelist, so we can unblock friendly but prolific pingers if need be.

Update: Version 1.1 of the Snark!™ Trackback Filtering System is up and running. When you need to protect a blog that's precious to you, go with the very best: Go Snark!™