Version 1.2 of Snark!™ - our new trackback filtering system - is up and running.
You can see real-time statistics here
Hooray for Snark!™
Update: Version 1.3 of Snark!™ is now up and running with a high score list in addition to the running totals.
Update: Version 1.4 of Snark!™ offers an automatically generated blacklist.
Update: Version 1.5 of Snark!™ tracks and lists IP addresses as well, though that's only been running for a few minutes so there aren't any addresses listed yet.
Update: Version 1.51 has the added feature of not crashing when there's no spam, a situation that never arose during testing.
Update: Version 1.6 of Snark!™ automatically and permanently blacklists anyone whose spam ranking goes over the magic number.
Update: Version 1.7 keeps track of the effectiveness of the various filters. Those stats are a bit screwy right now because of... Well, because. But they will improve over time.
Update: Version 1.71 fixes some bug or something. I've forgotten.
Update: Version 1.72 makes the decay algorithm fully adaptive. Basically, that means that high-volume spammers are considered particularly annoying. This makes no difference for our internal filtering, but I think makes the stats more reflective of the actual evilness level of particular spammers.
Update: Currently running version 1.84, which addresses all known bugs and inefficiencies. It's able to process 20,000 trackbacks per second, removing roughly 99.9% of spam and so far producing no false positives. It is a thing of beauty. :)
Posted by Pixy Misa at November 24, 2005 07:53 AM | TrackBackThis is the bit I like:
Blacklist Entries: 18 (plus 55 manual entries)
Session Uptime: 0 hours 18 minutes
Pings Received: 1013
Processing Time: 0.23 seconds
Snark filtered out 1013 trackbacks in 0.23 secooioooooooooooooooooooooooooooooooooo sorry, I just broke one of the keys on my laptop. Hang on. Nnnnngggh. Right.
Snark filtered out 1013 trackbacks in 0.23 seconds. Movable Type takes twice that long to filter out one trackback.
Posted by Pixy Misa at November 25, 2005 11:36 AMAnother funny this is what I wrote a couple of weeks back: We got 1.3 million trackbacks over the past 30 days, and 550,000 comments. About 99.8% of trackbacks and over 90% of comments were spam. That was just an off-the-cuff estimate, not the result of careful calculation.
But Snark says:
Since Startup: Total 39842, Good 78, Bad 39764, % Spam 99.80
Not bad for a guess!
Posted by Pixy Misa at November 25, 2005 01:37 PMI've just been deleting trackback spam from my blog for the third time today.
Dammit, now I want to come back to MT 2.6!
Posted by Pixy Misa at November 25, 2005 02:00 PMTweak tweak:
Blacklist Entries: 23 (plus 60 manual entries)
Session Uptime: 5 hours 44 minutes
Pings Received: 1530
Processing Time: 0.24 seconds
Latest version (1.84):
Blacklist Entries: 23 (plus 60 manual entries)
Session Uptime: 8 hours 28 minutes
Pings Received: 1024
Processing Time: 0.14 seconds
Aha! I wondered why my inbox wasn't filled with millions of spam pings here lately. I had attributed it to a higher power. It wasn't, however, the Great Pixy Misa who came to mind.
Posted by Tig at November 27, 2005 02:40 PMYay!
It does seem to have done the trick. There's still a bit of spam leaking through, but it is enormously reduced.
As I noted, my blog, which is running on MT 3.1 and not protected by Snark, is getting more trackback spam than all the MT 2.6 blogs put together. Since I installed version 1.2, you've got an average of one-third of a spam each. I'm getting thirty a day.
Posted by Pixy Misa at November 28, 2005 04:09 AMSo now you just shrink wrap and sell for $10 a pop - every blogger in the universe is at your mercy! Yet again, Nobel Prize for Blogging Services goes to Pixy.
Posted by Simon at November 28, 2005 06:50 AMProblem with that is it only works well if you have a large number of blogs. I suppose you could also do it with a one hour delay instead of one minute.
However, by keeping it to ourselves we make munu an even more enticing place. Let's face it, the trackback spam was getting to be a real drag, but now it's just gone. (Well, not for me; I just got another 20 in the past two hours. I'll have to do something about that.)
Posted by Pixy Misa at November 28, 2005 10:15 AM