100,000 Served!
PingsThat's 99,735 spams you don't have to delete, because Snark is on the job.Since Startup: Total 100034, Good 285, Bad 99735, % Spam 99.70
Last Minute: Total 1, Good 0, Bad 1, % Spam 100.00
Filters: Whitelist 24, Blacklist 34187, Age 6561, Volume 33747, Format 2, All Rejects 74497
Processing Information
Blacklist Entries: 44 (plus 71 manual entries)
Session Uptime: 31 hours 51 minutes
Pings Received: 23453
Processing Time: 1.41 seconds
I tried using some deferred posting and it sort of is going haywire. It's posting, but it's not waiting for the designated time. I've got my time set to EST, but the auto-poster seems to be just posting almost anything from the current day that I set up as deferred. Is this a server time issue, or is it me?
It doesn't seem to be posting things I've got deferred until a later day, and it seems to not post something that's got a time set about 6 hours in the future...
Actually, the more I think about it, it does seem to be 5 hours or so off -- I've got code that gets the current date and last night when it was after 8pm my time, the date had advanced to the next day...
Ok, I've got a question for the technical people. I want to use the value from a php variable as a parameter value in an MT tag. For example, I want to set the categories attribute of a MTEntries tag from a php variable:
MTEntries lastn='2' category='$thiscat'
However, when I try that, I get an error when rebuilding that says No such category 'thiscat'
I've tried various different options here. If I make it look like this:
MTEntries lastn='2' category=\"$thiscat\"
it lets me compile, but it ignores the category (it selects all posts in the mtentries section.
So, any ideas how/if this can be done?
I know that using some combination of Fantastico and other assorted goodies I can set up a page that runs forums...my only question is how? Step by step from start to finish would be greatly appreciated.
Thanks and happy Thanksgiving.
Version 1.2 of Snark!™ - our new trackback filtering system - is up and running.
You can see real-time statistics here
Hooray for Snark!™
Update: Version 1.3 of Snark!™ is now up and running with a high score list in addition to the running totals.
Update: Version 1.4 of Snark!™ offers an automatically generated blacklist.
Update: Version 1.5 of Snark!™ tracks and lists IP addresses as well, though that's only been running for a few minutes so there aren't any addresses listed yet.
Update: Version 1.51 has the added feature of not crashing when there's no spam, a situation that never arose during testing.
Update: Version 1.6 of Snark!™ automatically and permanently blacklists anyone whose spam ranking goes over the magic number.
Update: Version 1.7 keeps track of the effectiveness of the various filters. Those stats are a bit screwy right now because of... Well, because. But they will improve over time.
Update: Version 1.71 fixes some bug or something. I've forgotten.
Update: Version 1.72 makes the decay algorithm fully adaptive. Basically, that means that high-volume spammers are considered particularly annoying. This makes no difference for our internal filtering, but I think makes the stats more reflective of the actual evilness level of particular spammers.
Update: Currently running version 1.84, which addresses all known bugs and inefficiencies. It's able to process 20,000 trackbacks per second, removing roughly 99.9% of spam and so far producing no false positives. It is a thing of beauty. :)
I had a URL to a nifty site that would encode html, php, etc. with ASCII characters so that I could display it on a web page. I think Willie posted the original link, but I could be wrong. If anyone has the URL to that, could you post it in the comments please?
Thanks, Eric - http://grumbles.mu.nu/
Okay, never mind. But, just to make life easier now that I've found it:
In case you hadn't noticed, it's time for the annual Weblog Awards, hosted by Kevin of Wizbang. The list of MuNu blogs is endless(okay, not really: I'm just oo lazy to type them all in) and there are plenty of categories for which you can nominate your fellow MuNuvians. Here are the categories:
Best Blog
Best New Blog (Established after November 19, 2004)
Best Group Blog
Best Humor/Comics Blog
Best Liberal Blog
Best Conservative Blog
Best Media/Journalist Blog
Best Technology Blog
Best Culture/Gossip Blog
Best Sports Blog
Best Photo Blog
Best LGBT Blog
Best Military Blog
Best Blog Design
Best Podcast ***
Best Video Blog ***
Best Religious Blog ***
Best Parenting Blog ***
Best Law Blog ***
Best Business Blog ***International
Best Canadian Blog
Best UK Blog
Best European Blog (Non UK)
Best Asian Blog
Best Middle East or Africa Blog
Best Australia or New Zealand Blog
Best Latino, Caribbean, or South American BlogEcosystem Based (View the TTLB Ecosystem November 14th snapshot)
Best of the Top 250 Blogs
Best of the Top 251 - 500 Blogs
Best of the Top 501 - 1000 Blogs
Best of the Top 1001 - 1750 Blogs
Best of the Top 1751 - 2500 Blogs
Best of the Top 2501 - 3500 Blogs
Best of the Top 3501 - 5000 Blogs
Best of the Top 5001 - 6750 Blogs
Best of the Top 6751 - 8750 Blogs
Best of the Rest of the Blogs (8751+) ******* New categories for 2005.
Get those nominations in now!
In case you found the notice below a little confusing, here's the news: Trackbacks are back in business!
I've just finished hacking together a triple-bypass for the Movable Type / MT Blacklist system we've been suffering from.
The new system isn't amazingly sophisticated, but it does have some useful features:
Downside? Well, pings can take up to a minute to appear. But then, they used to take a minute to post. Also, if your ping fails for some strange reason, you don't get an error. It just says "Okay!" and drops it down the toilet. So if you have any problems, let me know.
Update: Doesn't seem to work for trackbacks from Wordpress. I'm not sure why; it seems that Wordpress does something funny/different when it sends the ping. Investigating. I don't know how to use Wordpress.
Update: 30,322 trackbacks received by the new system. 238 were passed on to stage three, where they still need to get past MT Blacklist. There's still quite a bit of crap getting past stage two, but it is weeding out 99.5% of the spam while using 0.4% of the resources of the old system. I count that as a win.
I'll look at tweaking stage two further. That part is all my own code and written in Python, and since it's a generic trackback/comment filtering system, I can easily re-use it in Minx, so the effort won't be wasted. But I think it's good enough for now.
I've tried pinging from MT 2.6, MT 3.1 and Wordpress, and they all went through fine. If you are running something else, feel free to give this post a ping and we'll see how it goes. Remember that it can take up to a minute for your trackback to appear.
Update: New feature that will be going in tonight: Tunable throttling. Right now there's a hard limit: 5 pings from any one (non-munu) source in a batch (which represents one minute) and they all get deleted. But if some spammer sends us 10 pings in one minute and 4 the next minute, the 4 pings go through.
With the new system, it holds a grudge. So if you send 10 pings in one minute, the next minute it still counts you as having sent those pings - less a tunable percentage (currently 10%). So in minute one, you send 10 pings, they get dumped. Minute two, it considers you to have already sent 9 pings, so anything else you send gets dumped and bumps the counter up. At 90% grudge level, it would take 9 minutes before it would let them send even one ping through.
Any number less than 1 gets dropped completely, so people sending a ping here, a ping there, shouldn't have any problems. And as before, it doesn't affect intra-munu pings at all.
Update: Oh yeah, I'm going to expand I've expanded the intra-munu thingy to a more general whitelist, so we can unblock friendly but prolific pingers if need be.
Update: Version 1.1 of the Snark!™ Trackback Filtering System is up and running. When you need to protect a blog that's precious to you, go with the very best: Go Snark!™
Due to astronomical levels of crappiness inherent in Apache, CGI, Perl and Movable Type.
All the hard work is done; all I have to do is get the script to return a valid XML repsonse. Not only will it not do this, it takes up two seconds of CPU time not doing it.
All it has to do is write one line to a log file, return a five-line XML reply, and quit.
Can it do it? No.
Does it even fail quickly? No.
I'm going to replace it with some Python code and hard-coded SQL.
I'm going to replace it with some direct calls to the Perl CGI library and a switch on request_method() because the trackback script has two functions.
Bah.
It now logs a trackback in 40 milliseconds. Still far slower than I'd like, but 50 times faster than it was. And you're just not going to get much faster with a Perl-based CGI application. To do better you need something like mod_perl, or PHP, or a persistent server. And I'm not going to hack MT to that degree, thanks.
Anyway, I'll let it accumulate trackbacks for a bit and then run program two, and if that works well I'll run program three, and if that works well, I'll set it all to run automagically.
It can't be worse than what we had.
Update:
P-I-N-G-O
P-I-N-G-O
P-I-N-G-O
And Pingo was his name-o!
It works!!!!!!!!!!! :D
Oops. The call from stage two to stage three bobbled it. Fixy fixy.
No oops - it worked, but the error log went to the wrong place. No problem.
A NEW TRACKBACK SYSTEM IS ARISING! ITS VICTORY IS AT HAND! THERE WILL BE NO DAWN... FOR SPAM!
Check out my iPod version of my blog. I stole the code from someone some time ago. Feel free to steal it from me. Though I don't think it's really all that useful... just a toy...
1. When I save a post, it kicks me out to the login screen. So I log back in, and my post is there, saved as a draft, and I have to re-save it.
2. I can not, for the life of me, get a trackback through to anyone running a version of MT higher than ours.
Those two are pre-new server issues, btw.
3. My skin-switcher thingie that Phin installed is broken.
Thanks in advance for any help. And double thanks a million to Pixy for all his hard work. Really, man, you need a tip jar.
UPDATE:
Okay, I figured out the outgoing t/b problem. For some reasons, the spam filters must not like Vince Aut Morire, because when I changed the name of my blog to the Vinnie Musings Pundit Report, the t/b went through to the blogs I'm having trouble with.
Sigh.
I'll be putting a new trackback system in place this weekend. It's a two-stage process. The original script will simply log the fact that a trackback has been requested and acknowledge it to the sender. Then a second script will come along periodically, read the log of requests, and process them.
How does this help?
Three ways: First, when we get hit by a spam flood, we can get fifteen trackbacks per second. The new script doesn't do any work; it just writes the trackback to a file, which takes a small fraction of a second. This means less work for Fluffy, less chance for trackbacks to get turned off, and so on.
Second, if we get 50 copies of the same trackback, it's certain to be spam, and we can just ignore it without doing any more processing. We don't even need to use the blacklist for this.
Third, there's only one copy of the script that actually applies the trackbacks ever running on the server. Since the server has two processors, no matter how many trackbacks we get, it can't overload the server. It might get behind in its processing, but that's pretty much the worst that can happen.
I read a comment over at Ace's from LauraW... she mentioned that it would 'more organic' for a host such as Pixy to sell ads via the MuNu blogger group... That got me to thinking... I would certainly not mind putting up ads over at Madfish Willie's and let the proceeds go to Pixy to help offset hosting costs such as server space, bandwidth, and his labor efforts... I know that some of you already have some sort of blog ads posted, but would the rest of us be willing to participate in some group thingy for Pixy (if he would even consider it)?
Just thinkin out loud is all....
When configuring the new servers I set them to Sydney time (AEST) because that meant I wouldn't have to keep mentally adding or subtracting 10 or 11 hours every time I looked at something.
Unfortunately, that's screwed things up for other people. So I've set the clock back to GMT.
If suddenly things go really screwy, I had nothing to do with it.
Well, actually I am a dork, but don't tell anyone, ok?
Anyhoo...I've been having some problems with the times on my posts and I was wondering if anyone else has been having the same issue. Since yesterday my blog posts are coming out as if I'd posted them in EST instead of CST---the time zone of discriminating conoisseurs everywhere and my personal favorite since I happen to live in it. I've gone in and resaved my configuration and rebuilt the site, and posts are still coming out stamped in East Coast time.
Anyone have any ideas on this one? Thanks in advance for any help.
It seems that ever since the Mu Nu upgrade, the script I have for the daily Day by Day cartoon advances me ahead by about half a day - i.e. the cartoon for today (U.S. Eastern Standard Time) Wed, November 16th doesn't show, but tomorrow's 'toon for Thurs, November 17th shows up as today's strip. Which means there's nothing there because the Nov. 17th strip hasn't been published yet.
I have no idea if it's because the script is somehow set to Australian time. Has anyone else had this problem.
Hey Pixy and co. -- is there any way for you (or me) to add a favicon that'll appear in web browser address bars when people visit my site?
Hey all,
On my old blog (which used MT version 3.14) I figured out where to place the code for a small animated .gif right before the title of each new blog entry (for the main page). But I'll be damned if I can figure it out again on version 2.64!
Any help? :)
Our new hosting company, DedicatedNOW, blessed be their name, will be conducting some maintenance work this Friday.
Midnight to 2AM EDT: Network upgrades - adding new redundant links. They estimate that there will be no more than five minutes of downtime during that period.
2AM to 6AM EDT: Power maintenance. No downtime is expected during this work.
I'll keep an eye on things, but it shouldn't significantly affect us.
As I've mentioned, documentation for LifeType - at least, user documentation - is pretty patchy. There is a LifeType wiki, but it's mostly for developers, not users.
I've started putting up some documentation on the plugins on our new tech support forums. I'll add more as soon as I get a chance.
Everything?
He is now the Skeptics Society Forum. Which is cool, but I've a feeling that isn't exactly the home Jim envisioned. I've tried iit on my blog and on other's blogs. I haven't found any other weird transporter glitches, but I haven't checked everyone.
Want to play with LifeType? Just browse on over to my.mu.nu and sign up for an account. Offer open only to Munuvians and friends of Munuvia. Spammers will be shot.
Oh yes: This is a test system, running on beta code. If we decided to go with it - which is a distinct possibility - it will be cleaned out and reinstalled, so don't get too carried away.
I'm interested in people's thoughts, particularly on the template system. It's based on the Smarty template engine, which is fast, flexible, and powerful, but yucky. It suffers from the Designed by programmers for programmers mindset.
There are a few other things I don't entirely like about LifeType, but it works, it is free (free of charge and open source), and it actually does what we need. There is no other open source system I know of that does. Most (Wordpress, Textpattern, Pivot) don't do multiple blogs at all. Those that do (Nucleus, b2evolution) don't sufficiently separate the blogs, so you don't get the level of control that you are used to with Movable Type.
Movable Type, and even more, Expression Engine, do what we want, but both are sold under licenses that preclude doing what we do.
The best solution would of course be Minx, which is still progressing (albeit very slowly). If I can jump-start things with LifeType, then even if we move to Minx in a year's time we will have had a year of running on a good system.
Oh yes: LifeType was called pLog until a couple of weeks ago, and they're still tidying up the new release of the code, so it kind of says "pLog" everywhere. Don't worry about it; there will be a new version out shortly and we will be installing that before we go live (if we go live).
Hey, y'all. I have NO idea what happened, but I seem to have broken my blog. One minute it was up and running, and the next, just gibberish. I didn't touch it. I swear. Any pointers out there? Please keep in mind that I'm relatively HTML and EVERYTHING ELSE illiterate. Not like I needed to remind, but I'm just saying. Any help would be GREATLY appreciated.
Thanks to MuNu guru Pixy, we've got new servers and new toys to play with. This definitely deserves a celebration, and when you celebrate, you want food, right?
Work with me here.
Rocket Jones is hosting this week's Carnival of Recipes, so It'd be cool if my fellow Munuvians swamped me with submitted recipes, celebratory or otherwise.
It's simple to do too:
1. Post a favorite recipe to your blog.
2. Send a link to your post to Recipe -dot- Carnival -at- gmail -dot- com.
3. Wait for the weekend, when the Carnival is posted and be amazed at the traffic spike!
It's a great way to draw new traffic to your blog too, so consider hosting a Carnival in the future.
Thanks!
How should I go about 'backing up' my blog?
I know that Pixy might/probably does daily/hourly back ups, but I'm thinking that I should be a responsible blogger and do it myself.
Anyone care to leave a step-by-step method for this? And here's another stupid question: when I back it up, I'll be backing it up to a spot on my notebook i.e. a local hard drive, right?
Waiting and twidling my thumbs....
I'm looking for a few people interested in trying out a new blogging system - LifeType.
I've set it up at my.mu.nu. At the moment, blogs there will appear at my.mu.nu/blog/blogname or my.mu.nu/user/username, such as my.mu.nu/blog/trixielicious, my grand-daughter Trixie's new blog, which also appears at my.mu.nu/user/trixie. It can also be set up to provide proper subdomains like trixie.mu.nu, but I haven't done that yet.
It's got lots and lots of features, which makes the user interface a bit confusing, but that can be tweaked. (In fact, it can be changed completely; the user interace is templated just like the blogs.)
This is the closest thing I've seen to what we need here at mu.nu (apart from Expression Engine, which we're not allowed to use due to licensing restrictions). LifeType on the other hand is open source, and we can use it any way we want.
If you'd like to play around with it, leave a comment here or email me.
By the way, it's worth mentioning that (with the raft of plugins I've installed), LifeType's anti-spam features include Bayesian filtering, captchas, forced moderation, open relay blacklists, hidden input fields, trackback validation, the rel/nofollow thing, and host and content blacklists. All of which can be turned on or off for each blog.
In Chinese, Nu means woman
A Nu is a fictional species from the video game Chrono Trigger.
* .nu is the Internet Top-level domain for Niue
Niue is an island nation located in the South Pacific Ocean. It is commonly known as "Rock of Polynesia". Although it is self-governing, it is in free association with New Zealand. This means that the sovereign in right of New Zealand is also the head of state of Niue, and most diplomatic relations are conducted by New Zealand on Niue's behalf. Niue is located 2,400 kilometres north-east of New Zealand in a triangle between Tonga, Samoa and the Cook Islands.
According to the Jargon File, a collection of hacker jargon and culture, Mu (here pronounced "moo") is considered by Discordians to be the correct answer to the classic logical fallacy of the loaded question "Have you stopped beating your wife yet?" [1]. Assuming that you have no wife or you have never beaten your wife, the answer "yes" is wrong because it implies that you used to beat your wife and then stopped, but "no" is worse because it suggests that you have once and are still beating her. As a result, various Discordians proposed "mu" as the correct answer, alleged by them to mean "Your question cannot be answered because it depends on incorrect assumptions". An equivalent English reply would be 'not', instead of 'yes' or 'no', as 'not' is one possible meaning of 'mu'. Hackers tend to be sensitive to logical inadequacies in language (one anecdote relates of Richard Stallman that he once was asked, in regard to the Editor Wars, if he preferred Vi or Emacs; he answered "Yes."), and many have adopted this suggestion with enthusiasm.
Ok this is weird. I made an update to my main index template and the change shows up on http://www.sacramentorepublicrat.com but NOT on http://www.sacramentorepublicrat.mu.nu
Though things looked dicey around 2 AM when Movable Type absolutely refused to work.
The new servers are quite zippy; even a full backup can't do that much to slow them down. Which is good, because we are moving to a twice-daily backup schedule.
There are probably still some odd corners of misbehaviour. If you do trip over something, post or comment here or send me an email.
And yeah, I know about this. I'm getting the latest updates to the portal software to see it that helps. Otherwise, it's some tedious SQL hacking for Pixy.
Is your little corner of Munuvia unexpectedly blowy-uppy following the move? Something doesn't seem to have come across quite right - or at all?
Leave a comment here.
As far as I know, all the blog stuff should be working. There are some problems still with the Munu portal which I need to sort out, and some of the automatic background thingies aren't running yet. Oh, and trackbacks are turned off; I'll turn those back on shortly.
Anything else, give me a yell, and I will poke at it with my Stick of Poking +1, +3 against databases.
Moved everything across to new servers. All happy and good. Load up Movable Type databases, all goes okay.
Start Movable Type. Refuses to connect to database. Oh, oops, MySQL 4.1. Known problem.
Change database authentication settings. Now we get a different error.
Change configuration. No good.
Change user settings. No good.
Change Movable Type settings. No good.
Move database to local host. No good.
Do all of the above at once while crawling the web in search of anything that might give me a hint. No good.
The database works. I can access it through the MySQL command line; it's just Movable Type that rejects it.
Dig out Perl DBI docs and hack Movable Type database drivers. Good.
But not what I wanted to be doing at 3 o'clock on a Sunday morning.
Does anyone know of anyway to spell check in MT 2.64? I tried writing my posts in Word and then copy/paste but then the post is all messed up when I paste it in to MT. Any ideas other than a dictionary??? I tried the IESpell but I am on Mozilla.
Thanks,
SR
We got 1.3 million trackbacks over the past 30 days, and 550,000 comments. About 99.8% of trackbacks and over 90% of comments were spam. Over 99% of the spam got blocked, but that still left a lot of cleaning up.
You can turn off comments and trackbacks for specific posts in Movable Type, and we have set it up to do so automatically. Unfortunately, this doesn't do anything as far as I can see. It certainly doesn't stop spam.
So I've forcibly closed comments and trackbacks on all posts more than sixty days old. When I say "forcibly" I mean I went and hacked the Movable Type comment and trackback routines directly.
This appears to work. We still have Fluffy on the job, and we still have MT Blacklist backing up Fluffy, and hopefully this new measure will take us up to blocking 99.9%. We'll see how it goes.
This weekend - specifically, Saturday the 12th of November - we will be moving to our new servers. I expect this to take four to six hours, assuming that nothing goes horribly wrong. And if something does go horribly wrong, I will probably just postpone it until the following weekend. During that time, you won't be able to post, readers won't be able to comment, and we won't get any spam.
Every cloud, silver lining. :)
Our current server, Mew, is getting old and tired. It's a dual Xeon 2.4, something that hasn't been made for years. It was bright and shiny in its day, but lately the wheels have started falling off.
Our new servers - yes, servers, plural - are called Kei and Yuri. Each is a dual-core Pentium D 2.8, with 2GB of memory. Individually they're 50% to 100% faster than Mew, and they're connected via a private network, allowing them to co-operate on dishing out your bloggy goodness.
Yuri will be running Movable Type 2.6, which accounts for the bulk of the munu blogs, and a majority of the workload overall. Kei will be handling the various forums, the Movable Type 3 blogs, and the various WordPress blogs and other sites. Kei will also be handling the databases for all the applications, taking some of the work that Yuri would otherwise be doing for MT 2.6.
Each server is backed up to the other one twice a day, so even if we have a complete system failure, we can be up and running again quickly.
In addition to the shiny new hardware, we have shiny new software. Newer versions of Linux, MySQL and CPanel for starters. Fantastico, which died recently after an upgrade, is back. XController has arrived to make CPanel pretty.
Once that's done, I'll be addressing our blogging software. We have to move off Movable Type; we have no choice in the matter. However, no-one will be forced off MT before they're ready. The new servers will be able to cope with the existing users just fine; the problem is that further growth will bring serious problems. We're already the largest MT installation in the world, and worse, we are stuck on version 2.6 due to licensing issues.
I have a plan - but more on that later.
For the last three days, everytime I try to send a track back it wont go through. It doesn't matter if it's a MuNu site or not? I'm just curious if others are having this problem or if I'm doing something horribly wrong that is going to result in the internet exploding all over my desk.
I've been getting loards of trackback spam lately-are we any closer to a magical cure to remedy this? I'm a little tired of seeing 10 comments trackbacks, all letting me know about blackjack or animal porn.
Strangely, all thr trackbacks seem to come from the same posts, some very old some more recent.
Thanks!
I've gotten a couple of trackback spams loaded to my site. Normally I get notified via email when someone sends me a trackback, which means I can find the post and delete the spam. These didn't send email, and I don't know of a way to hunt them down and delete them.
Any ideas?
How can someone leave a comment when the IP address they're using has been banned, for quite a while even?
I am assuming the recent spam attack is causing the error message one gets when trying to comment? The message reads:
An error occurred:Invalid [] range "s-b" in regex; marked by <-- HERE in m/ http://mp3-audio-search.tripod.com| http://parrot.stopsneg.com/parrot-picture/|"http://customordered.com/penis-enlargement|([\w\-_.]+\.)?(l(so|os)tr)\.[a-z]{2,}|(babes|porn|boobs)[\w\-_.]*\.[a-z]{2,}|(cialis|meridia\b)[\w\-_.]*\.[a-z]{2,}|(diet|penis)[\w\-_.]*(pills|enlargement)[\w\-_.]*\.[a-z]{2,}|(free)[_.\-]?sex([\w\-_.]+)?\.[a-z]{2,}|(levitra|lolita|phentermine|viagra|vig-?rx|zyban|valtex|xenical|adipex|meridia\b)[\w\-_.]*\.[a-z]{2,}|(magazine)[\ at /home/blog/public_html/cgi/extlib/jayallen/Blacklist.pm line 3098.
FYI.
