Some of you might remember, back about 15 months ago, when we were on the old old server, it used to take something like 30 seconds to leave a comment. That was before we moved to the new old server, and before we moved from Berkeley DB (boo hiss!) to MySQL (yay!)
Back then, I started work on a blogging program I called Minx. Minx was written in Python and used a database called Metakit, and it was really really fast, at least until you started getting a lot of posts and comments, whereupon it ate all your memory and died.
So much for Minx 0.1, which bit the dust around January 2004.
Minx 0.2 re-arranged the database a bit, and came along in April 2004. However, by that time we were on the new server and things were relatively speedy again, so there was much less urgency to Minx (and I had a lot of work to do with the migration, and a swarm of new MuNus who joined us around that time). And as it turned out, Minx 0.2 also, once you got past a certain point, ate all your memory and died.
Minx 0.3 came along in December, just before my brain got eaten by hackers and spammers, something that lasted through January and February. March I basically went to bed and played Final Fantasy X and Rumble Roses.* This month I dug out Minx and started playing with it. Minx 0.3 was the best yet, but after a certain point, it ate all your memory and died.
The problem in all three cases was the Metakit database, which is extremely fast and flexible, but has a nasty habit - if you push it too far - of eating all your memory and dying.
And so, a couple of weeks ago, I came up with this startling insight: Minx might work better if I didn't store all the data in Metakit.**
And so Minx 0.4 was born. And it's even faster and better than before, and it still works when you throw - What was it? Right. - when you throw 4000 blogs with a total of 4 million posts (and 20 million comments) at it. At that point, my home machine was starting to struggle a little, but certainly hadn't eaten its memory and died. Minx 0.4 uses Berkeley DB (boo - I mean yay!) only it uses it completely differently to Movable Type, so that it actually works.
Soon - maybe even very soon - the Minx Dev Blog and Ambient Irony will be making the Great Leap Sideways and switching to Minx. You'll know it's happened when trackbacks suddenly stop working. Well, unless they come back five minutes later, in which case it's just Fluffy the anti-spam watchdog at work.
Oh yes, there was a point to all this. Munuvians, if you're interested in the future of your blogging platform™, hop on over to the Minx Dev Blog and take a look around. If that makes your brain hurt, wait for the test release, coming soon to a server near you.
* I'm not a sexist pig, I just play one on TV. Um, literally. At least it's better than Dead or Alive: Extreme Beach Volleyball.
** There's slightly more to it than that. It involved a full understanding of Metakits on-disk and in-memory data structures, and realising that there was no work-around for the problems I was encountering. No matter how I split up the data, as long as I stored it all in Metakit, it would eat up all my memory and die.
[Also posted at Ambient Irony]
Posted by Pixy Misa at April 27, 2005 06:06 AMIt's looking pretty slick Pixy, can't wait to try it out. Oh, and by the way, please, please, please ....... PHP not CGI. I like PHP so much more/better/yay!
Quick and offtopic. My regular readers and I have noticed that the MuNu anti-spam feature censors the word sociali$t because it contains the bad/evil/wicked string ciali$. Now, I don't want spammers coming round anymore than the next guy, but ..... it's kinda tough for commenters on a political blog to not be able to use the word sociali$t and try to remember to screw up the text on purpose so that the comment goes through. Is there something that can be done about that? Can I help somehow? Thanks in advance.
Oh yeah, before I forget, MuNu is awesome, thanks much, I'm having a blast.
Posted by Eric at April 27, 2005 07:59 AMI tried a couple of regular expressions to catch cialis but not socialist, but they didn't work. If there's a regexp doctor in the house, help!
As for Minx, I'll probably be going PHP. The Minx back end is written in Python as a persistent, multi-threaded server. The front end won't actually have much to do, just taking requests and feeding them to the server, and reporting the results back to the user. There are nice things about PHP - it doesn't have to initialise an interpreter every time; it has built-in session management. Also, Minx will generate PHP pages for output, so it probably makes sense to use PHP for the interface layer so the two can be melded in horrible and mind-twisting ways.
Posted by Pixy Misa at April 27, 2005 08:44 AMPixy - if you're looking for guinea pigs to test out the new stuff, I'm more than happy.
Posted by Tilesey at April 27, 2005 02:42 PM