So, want to get a quick chill down your spine? Have a huge database in MySQL where most of the tables are using the InnoDB storage engine? Running on a 32-bit version of Linux on a server with 4GB or so of RAM? Try this neat little trick.

Modify your my.cnf file and set the innodb_buffer_pool_size variable to a value north of 2GB. Sounds like a good idea, right? The more memory you allocate to the InnoDB Buffer Pool, the faster InnoDB table access will be. Cool. Change that setting and save that bad boy. Jump over and do a mysql restart.

MySQL shuts down normally.

MySQL starts to come back up, but seems to take a while. Hmmm…. After a minute or so, it completes startup and says [ok].

Except none of your data is there. At least, no tables that use InnoDB are there.

AAAAAAAAAAHHHHHHHHHHH!!!!!

No warnings, no error message, no tables. And your blood pressure is at about 215/180.

After quickly issuing a mysql stop, editing the my.cnf to restore the original settings, and doing a mysql start, you’ll see the tables back and all is well.

It should take about 20 minutes to calm back down.

It’s not like I’m speaking from experience or anything…

update: Just so this doesn’t hang out there… the problem was I attempted to allocate more than 2GB of RAM to a single item, which causes problems on most 32-bit versions of Linux… Had I been on the 64-bit version of RedHat, I probably would’ve been fine. Since I wasn’t, the InnoDB storage engine didn’t start up, and therefore all tables using InnoDB didn’t show up. No data loss whatsoever; I just found it “interesting” that the startup of MySQL didn’t warn me that InnoDB had choked and died. phew.

Viewzi.

we’re launched.

go there.

Viewzi is set to launch our Public Beta starting Monday. This week’s been full of last minute cleanup, bug fixes, and features squeezed in at the last minute. But we’re nearing the finish line and at 7:00PM Monday night, we launch. What this means is anyone will be able to search using Viewzi without having (or creating) an account. We call that “taking the wall down.”

Mind you, we’ll still be in “beta”. We’ve still got lots to do, bugs to squash, features to complete and roll out, etc. Lots of stuff to make Viewzi an even better, more fun way to search.

It’s been quite a ride at this point, and I only expect it to speed up. As a team, we’re really starting to get our legs under us and start hitting on all cylinders. And the feedback we’ve been receiving is quite flattering; very positive and constructive. We didn’t expect as many people to “get it” as apparently have over the past 5 or 6 weeks, and it’s really great to hear. We’re very excited about where the product is headed, and anxious to show the world what we’ve been working so hard on.

Well, 71 hours to go (really less, more about that later). Back to work.

Viewzi. Changing the way you look at Search.

Well, Viewzi made the news tonight… the 10:00 CBS local evening news to be exact. My part even made the cut. It was about 45 seconds on the 7:30 sister-station news (channel 21), and then a full 2 minutes (with two “coming up” teasers!) on the 10:00 on the Dallas CBS affiliate, channel 11. It repeated at 12:30. Very cool.

And we got hammered.

Within 15 minutes of the story on the 10:00, we had received several hundred new user accounts, and the queue manager for the screen capture subsystem was smoking. Almost literally. It never crashed, but it bogged down quite a bit and is still working out from under the deluge.

I’m in the middle of rewriting the queue manager (the part of the code that delegates screen capture tasks to the many servers we have capturing website images), but I haven’t finished it yet and rolled it into production. So, we’re still running on the old code, which is having a hard time keeping up. I think the architectural changes I’ve made to the system will allow it to scale much more gracefully, but right now it’s kinda falling on it’s face.

The rest of the system has held up beautifully, though. For a little while there we were executing almost 35,000 SQL queries a minute, and the database server hardly broke a sweat. The rest of the architecture is sound. But screen capturing is a nasty business. Once we roll out the new code, we should be able to literally have hundreds of screen capture servers running to keep up with the load.

The feedback has been very good, overwhelmingly positive, and we’re still seeing new users sign up at a faster clip than we have since we went on the air about 5 weeks ago. It’s pretty exciting, and crazy at the same time. I just hope I have a little more time to work the kinks out before we hit the national news.

If you’re into software development, this will make a lot of sense.

Not exactly what you might think the title implies, but The Perils of JavaSchools by Joel is an excellent read. All I can say is Amen.

…what I’d like to claim is that Java is not, generally, a hard enough programming language that it can be used to discriminate between great programmers and mediocre programmers. It may be a fine language to work in, but that’s not today’s topic. I would even go so far as to say that the fact that Java is not hard enough is a feature, not a bug, but it does have this one problem.

If I may be so brash, it has been my humble experience that there are two things traditionally taught in universities as a part of a computer science curriculum which many people just never really fully comprehend: pointers and recursion.

The thing that makes me nervous is that there is an ever decreasing number of people (especially coming out of today’s CS programs) that actually “get it”. And by “get it” I mean people that actually understand the inner workings of the systems they build upon. How the languages that make things easy for them actually translate to the ones and zeros that do things.

The attrition of those individuals (people of my generation and earlier) will turn out to be one of the largest problems the computer science field will face over the next 10-20 years.

Wow. Looks like Ray Ozzie has switched sides, big time.

Microsoft on Thursday said it will acquire Groove Networks and make Groove’s founder, Ray Ozzie, chief technology officer at the software giant.

I knew that Groove was a major Microsoft player, but wow, didn’t really expect that one. Guess the money was right.

As I suspected and mentioned, Google suggest is getting a lot of play around the net. A Notes developer has blogged a Domino implementation (perhaps I should send him my code)… Another developer has blogged a complete dissection of how the Google version works… And, lo and behold, Apple has an article describing the IFRAME/Javascript technique. They call it Remote Scripting… I kinda like backchannel conduit better, but hey, I’m not a marketing guy.

Glad the technique is getting some airtime. It’s pretty handy.

The thing I find funny is how everyone is so amazed at the speed. But think about it. You’re doing a single trip back to the server for a file that’s probably less than a couple hundred bytes. That’s a single IP packet each way. Things are fast when you’re not loading a page with a bazillion graphics and whatnot on it, causing countless roundtrips to the server. Couple that with HTTP 1.1 holding the socket open and communication is pretty fast.

There’s a lot of hay being made around the Internet about Google’s new Suggest feature. Just so we’re all on the same page, Google Suggest will automatically return search results as you type in the search box. Of course, for this to be usable, it has to be unobtrusive, which means no full-page roundtrips to the server. Google’s implementation does this, and it works quite well.

Yes, it’s cool. But the demonstration of technology used is not new. Just had to get that out there, because a lot of people are saying WOW like it’s the latest sliced bread.

The technique used to accomplish this bit of slight-of-hand is the XMLHttpRequest set of routines, introduced in Internet Explorer. On other non-IE browsers, they appear to be using what I call “the IFRAME Conduit” technique. Both are very similar, with the former obviously a little more sexy since it uses XML. But both are equally valid and accomplish the same end-result: the ability to refresh data on a portion of a HTML page in the browser without a roundtrip to the server.

I discovered the IFRAME technique while working on a project for a large client that needed a very interactive interface in the app we were building. I won’t use the word “invented”, although at the time I had never seen the technique used elsewhere, and since have only seen it used in one or two other instances besides this entry from Google. Later in the project, for a slightly different use, we utilized the XML flavor. Both worked well, and achieved a level of functionality within the browser interface that most web apps can’t dream of. I’ve since used the IFRAME technique on a number of other apps. It turns out to be really simple.

Basically, on your page, create an IFRAME of almost no dimension (1×1). You can’t make it invisible, because on IE for the MAC, that renders it unusable for some strange reason, but you can make it borderless and at the bottom of the page so that the user never sees it. Then, when you need more data from the server, just set the document.location for the IFRAME to a URL on your server that will take requests. The code on the server that you execute returns a simple generated javascript who’s responsibility it is to populate various fields or do other DHTML things on the IFRAME’s parent document. That’s it. Round-trip to the server and no page refresh.

You can use this for all sorts of nifty interface tricks, and without having to send down the entire weight of the surrounding page, it’s very quick.

XMLHttpRequest or IFRAME? Dunno. The IFRAME is a bit more cross-browser friendly, but I’d imagine that a variant of XMLHttpRequest will make it into the ECMAscript spec at some point. Personally, I think XML can get a bit bloated; I can programatically create very efficient and small javascript in the response, so I typically go that way.

Kudos to Google for a very public implementation of a nifty technique.

Someone wanna tell me why iChat, when sitting completely idle (only logged in to Rendezvous, not logged on to Oscar), takes up 256MB of virtual memory? And that’s after I restarted it… earlier today it was eating up over 350MB of VM and over 120MB of real RAM… while not doing anything.