OSCON Day 0: Scalable Internet Architectures

One of my favorite presentations from last year was Theo Schlossnagel's presentation on Whack-a-mole, so when I saw him giving a full tutorial on scalability this year, I had to go and check it out. And this year I wasn't disappointed either -- Theo presented a solid tutorial that exuded his practical experience in this field. Of course its impossible to summarize four hours of a tutorial in a blog entry, so I'll try to summarize Theo's three simple rules that he applied repeatedly in his presentation:

1. Know the system you're trying to scale. It is important to carefully study the system you're trying to scale. Any time that a (sub)system is glossed over and not carefully examined for potential pitfalls, some hidden gotcha that was not considered early on in the planning phase, will come to to bite you later. The sooner you understand all of the details of a system that needs to be scaled, the easier and cheaper it will be to address the problem. Failing to spot a potential problem early on drastically limits the options and increases the cost to fix the problem. For example, a problem that is identified and solved in the design process will be cheap to address as part of the whole process. Fixing the same problem much later, say in production, can be catastrophic. If the entire system is down and business has stopped, the cost to the company astronomical both in dollars and in customer perception. To avoid this understand all your problems fully and never ignore a potential problem.
2. Complexity has costs. You can only have a certain amount of complexity in a system before the system becomes impossible to maintain or scale up. You should carefully examine new technologies before you decide to embrace them in your project. If you create a complex system comprised of too many disparate technologies, you limit your overall scalability. For instance, scaling a system that requires dozens of specific versions of software may end up having conflicting requirements. If package A requires package B version 1.2.3 and package C requires package B version 2.1.1, then you need to resort to fancy footwork to make the system work. And the more fancy footwork is required, the more complex and error prone your system becomes. I really have to agree with Theo here -- I think that complexity should be treated like a scarce resource -- in scalability and many other aspects of software engineering.
3. Use the right tool for the job.You should never blindly throw pieces of technology at your problem. Traditional computer science teaches us to reuse existing pieces of technology, rather than constantly inventing new pieces of software. However, throwing a familiar off the shelf tool at the problem may give a poor result, whereas some elbow grease and a custom piece of code may solve the problem much more efficiently. If knowing the system you're attempting to scale is important, so is knowing the technology (and its limitations) that you're going to throw at your problems.

Theo kept reaching back to these points throughout his talk -- other points he stressed repeatedly were the use of a clearly established release procedure. Unless the team that rolls out new software onto production servers has documented procedures to follow, mistakes will be made. And as systems grow in size, the likelihood of fatal errors increases dramatically. To spare yourself from this fate, document your release procedure and use a version control system to keep track of everything you do. Again, seems like common sense, but often it is not.

Robert Kaye is the Mayhem & Chaos Coordinator and creator of MusicBrainz, the music metadata commons.

oreillynet.com Copyright © 2006 O'Reilly Media, Inc.