Analyze things long enough, and you end up with general statements that are so far from what people might be interested in that it loses meaning for everyone. That’s the risk one takes with criticism that keeps looking skyward or at the “big picture”. But what the hell, it’s Thursday.
If, in fact, you were a kid at some point, you probably built something. I am talking less about the Lego/Erector Set/Capsela sort of pre-formed kit, but where you, you hapless little bastard, go outside to a nearby location and attempt to make something out of parts and then use it. A good example that comes to mind is a treehouse. In Kid Engineering School, you learn (innately) that a treehouse consists of a bunch of nearby trees and you grabbing some 3/4″ nails and some spare wood of dubious quality/usefulness, and then you nail it all to a tree and next thing you know there’s sort of a thing up there and then you sling some more wood across it and tah dah, treehouse.
Some of my fondest, wonder-filled memories are of treehouses, believe me. I have memories of ones I built that were 6 feet off the ground, and I recall, with great awe, one particular treehouse (located here) which was easily 50+ feet in the air and required careful climbing of a tree (with convenient occasional footholds) and afforded you a wonderful view of the river it hung over. Like nests built by particularly large birds, my neighborhood was riddled with treehouses. But this is critical: They were built very, very poorly.
If I think back to these things, it’s amazing some of them didn’t kill us. Nails driven haphazardly into boards is not how you build a treehouse you intend to stick around; bolts and drills are the order of the day, along with concrete pilings if you can build it closer to the ground and still have it look cool. Minimum engineered effort would call for at least a tiny bit of weatherproofing (most treehouses were simple platforms) to prevent your nails and the rest from rusting out.
But who cares, you’re a kid, you can live forever, and if the whole thing explodes when you all step on it, it’ll just mean a few silly editorials in the paper and all the parents locking the hammers in their toolsheds out of reach.
Your treehouse is cool. It is built quickly, happily and with an urge to get the job done and start having fun. But it will fail under load.
Extrapolate this into things both material and immaterial. Your high-school club, that conflagration of students who meet after school for some reason, be it journalism, acting, writing… this is assembled with some basic goals, and the school often requires an advisor of some sort for it to be official, along with some basic rules, and positions pre-defined. This helps the club in several ways, gets it funding, makes it so there’s a voice of authority available to handle a bunch of possible disputes, and a clear power structure to work in. This also fails under load. But, in this case, the load capability is much better. It takes something really intense, like a controversial play or someone being an intense right asshole and being the buddy of the advisor and so on, before crap just totally falls apart and the club dissolves, or has to reform in a way to avoid ass-boy.
Organized sports, which are a club as well, are even better designed. (I’m working in generalities to prove a point; be gentle.) You have uniforms, set rules of engagement (the sport’s rules), often a bunch of guidelines of how the entire enterprise must function, an advisor whose absolute job is to maintain the organized sport team, and specific from-the-school funding for this activity. It also has wide and distinct promotion of this sport, an imbued sense of pride and representation in members, and a host of other safeguards in place.
Now, obviously things can still go entirely south, but the load must be incredible: an enabling parent who is allowing their sports-going kid to host wild parties, a member dealing drugs for an extended period of time without being caught, a solid intense scandal that goes on for some time before being revealed, or a goddamn gigantic sack of mad cash being dropped on players’ heads to either encourage them to cheat or to lure them into the college or professional leagues upon graduation. Whereas the poetry club can probably break up if one person’s a dorkface, it would take a concerted, quality effort to really get a sports team to be completely wiped off the map at a school. Or lack of funding, whichever comes first. Both fail under load, but the loads required are radically different in nature.
In the case of the treehouse, the after-school club, and the sports team, they all can fail under load, but they can all be re-built better. Just because your club dies doesn’t mean clubs are bad and shouldn’t be done, just that next time you should think out the consequences of certain choices. Just because a sports team is poisoned by the fact that the quarterback’s a raging alcoholic doesn’t mean it isn’t possible to put safeguards in place to prevent this from happening again, safeguards that don’t ruin the entire experience of the sport for the sake of this safeguard. It can be done. It is sometimes difficult and a real pain, and it changes the nature of things, but it can be done; the maximum capacity of stress can be increased.
And just so we’re not thinking I’m going in one direction with this, I also realize that the greater load capacity you build into a project, the more likely that problems of a great and terrible nature can therefore flourish because the organization is strong enough to withstand it. The load can be good (a lot of people want to be involved) or it can be bad (from the ground-up, the entire organization is corrupt and so the ethics framework is decorative and leads to a scandal of national proportions down the line when the issue comes to a head).
So, yes, I’m speaking here, ultimately, of a lot of web projects, a lot of websites. Certain aspects of the environment, ones in which binary measurement comes into play (is the site up? is the site down?) are very quickly handled by technological solutions. Mirror them. Get more machines. Build in load balancing. Optimize that crap-ass code. Rate-limit stuff. After a while, and this is the case in some of the biggest sites, you will successfully have a site that, when accessed via web browser by anyone on the internet, will be there. You will have properly engineered for load, and the pure mass of people will not activate this binary condition (is it down, is it up).
We were faced with this issue, web wise, within almost days of the spread of browsers and the availability of known web servers. There were so few sites, that when they got listed on the “What’s New” page, they would absolutely die, like throwing a hamster into a fire. For context, here’s what one of the What’s New pages looked like. Imagine the entire web-browsing public finding out about the one new website on a given day, and what that website/website machine had to endure. It was ugly. But, of course, this led to better development of web server software, improvements in resource management, improvements in engineering. The goal was clear: make this bastard not crash, and so it was a matter of throwing enough programming skill to ensure that could be achieved. Huzzah. Nowadays, a fairly simple machine can sustain many hundreds of thousands of accesses in a day, thanks to over 10 years of steady, intense programming by many people.
So when I go crazy over Wikipedia and selected other sites, this is why: they fail under load. No-one who knows what they’re talking about declares that a wiki, in and of itself, is an evil or wrong thing. If you don’t want other people editing it, you can restrict access to as many people you want, including just yourself, or nobody. If you want people to register beforehand, you can. If you want it to only cover a small, manageable subject, you can. It’s just software after all, and if you run into problems, you can take a number of precautions and energy and, through engineering, beat back the load-bearing problems, move on. It’s a matter of recognizing there’s a problem, and working on it.
Taking it further, nobody who knows what they’re talking about declares that any “Web 2.0” application, which seems to be getting defined as “people add junk to your site for free”, is itself evil or bad or beyond hope. Right now, there’s a lot of noise going around because of a guy named Andrew Keen, who wrote a book called “Cult of the Amateur” where he kicked a lot of current internet technologies in the yam-bag. After spending some time reading up on his viewpoint, I can tell you that I consider some of those kicks to be properly and well-placed indeed, but they are unfortunately swarmed in an endless random cloud of nut-kicks which unfortunately diminish his arguments down into the realm of foolishness. It’s one thing to observe that there’s a lot of poverty in the world; it’s another thing altogether to start saying that the poverty is due to over-investment in space programs or the prominence of monkeys in warm climates, or that the solution to this poverty issue is to set Belgium on fire. So, unfortunately, a person who makes some good points reveals, in his words, that he has only done so because he carpet-bombed online reality and declared victory.
In the reactive, twitchy way that people who have a lot of time, energy and money invested in what are called “Web 2.0″/”Internet” endeavors can always do, Keen has been set on fire and thrown into a larger, more deadly fire. All well and good, and the debate is entertaining if nothing else. But the whole issue is framed wrong, in my opinion. Instead of going “there should be no wikipedia” or “there should only be printed sources” or “there should be credentials and limitations on sources”, which are all just straw men, the real argument is “Where does Wikipedia really screw the pooch? What steps should be taken to avoid unwarranted pooch-screwing?”
There’s the “idea” of many websites and there’s the “actual” website. I’ve sat in multiple panels now, where people are arguing with me, and they keep mentioning the Wikipedia in their heads, which very few people actually connect to, and I’m arguing about the Wikipedia that exists, which a massive amount of people connect to. There is a huge gap there, and it’s growing wider over time. And by simply considering its problems actual problems instead of wayward background noise in a deafening and beautiful cacophony of expression, maybe some of those issues can be fixed. I’ve gone into many of them before; no sense in making this longer than its already excessive size.
Youtube fails under load; its comments sections are poorly designed and ruin a lot of the discussions for almost any video by coating them in profanity and chaotic viciousness. Flickr, meanwhile does not currently fail under load; an awful amount of effort has been made to contain asshattery and there are people whose jobs are contingent on ensuring this. Similarly, Amazon does not fail under load; allowing reviews and writings (and yes, the system can be gamed quite readily, but there are near-instant ways to suss out one-time PR flack accounts and often you can read the book itself for a few pages and make an informed judgement). Amazon works constantly to fix potential load-breaks, and it shows.
This is an engineering problem, as ubiquitous and critical as those first wayward-driven nails in your tree fort and branching up into the most visited online sites in the world. If you refuse to recognize its importance, if you think of people issues as minor annoyances towards a glory of a newly minted era of freedom midwifed by connectivity, your project will fail under load. And yes, it will likely be rebuilt, either from scratch or as a major upgrade, but doesn’t it make sense to be a little proactive? Just a little?
Because the load is never going to get lighter.
3 Comments
