ASCII by Jason Scott

Jason Scott's Weblog

Outage —

To ensure that things would run smoothly at the demoparty that I was hosting, I worked for no short time on a Partymeister server. Partymeister is basically demoparty maintenance software which allows for ease of voting, tabulation, messaging between attendees and a dozen other functions. It’s really cool, but required a lot of careful, quiet setup and planning. Therefore, I set a server up locally and spent a couple weeks customizing it.

I packed up the server along with a bunch of other junk into a rented van and drove single-handledly from Boston to Cleveland in a single 10-hour trip. I do not recommend this course of action to anyone. Bring a friend. By the end of the trip, your road has turned into a cotton-candy path with unicorns on either side trumpeting “I’ve Got a Lovely Bunch of Coconuts”.

After arriving and napping a tad, I booted up my laptop and instinctually checked a couple of my websites at home. Some worked, some timed out. Weird. I checked further; all the timed-out ones were on the same machine. Weeeird. And that machine was right next to the one I had unplugged downstairs and brought with me….

Slowly I turned, inch by inch, step by step…

Yes, that’s right, I’d taken the machine that hosted cow.net, bbsdocumentary.com, getlamp.com and a host of other sites offline, loaded it into a van, and driven west with it for 675 miles.

It was now in a hotel room with me instead of in my basement.. and the machine with those weeks of work was up, just fine, running in that same basement!

This is an acid test for how one approaches their websites, data, and machines. Could I recover, even though I’d removed a vital part of my setup, especially the one that makes money?

As it turned out, I was able to recover by taking the now-unused IP address of the server in my hotel room and then attaching it to the server that was still running, and then transferring my website data from the “staging server” that isn’t directly on the internet onto this now-more-loaded server. At this point, I made another delightful discovery: I kind of set up the webserver wrong and had “Virtual Name Hosting” done in a way that never should have worked but which sort of did. This required more editing of the configuration files. When I finished that, I got distracted (I was at the conference, after all) and forgot to put it back up. So all in all, I gave all my websites a pretty good punch in the gut.

I was able to recover most of the Partymeister settings as well, although that was relatively painful.

I mention all this mostly to make note of the fact that I was able to recover relatively quickly, even with the complete and total physical removal of a server and myself from the basement with my machines. This was because of my approach of keeping data in several places, and being able to transfer them between each other. I know enough about the webservers and the operating system to “fake up” a second machine on top of another, to get it all working.

But isn’t it interesting that we’re still at a point where a person still can so easily cause catostrophic failure to their own sites, their own web services. If I’d been hosting everything major with a hosting provider, this might never have happened.

But to do that removes, utterly and finally, that feeling of strength and self-sufficiency that I feel when I know I am 100 feet from my websites most of the day. This is a strong, powerful feeling and it makes me feel good about myself. But the apparent price I pay is that being in a rushed mode or doing something new and unusual could doom it all for days on end.

Every once in a while, another item, another aspect of things I used to do by myself so long ago are being turned into pennies-per-day services I could go to a hundred places to have done properly. It’s obvious where the trend is going. And often, they do it better, unless you consider being able to hear your disk drives whirring to be a vital part of the administration process.

If you do, my advice is to go record the sound now, and leave it as a running audio loop in your mp3 player while you work.

Based on this whole unexpected, stupid outage, I’ll probably host that audio loop somewhere else.


Categorised as: Uncategorized

Comments are disabled on this post


One Comment

  1. beoba says:

    What sort of bandwidth do you have going for your servers? How much is it?

    Comment bug: Clicking “preview” is apparently counted by as a post; a quick write->preview->post brings up a “you’re posting too fast” error.