For the short time it was up, torrents.textfiles.com was pretty popular. The idea behind it was good, and still holds: every once in a while, people get really excited and try to download all of textfiles.com. The site is many gigabytes (assuming you go for some of the sub-sites as well) and so this takes a while. Unfortunately, the massive number of small files and the fact that some people use some pretty poorly-working clients to attempt massive parallel transfers of the site means that my bandwidth gets nailed.
Here's what nailed bandwidth looks like. The blue line is the outgoing, and the green line is the incoming. The reason the blue line is sometimes flat at the top is because they have maxed out the T-1 that TEXTFILES.COM and the related sites run on.
There's really no need for this to be happening. Sometimes, people are pulling things down and they find the throughput to be less than enjoyable, but that's life when something's free; that's why I'm going to be becoming a non-profit organization this year.
For people who want "THE WHOLE SHEBANG", a bittorrent connection is the obvious solution. Then everyone who wants multi-gigabyte hits (actually, the main TEXTFILES.COM archive works out to about 700mb) can just grab it that way and help each other along the way.
The thing is... bittorrent seems to have some major issues with the way I want to use it. So I'd like some help.
The way I want to use it, Bittorrent would replace an HTTP link or an FTP link. You would go to torrents.textfiles.com and there would be a set of files offered, where you would grab the torrents and then you download them. I would be the seeder, I would be offering up the new seeds, and you'd choose from them.
Basically, I want torrents.textfiles.com to be almost exactly like LEGALTORRENTS.COM.
The thing about a lot of tracker/bittorrent sites is that they are very simple to use from the user-side. If you have the client installed (and really, it's on the level of Flash to install a bittorrent client) then you click on the magic torrent link, it asks where you want to dump the file or directory, and "off it goes". There's a number of clients of varying usefulness, but they all kick in nicely. The issues are on the server side.
Part of it is the webpage. I like how it shows you statistics of how much has been transferred and how many people are using it. I understand this tends to require PHP, because that's how many people think now, and that's OK. I do not think MySQL is OK.
I understand that I have to run a tracker, but I do not understand why trackers are so difficult to deal with in UNIX; they all want to shoot stuff to the screen and I can't find a way to do an easy tracker startup script. It's all hopelessly complicated for the relatively easy thing I want done (start a tracker, serving these files).
If someone could help me, I would appreciate it, and textfiles.com would be a better site for it.
Update: A number of folks have assisted me, and my own research was redirected in several helpful ways.
Ultimately, the solution I am going with is to use the original python bittorrent client, with a script that redirects the logging output to disk, and a startup script which does all this setup on reboot. For the nice torrents.textfiles.com webpage, I will be using a nice simple PHP script called Torrent Trader Lite, which does all I need it to do, without requiring MySQL or a bunch of other stuff. I will be creating torrents shortly. Thanks again.
As I promised, I would be updating the BBS Documentary site much more frequently, to make up for the production falling behind schedule. Here are the last three updates on that site, put into one location for the people keeping track of the production view syndication feeds and the usual trickery of weblogs.
January 23, 2005
The Eye of Doom has signed off on three of the episodes.
Who or what is the Eye of Doom? A friend of mine going a ways back who has an amazing talent for picking out all the slightest imperfections and oversights that crop up when you're working on a large-scale project like this documentary. Maybe a boom mike gets into a shot, or a filter is applied poorly, or the framing could be much improved. It takes an intense, detached, involved eye to do so, and he has it, so I call him the Eye of Doom. It's Doom because of course any major errors he finds means lots of work for me, but better now than after the DVDs are pressed and I can do nothing about them.
Every time the Eye asks me about a specific setup, like "Why is this shot like this?" or "Why do you cut from here to here?", I can often reply with a paragraph or two, specifically about this choice. This is a sign, perhaps, that I've spent a whole heck of a lot of time editing.
I kind of wish I could have done some little project before this one, some easy subject with a tiny but clear cast of characters that would have been a few months of work before this insane monolith of a project, but there you go. Every task in this project is really darned huge, and very time consuming. Just subtitles have been a big deal, not to mention the commentary tracks, or really, even the selection of music for the soundtracks. Big, I tell you.
Mostly, I am fixing up sound. Sound has turned out to be the big deal with this project; with interviews conducted in hallways, computer rooms, conventions, outside, and even an in-use band rehearsal space, we have a lot of background noise on some of these tapes. I've been able to remove a lot of it, but it takes time. There's also some wide variances in how various people spoke, so I'm working with that as well. This is a case where I made a specific choice and trade-off a few years ago; I chose to do the travelling alone, instead of with a crew, because I knew that I would kill a crew or lose some good friends to take the trips I took. So with just me alone, there's only myself to look through the lens, consider the shot, set up the sound and take readings, and then conduct the interview. During the interview, I check the tape, check the lights, make sure the camera is still running, verify the framing, and, oh yeah, ask a series of questions both from my own notes and based on the anwsers the person was giving at the time. To be honest, I should be happy the whole production isn't shot upside and backwards in black and white.
At some point I will discuss my thoughts on the copying and peer-to-peering that will happen with this project, but not today. After my stuff is out, I'll likely write up a few thoughts about it.
The Eye of Doom is now viewing the bonus footage. I can't wait.
January 28, 2005
Another episode is basically finished, HPAC. It goes to the Eye of Doom on Monday, along with whatever else I finish. I haven't been working exclusively on HPAC this last 5 days; it just happens to be the final pieces clicked into shape with it first. I hope for a couple more to follow over the weekend so I can send the EoD three episodes again. Eye of Doom wants Eye Candy.
I'm very proud of HPAC. I knew I would have to have some amount of coverage of "Underground" BBSes and the whole hacker situation, but I didn't want it to be another rotten swipe at the subculture. We have quite enough of those as it is.
Basically, I try to cover Hacking, Phreaking, Anarchy and Cracking in a general manner as applies to bulletin boards. (There's also a big sequence about Handles). In terms of cracking, I mean removing software protection from computer games. It's quite a piece; very interesting to watch for me, even though I've been basically staring at it for months.
I have no idea what to make of it in a larger context of what people will think about it, so that will be interesting as well. Well, onward.
January 29, 2005
Another episode is finished! SYSOPS AND USERS, one of the more involved of the episodes and a personal favorite finally got the treatment of the sound and video mix that I was looking for, and will be leaving for the Eye of Doom along with HPAC. I wonder if I can do three in one weekend?
SYSOPS AND USERS covers what it says: the people who ran BBSes and the people who used them. It's basically a mass of anecdotes, memories and reactions of being on a BBS. It absolutely makes no attempt to explain what BBSes are and what their context is: that's what the other episodes are for, especially BAUD. And that's one of the reasons I wanted to make the documentary into a multi-episode collection: this documentary episode does not easily stand on its own if you have no idea what a BBS is, or have much knowledge about them beyond that. It's lifted up and supported by the episodes around it, and that's just the way I like it.
It's actually strange in several ways: it has no background music (too distracting), and no names under who's speaking. I made these two choices because the cutting is so intense and constant that you end up with this confusing blur of names that are NOT relevant to the episode. I will be using the unique technology of the DVD format to allow people who want to know who's who to see that; one of the subtitle tracks will be the name of whoever is speaking! So you just click over to that, and then you know. Everyone's credited at the end, of course.
If I'm called upon to describe it in a more artistic/filmmaker sense, the idea of the SYSOPS AND USERS episode is to have the same general feel of a BBS message base; people posting one after another, commenting, threads popping up, back and forth discussions and disagreements and the rest. On a more realistic sense, it's basically a lot of interviews cascading down with cool stories about the BBSes.
So that's two in one day! That's what I'm talking about.
As I progressed farther down the path of the BBS Documentary project, I started to consider where to take the textfiles.com project next. I'd been running it for nearly 8 years, of which 3-4 of them were involving the documentary itself. Obviously, the project should continue the process of providing history, historical context, and other related information about bulletin boards and subjects tangental to them, but what else? And how could I keep up my energy?
The original idea I had was to start something called "Textfiles University", which would provide a framework of self-teaching classes and lectures related to different parts of BBS culture. But there were two major problems with this. First of all, it would be nice to have it be TEXTFILES.EDU or otherwise some sort of accredited organization... a basically impossible goal. The other was that even though I would take a lot of personal humor and enjoyment out of the "university" role-playing and front that would be provided, I would eventually run into way too many roadblocks and issues. Most of these would be when more established organizations and individuals would wonder why I was putting on an "act" when I should just indicate what I really am: a person who does history. A historian.
Another major issue that was starting to creep on me was resources. I've run the TEXTFILES.COM out of pocket expenses since its inception, and the key to making that work was the fact that since it was, completely, textfiles, it wasn't a big hit for bandwidth. The files compressed well, the system flowed smoothly; it just all worked very well on that level. But then came audio.textfiles.com.
There had been other non-textfile sites on textfiles.com; I certainly wasn't going to play the game of "if it's not text, it's garbage", because the world wasn't 100 percent text at any point; there were always programs and attempts at images and everything else that didn't fit into ASCII. I started adding lots of subsites to textfiles.com to accomodate that, including sites for pdf files and old Apple II magazines and so on. It was always going to grow in that direction.
But audio.textfiles.com broke everything, because every mp3 or ogg file that was being created or which I pulled from the past was huge. Very huge. While most textfiles topped out at 100 or 200 kilobytes, even the most basic of the mp3 files were going into the megabytes, perhaps the tens of megabytes. Disk space has gotten spectacularly cheap, but connections have still been somewhat expensive once you start talking about a lot of large files being transferred constantly. Textfiles.com was now outgrowing its T-1. A T-1, I again point out, that has come directly out of my pocket.
There are solutions, of course; limit myself to just textfiles, keep the collection "private", look into better "hosting solutions", and all the rest of it. And yes, I could do that and it would probably solve the problem today. Or for a few months. But somewhere down the line, I will inevitably add another bit of history, some extra bits of information and collections, and something else will burst at the seams. Maybe it'll be buying another hard drive, or finding that I need a more powerful computer, or finding that the hosting solution I am with doesn't like the content of the site. Or anything else. Obviously, this whole thing needs to go to what the "X-treme" sports announcers prefer to call "The Next Level".
I loathe advertising. I do not like it in general, and when I see it used on websites, it just ruins me internally. I am glad there is software to deaden a lot of the experience for me via my browser, but I see a site get slowly taken over by advertising and promotions and all the rest of the foolishness associated with it, and you see the whole thing become a horrible mess. I truly hate it. I will not do it on any of my sites.
I receive fan mail. Not a whole lot, but enough that I think it's above the norm for a website. People tell me how much they enjoy going through textfiles.com unfettered by passwording systems, click-throughs, banners, interstitials, flash animations, and all the rest of the crap that has been strung over the online experience like marine netting. People come, they see what they want, they get it, they leave. Or they browse for hours, seeing what comes up, and then walk away with something they didn't expect to. And they come in large numbers: over 300,000 people visited textfiles.com in December of 2004. Three hundred thousand!
Obviously this site is wanted. Obviously, it is something worth doing. Obviously, my original plans to do this for a few nice years back in 1998 are out the window and I'm pretty much committed for a good time to come. So, how do I make sure it survives?
I have decided to create a non-profit organization called "The BBS History Foundation".
The goal of this organization will be to collect as much BBS history as possible, as well as tangental information related to subjects that drove BBSes. There are general computer history organizations and there are general online organizations, but I am aiming specifically at the BBS story, the BBS culture, and the related items from that time. By being non-profit, I can separate the work related to textfiles.com from the rest of my life, and I can bring in others to do work and help with projects without doing what I do now, which is basically everything. Most importantly, I can take donations; donations of bandwidth, of hardware, of equipment related to scanning and capturing information, and I can take monetary donations. It can all go towards running the site, not get stuck in some scary sort of half-Jason-half-textfiles.com bank account that I pull out of on a whim. Becoming a non-profit is not difficult, but becoming a tax-exempt one is, but I'm going to take a shot at it.
It means, ultimately, that I will lose some of the control of what I've been doing for so long. But on the other hand, it means that this whole project will partially take on a life of its own, will surprise me, and even more importantly, will grow beyond one guy and his basement. I think this needs to be done. I want to do it right. And I promise you, I intend to go through many phases of feeling about this, but none of them should be "regret".
The Eye of Doom has signed off on three of the episodes.
Who or what is the Eye of Doom? A friend of mine going a ways back who has an amazing talent for picking out all the slightest imperfections and oversights that crop up when you're working on a large-scale project like this documentary. Maybe a boom mike gets into a shot, or a filter is applied poorly, or the framing could be much improved. It takes an intense, detached, involved eye to do so, and he has it, so I call him the Eye of Doom. It's Doom because of course any major errors he finds means lots of work for me, but better now than after the DVDs are pressed and I can do nothing about them.
Every time the Eye asks me about a specific setup, like "Why is this shot like this?" or "Why do you cut from here to here?", I can often reply with a paragraph or two, specifically about this choice. This is a sign, perhaps, that I've spent a whole heck of a lot of time editing.
I kind of wish I could have done some little project before this one, some easy subject with a tiny but clear cast of characters that would have been a few months of work before this insane monolith of a project, but there you go. Every task in this project is really darned huge, and very time consuming. Just subtitles have been a big deal, not to mention the commentary tracks, or really, even the selection of music for the soundtracks. Big, I tell you.
Mostly, I am fixing up sound. Sound has turned out to be the big deal with this
project; with interviews conducted in hallways, computer rooms, conventions, outside, and even an in-use band rehearsal space, we have a lot of background noise on some of these tapes. I've been able to remove a lot of it, but it takes time. There's also some wide variances in how various people spoke, so I'm working with that as well. This is a case where I made a specific choice and trade-off a few years ago; I chose to do the travelling alone, instead of with a crew, because I knew that I would kill a crew or lose some good friends to take the trips I took. So with just me alone, there's only myself to look through the lens, consider the shot, set up the sound and take readings, and then conduct the interview. During the interview, I check the tape, check the lights, make sure the camera is still running, verify the framing, and, oh yeah, ask a series of questions both from my own notes and based on the anwsers the person was giving at the time. To be honest, I should be happy the whole production isn't shot upside and backwards in black and white.
At some point I will discuss my thoughts on the copying and peer-to-peering that will happen with this project, but not today. After my stuff is out, I'll likely write up a few thoughts about it.
The Eye of Doom is now viewing the bonus footage. I can't wait.
It's getting towards the end of the month, isn't it? Oh dear. Well, it's not for lack of my working on this project; I am currently clocking something in the range of 10-14 hours a day on it, while keeping a day job. You can imagine what I look like right now.
BAUD got my final once-over and is being rendered to go away for final inspection. Basically, I went through it twice, once for sound, and once for visuals. Nothing was allowed to pass for "next time" because there is no "next time". It is what it is, and I'm happy with it.
BAUD is the first in the series, so it was very important to me to have it cover its bases well. It goes into the forces that predate the BBS, the story of its beginning, and the story of the people who came after the pioneers and turned BBSes from a cool experiment into an event to be reckoned with. It is where Ward and Randy come forward and get the bow they richly deserve, and where a lot of people who were around for the two beginnings get to give a sense of what it was like.
I say "two beginnings" because the way I frame it, and the way that it felt in researching the story, is that there were two separate but very important sets of "first-timers": the people who were building kits and making modems work with these kits and who started the BBS, and the kids who begin their childhoods with the purchase of an 8-bit home computer and who turned the BBS into a place to be at, not just a thing to test. So they're both in this episode. Along the way, we cover a lot of ground, and stop pretty soon in "the big story". That's what the other episodes are for, after all.
Each of these episodes has to get this vicious once-over before I consider them done. Some are in much better shape than others. For the record, ARTSCENE has 12 tracks of audio and 8 tracks of video. You can imagine what editing something like that is when trying to tweak sound quality or get scrolling just right. My eyes cross just thinking about it. Most of the episodes are just a small round away from being done.
The subtitle people have been just absolutely astounding. Christian Wirth has jumped in and has transcribed over 4 hours of footage and counting. And the Spanish translation team is working at full bore; every day they send me a translated copy of one or more of the bonus footage clips, with the episodes coming shortly. It's great to get help there.
This is probably the last time I'll mention this for a while, but it really is going quickly and well.... just on a shift of a month and a half from where I was expecting it all to be. Otherwise, this is a smooth time. I've decided that everyone who pre-orders will get some sort of gift, some trinket to distinguish that they believed in me and bought ahead of time, before everyone knew what the project was. I haven't decided what that will be yet; I'll take suggestions.
Quite a ball has begun rolling in other quarters regarding Wikipedia and various criticisms from various quarters. As it is, I enjoy all the verbiage flying around, and reading the varying complaints and defenses that are going on. I have tried some social experiments with Wikipedia in the last month on and off (the Documentary takes precedence over everything of course), and intend to eventually report on those observations and findings in a future time, when I am not trying to get a major production out the door.
But I did want to step in and lance some of the growing redundant arguments going on in all the commenting I'm seeing posted, since I'm now seeing my essay used for these arguments pro and con and that's not entirely what I intended.
First of all, Wikipedia has, basically, forked. People have been running copies of Wikipedia that are following different rules for inclusion and exclusion. The problem is what a lot of projects encounter when they fork, which is brand recognition and brand dilution. Right now, Trillian has it set up that you can use Wikipedia to look up terms. That's a lot of weight placed into Wikipedia, to the detriment of other approaches that are not getting such attention. Until another project achieves critical mass or a big following that gets such high-profile regard, it's really a one-horse race, and so people are fighting over control of the horse. I think eventually we'll see a Coke-Pepsi situation, where the now-battling factions that are tearing apart Wikipedia on the inside will spit off into two major approaches with different goals, and that'll be that. It will be quite ugly, as, really, a lot of Wikipedia's internal political structures are.
People are defending Wikipedia by downplaying its importance. I am seeing an awful lot of arguments by people responding to criticism about Wikipedia by going "well, duh, it's just wikipedia", or "don't put so much weight on Wikipedia, it is what it is". I call this an "auto-straw man argument" or a "decoy self-ad hominem attack", although of course I'm sure the debate world has a more efficient term. Basically, you respond to criticism about your project by demeaning and debasing it yourself, before anyone else could get a chance to, and then stand there with your arms crossed at your swift hari-kiri in a swordfight. It's a pretty vacuous tactic; obviously Wikipedia is an important project, or so much effort would not be expended on it. And if it's important enough that people are literally pouring weeks and months of their lives into it, it's important enough to question basic tenets of how it is functioning. Look around for this argument by fervent Wikipedians. It's scarily everywhere and should be dismissed.
My primary disagreement with Wikipedia's approach is not about expertise, accuracy or quality; it is about procedure energy dispersal. The arguments about Wikipedia are being hijacked left and right by indicating this is a battle by the Old Guard against the New Better Way, or that this is a hue and cry by ivory-tower academia trying to prevent a pretender to the Throne of All That Is Good Intellectually. This makes someone who spends a lot of time on Wikipedia feel good about themselves (I'm fighting the Power) but my issues as stated in my previous essay were not about whether Wikipedia was in competition with other reference sources, but how minor procedural decisions have essentially doomed it on its own.
As an off-the-cuff example, Wikipedia has a login system, wherein for free and with no effort, you become a "Person", an entity with a name and a history and even your nice little page that you can use to build a fun little world of pictures and information about your work on Wikipedia. It is essentially effortless, and it is pretty easy to create a mass of user accounts and foment your opinions in votes and other situations. (This situation is called "sockpuppetry" by active Wikipedians and is a frequent call-out in votes; people who agree with someone they don't like can be accused of being a "sock puppet" and a meta-battle ensues). But this level of unaccountability isn't an adequate situation in the current Wikipedia political structure; instead, they allow totally anonymous full-content editing by random users. In other words, no accounting at all. People don't even have to submit to a rubber-stamp login process to begin screwing with entries that someone may have just spent hours getting just right. How could that second person possibly want to continue to be a part of the process?
This is what I am talking about; pure procedural defects that I think are fatal to Wikipedia's continued usefulness as a reference material, or even as a repository of information. It is why I will be making no further contributions to it beyond meta-interaction. (My social experimentation has had some neat responses and things I can report on, but I've not really been editing anything significant or worthwhile.) Indicating that my essay is about expertise or committes or the rest is kind of missing the point.
What's the point? A quick list:
There is no barrier to entry to cause wide-spread changes to Wikipedia and this is bad.
Wikipedia allows Votes for Deletion and will, upon one of these votes, erase information that may represent a lot of work and effort, based on shifting and arbitrary standards, and this is bad.
Wikipedia has a large contingency of users who play the Wikipedia Rules of Etiquette and Procedure like they were Role Playing Games and function within them causing havok and personal gratification at the expense of moving the project forward.
Academic review, experts vs. non-experts, use of Wikipedia as a replacement encyclopedia, and other such high-level concerns are way down the road and not my concern; my concern, and ultimately the reason why I have stopped contributing to the project (and why many others have, too) rests in aspects much closer to Wikipedia's core.
A pretty valid question, one especially important to the hundreds of folks who have been grand enough to pre-order the documentary, going back to October.
When I first opened the pre-ordering, I indicated that it would be going out for the end of December. In the beginning of December, I realized this was not the case, informed all the orders with valid e-mails, and then shifted to "sometime in January". As I'm working here, I want to increase the openness of the process so everyone understands what's going on.
Basically, the project is very nearly complete. The episodes are basically done. I've laid out all three DVDs. Even the boxes have come back. Here's one of the thousands of boxes printed:
There are several reasons why it's not out to the printers yet. The first is subtitles. I've intended to subtitle most of the project, so that people who cannot hear can read everything being said. There's also a lot of different voices from the hundreds of interviewees, and it makes it much easier to follow. But it also means that I and friends are transcribing episodes, pacing them in a subtitling program, and doing best to make sure they work. This is a ton of work.
Next is refining all the sound and video. There's quite a few hours of footage, with a lot of variance of sounds, since this was all one guy making the movie. I've got it pretty much under control, but there's still a few spots where people sound too loud or soft, and I have to go through all the footage to fix that.
There's some issues with having a lawyer looking at some of the footage, mostly based around trademarks appearing on shirts and in backgrounds; most of those are handled.
And finally, there's rendering in MPEG-2, which takes my machines quite some time (since there's so many hours).
I'm working very hard on all this, but it really isn't a marketing trick or other subterfuge; I really am just one guy, and each time I'm doing something, that's all I'm doing; there's no team following me up, just a few family members looking for any classic errors or omissions.
The printing company has told me it takes an average of 3 weeks for turnaround on a printing project like mine, which includes a "check disk" phase, where I look at my DVDs back from the shop and run them on a DVD player, to make sure they work well before we make thousands of them. So there's a big lump of time.
One of the reasons I've stopped trying to get this project talked about in other circles is because I want it out and available for people to get in the mail immediately after they order it, and until these things are in my house and ready to be shipped, that's simply not "ready". People are still pre-ordering and I appreciate that very, very much, but I am just not doing anything on the promotion side until it's ready to go.
I'm going as fast as I can, I really, really am. I will likely never do another project like this again, and I want this out the door as right and as good as I can make it. I'll keep everyone updated in coming days.