FTP’s Bright Sunset and Frozen Night —
FTP is kind of over.
Now, don’t get mad at me for telling you this. It’s not like I’m the one killing it, and I’m certainly one of its biggest fans. It’s a really mature technology that does exactly what it’s supposed to do. It is flexible, pumps through almost anything, and has features that do everything you probably want to do with file transfer.
And when I say over, I don’t mean obsolete. And I certainly don’t mean unused in the present day. Many things still use FTP as a method of transferring files, and providing access to all sorts of material.
But it’s quite obvious this isn’t the way of the future. Companies and individuals that utilize constantly changing data, or data that needs to be distributed, utilize a whole other variety of technologies. Many of these are web-based, while others are special protocols that blow out over the web into devices and phones. If it’s starting up, and it needs to get you some data, it is probably not using FTP. If it is using FTP, it’s probably not telling you it’s using FTP. And people who need to get things done don’t reach out for FTP.
But more markedly than FTP the protocol, it is FTP sites in particular that are really on the way out.
Coming as I did from the bulletin board system era, a well populated BBS might have a few minor text files, followed in the early 90s by a CDROM in a drive, and maybe topping out with a stack of CD-ROMs for a few hundred megabytes of accessible disk space. There were a notable handful of massive bulletin board systems that had much more data, but these were unbelievably rare and often cost a monthly fee.
Compare, then, the experience of an FTP site on the growing pre-web Internet, which would have thousands or tens of thousands of files that dwarfed anything you could get through a BBS. The names of these sites were extremely obscure, reflecting host names of systems within departments and shoved into some dark worthless corner of a science lab.
Even though the name didn’t tell you what the contents were, these sites became so populated and so important that their names became synonymous with what they held. ftp.luth.se. ccnga.uwaterloo.ca. ftp.funet.fi. ftp.uni-stuttgart.de. Trust me – you know what these things were for, and what they held. They were stunning in their power and they were the true online libraries of their time.
This summer time of the FTP site dominated the 90s and into the early 2000’s. Support files, drivers, game demos, hilarious films, browser executables, pictures, you name it. Many of them became fairly organized shambles of files, containing thousands of some obscure aspect of online life. There were even websites to help people navigate these FTP sites, trying to find what you needed. A number of FTP search engines existed, although they often required the filename more than what they contained or what they represented.
Many of these FTP sites did their best to join the World Wide Web and its unique needs, littering themselves with.files and HTML overlays. The Gopher format allowed superior methods and browsing the information. But as gopher thought of favor, and the major browsers stopped supporting it, it was another doomed way to navigate.
Finally, we had the experience of various FTP sites going down, and mirrors of those sites becoming subdirectories of the remaining ones. This Russian nesting doll situation has grown ludicrous to the point of some sites being in amalgamations of dozens of previous ones. Besides a few other navigational headaches, this also means that the loss of an FTP site in the modern era could actually be the deathknell for hundreds.
For the past couple of years, but really picking up intensively in the past few months, archive team has been aggressively downloading these FTP sites. We are not pursuing at risk FTP sites – we are in fact considering all FTP sites to be at risk at this point. If it’s on FTP, it’s probably doomed.
The archive.org FTP site collection is now in the hundreds of gigabytes and is growing constantly.
Naturally, when somebody puts up a block of data like this, it doesn’t take too long for the “you missed a spot” nerds to show up and start critiquing randomly.
The most valid and yet invalid argument is that currently these FTP sites exist on archive.org as massive tar archives or zips. “Give us our FTP sites back like they used to be,” cry the people who cry about such things. Well, sorry. That ship has sailed. I think that ship is on fire. Oh well, that ship is actually now burning other ships.
Instead, the should be considered what they are: cryogenic capsules of masses of data, waiting for the sort of duration, extensibility, and data mining efforts that so many of our computers are becoming so good at. They can be split apart, refactored into new ideas, or even pulled back into some mega FTP site of the future. By making the clearest, least fiddled-with archives of these FTP sites, we give the future infinite options. Anything else would be kind of silly.
We are ramping up faster and faster to do this. FTP sites die quiet deaths – a letter to faculty, a dropped connection. They go quite gently into the night.
Buried on these sites are proof of versions of software that others claim never existed, unique pieces of art that lived only on bulletin boards but were pulled up as refugees in the early 1990s, and even one-of-a-kind pieces of code that might otherwise have disappeared. This is to say nothing of the drivers, support documentation, configuration programs, and other parts related to hardware and software now not just obsolete but potentially forgotten. The value was obvious.
As I and others spend this new year gathering up all of this data, I look forward to projects coming along that utilize it or reference it. That’s kind of why it’s done. Sure, I myself would love the ultimate FTP site providing me every piece of the 1990s computer world for reference and utility. But you have to have the data, to make it pretty.
Combined with the efforts to grab every piece of CDROM plastic that has ever existed, I hope the plans are clear. The world is lost so much of what is come before. But it won’t this time.
Not on my watch.