ASCII by Jason Scott

Jason Scott's Weblog

Back That Thing Up —

img_3918

I’m going to mention two backup projects. Both have been under way for some time, but the world randomly decided the end of November 2016 was the big day, so here I am.

The first is that the Internet Archive is adding another complete mirror of the Wayback machine to one of our satellite offices in Canada. Due to the laws of Canada, to be able to do “stuff” in the country, you need to set up a separate company from your US concern. If you look up a lot of major chains and places, you’ll find they all have Canadian corporations. Well, so does the Internet Archive and that separate company is in the process of getting a full backup of the Wayback machine and other related data. It’s 15 petabytes of material, or more. It will cost millions of dollars to set up, and that money is already going out the door.

So, if you want, you can go to the donation page and throw some money in that direction and it will make the effort go better. That won’t take very long at all and you can feel perfectly good about yourself. You need read no further, unless you have an awful lot of disk space, at which point I suggest further reading.

8847193192_f56971c32d_b

Whenever anything comes up about the Internet Archive’s storage solutions, there’s usually a fluttery cloud of second-guessing and “big sky” suggestions about how everything is being done wrong and why not just engage a HBF0_X2000-PL and fark a whoziz and then it’d be solved. That’s very nice, but there’s about two dozen factors in running an Internet Archive that explain why RAID-1 and Petabyte Towers combined with self-hosting and non-cloud storage has worked for the organization. There are definitely pros and cons to the whole thing, but the uptime has been very good for the costs, and the no-ads-no-subscription-no-login model has been working very well for years. I get it – you want to help. You want to drop the scales from our eyes and you want to let us know about the One Simple Trick that will save us all.

That said, when this sort of insight comes out, it’s usually back-of-napkin and done by someone who will be volunteering several dozen solutions online that day, and that’s a lot different than coming in for a long chat to discuss all the needs. I think someone volunteering a full coherent consult on solutions would be nice, but right now things are working pretty well.

There are backups of the Internet Archive in other countries already; we’re not that bone stupid. But this would be a full, consistently, constantly maintained full backup in Canada, and one that would be interfaced with other worldwide stores. It’s a preparation for an eventuality that hopefully won’t come to pass.

There’s a climate of concern and fear that is pervading the landscape this year, and the evolved rat-creatures that read these words in a thousand years will be able to piece together what that was. But regardless of your take on the level of concern, I hope everyone agrees that preparation for all eventualities is a smart strategy as long as it doesn’t dilute your primary functions. Donations and contributions of a monetary sort will make sure there’s no dilution.

So there’s that.

Now let’s talk about the backup of this backup a great set of people have been working on.

vj243rffv_ilmy-qedvtkw_r

About a year ago, I helped launch INTERNETARCHIVE.BAK. The goal was to create a fully independent distributed copy of the Internet Archive that was not reliant on a single piece of Internet Archive hardware and which would be stored on the drives of volunteers, with 3 geographically distributed copies of the data worldwide.

Here’s the current status page of the project. We’re backing up 82 terabytes of information as of this writing. It was 50 terabytes last week. My hope is that it will be 1,000 terabytes sooner rather than later. Remember, this is 3 copies, so to do each terabyte needs three terabytes.

For some people, a terabyte is this gigantically untenable number and certainly not an amount of disk space they just have lying around. Other folks have, at their disposal, dozens of terabytes. So there’s lots of hard drive space out there, just not evenly distributed.

The IA.BAK project is a complicated one, but the general situation is that it uses the program git-annex to maintain widely-ranged backups from volunteers, with “check-in” of data integrity on a monthly basis. It has a lot of technical meat to mess around with, and we’ve had some absolutely stunning work done by a team of volunteering developers and maintainers (and volunteers) as we make this plan work on the ground.

And now, some thoughts on the Darkest Timeline.

whyyyyyy_jscott

I’m both an incredibly pessimistic and optimistic person. Some people might use the term “pragmatic” or something less charitable.

Regardless, I long ago gave up assumptions that everything was going to work out OK. It has not worked out OK in a lot of things, and there’s a lot of broken and lost things in the world. There’s the pessimism. The optimism is that I’ve not quite given up hope that something can’t be done about it.

I’ve now dedicated 10% of my life to the Internet Archive, and I’ve dedicated pretty much all of my life to the sorts of ideals that would make me work for the Archive. Among those ideals are free expression, gathering of history, saving of the past, and making it all available to as wide an audience, without limit, as possible. These aren’t just words to me.

Regardless of if one perceives the coming future as one rife with specific threats, I’ve discovered that life is consistently filled with threats, and only vigilance and dedication can break past the fog of possibilities. To that end, the Canadian Backup of the Internet Archive and the IA.BAK projects are clear bright lines of effort to protect against all futures dark and bright. The heritage, information and knowledge within the Internet Archive’s walls are worth protecting at all cost. That’s what drives me and why these two efforts are more than just experiments or configurations of hardware and location.

So, hard drives or cash, your choice. Or both!


In Which I Tell You It’s A Good Idea To Support a Magazine-Scanning Patreon —

So, Mark Trade and I have never talked, once.

All I know about Mark is that due to his efforts, over 200 scans of magazines are up on the Archive.

headlock

These are very good scans, too. The kind of scans that a person looking to find a long-lost article, verify a hard-to-grab fact, or needs to pass along to others a great image would kill to have. 600 dots per inch, excellent contrast, clarity, and the margins cut just right.

cd-rom_today_05_aprmay_1994_0036

So, I could fill this entry with all the nice covers, but covers are kind of easy, to be frank. You put them face down on the scanner, you do a nice big image, and then touch it up a tad. The cover paper and the printing is always super-quality compared to the rest, so it’ll look good:

cd-rom_today_05_aprmay_1994_0000

But the INSIDE stuff… that’s so much harder. Magazines were often bound in a way that put the images RIGHT against the binding and not every magazine did the proper spacing and all of it is very hard to shove into a scanner and not lose some information. I have a lot of well-meaning scans in my life with a lot of information missing.

But these…. these are primo.

pcgames_01_fall_1988_0011

pcgames_01_fall_1988_0012

pcgames_01_fall_1988_0073

When I stumbled on the Patreon, he had three patrons giving him $10 a month. I’d like it to be $500, or $1000. I want this to be his full-time job.

Reading the patreon page’s description of his process shows he’s taking it quite seriously. Steaming glue, removing staples. I’ve gone on record about the pros and cons of destructive scanning, but game magazines are not rare, just entirely unrepresented in scanned items compared to how many people have these things in their past.

I read something like this:

It is extremely unlikely that I will profit from your pledge any time soon. My scanner alone was over $4,000 and the scanning software was $600. Because I’m working with a high volume of high resolution 600 DPI images I purchased several hard drives including a CalDigit T4 20TB RAID array for $2,000. I have also spent several thousand dollars on the magazines themselves, which become more expensive as they become rarer. This is in addition to the cost of my computer, monitor, and other things which go into the creation of these scans. It may sound like I’m rich but really I’m just motivated, working two jobs and pursuing large projects.

…and all I think about is, this guy is doing so much amazing work that so many thousands could be benefiting from, and they should throw a few bucks at him for his time.

My work consists of carefully removing individual pages from magazines with a heat gun or staple-remover so that the entire page may be scanned. Occasionally I will use a stack paper cutter where appropriate and will not involve loss of page content. I will then scan the pages in my large format ADF scanner into 600 DPI uncompressed TIFFs. From there I either upload 300 DPI JPEGs for others to edit and release on various sites or I will edit them myself and store the 600 DPI versions in backup hard disks. I also take photos of magazines still factory-sealed to document their newsstand appearance. I also rip full ISOs of magazine coverdiscs and make scans of coverdisc sleeves on a color-corrected flatbed scanner and upload those to archive.org as well.

This is the sort of thing I can really get behind.

The Internet Archive is scanning stuff, to be sure, but the focus is on books. Magazines are much, much harder to scan – the book scanners in use are just not as easy to use with something bound like magazines are. The work that Mark is doing is stuff that very few others are doing, and to have canonical scans of the advertisements, writing and materials from magazines that used to populate the shelves is vital.

Some time ago, I’ve given all my collection of donated Game-related magazines to the Museum of Art and Digital Entertainment, because I recognized I couldn’t be scanning them anytime soon, and how difficult it was going to be to scan it. It would take some real major labor I couldn’t personally give.

Well, here it is. He’s been at it for a year. I’d like to see that monthly number jump to $100/month, $500/month, or more. People dropping $5/month towards this Patreon would be doing a lot for this particular body of knowledge.

Please consider doing it.

Thanks.


A Simple Explanation: VLC.js —

The previous entry got the attention it needed, and the maintainers of the VLC project connected with both Emularity developers and Emscripten developers and the process has begun.

The best example of where we are is this screenshot:

vlcjs

The upshot of this is that a javascript compiled version of the VLC player now runs, spits out a bunch of status and command line information, and then gets cranky it has no video/audio device to use.

With the Emularity project, this was something like 2-3 months into the project. In this case, it happened in 3 days.

The reasons it took such a short time were multi-fold. First, the VLC maintainers jumped right into it at full-bore. They’ve had to architect VLC for a variety of wide-ranging platforms including OSX, Windows, Android, and even weirdos like OS/2; to have something aimed at “web” is just another place to go. (They’d also made a few web plugins in the past.) Second, the developers of Emularity and Emscripten were right there to answer the tough questions, the weird little bumps and switchbacks.

Finally, everybody has been super-energetic about it – diving into the idea, without getting hung up on factors or features or what may emerge; the same flexibility that coding gives the world means that the final item will be something that can be refined and improved.

So that’s great news. But after the initial request went into a lot of screens, a wave of demands and questions came along, and I thought I’d answer some of them to the best of my abilities, and also make some observations as well.

lunettes

When you suggest something somewhat crazy, especially in the programming or development world, there’s a variant amount of response. And if you end up on Hackernews, Reddit, or a number of other high-traffic locations, those reactions fall into some very predictable areas:

  • This would be great if it happens
  • This is fundamentally terrible, let me talk about why for 4 paragraphs
  • You are talking about making a sword. I am a swordmaker. I have many opinions.
  • My sister was attacked by a C library and I’m going to tell a very long story
  • Oh man, Jason Scott, this guy

So, quickly on some of these:

  • It’s understandable some people will want to throw the whole idea under the bus because the idea of the Web Browser playing a major part in transactions is a theoretical hellscape compared to an ideal infrastructure, but that’s what we have and here we go.
  • I know that it sounds like porting things to Javascript is crazy. I find that people think we’re rewriting things from scratch, instead of using Emscripten, which compiles out to Javascript as a target (and later WebAssembly). We do not write from scratch.
  • Browsers do some of this heavy lifting. It depends on the browser on the platform on the day and they do not talk. If there was a way to include a framework to tell a browser what to do with ‘stuff’ and then it brought both the stuff and the instructions in and did the work, great. Yes, there’s plenty of cases of stuff/instructions (Webpage/HTML, Audio/MP3) that browsers take in, but it’s different everywhere.

But let’s shift over to why I think this is important, and why I chose VLC to interact with.

First, VLC is one of those things that people love, or people wish there was something better than, but VLC is what we have. It’s flexible, it’s been well-maintained, and it has been singularly focused. For a very long time, the goal of the project has been aimed at turning both static files AND streams into something you can see on your machine. And the machine you can see it on is pretty much every machine capable of making audio and video work.

Fundamentally, VLC is a bucket that, when dropped into with a very large variance of sound-oriented or visual-oriented files and containers, will do something with them. DVD ISO files become playable DVDs, including all the features of said DVDs. VCDs become craptastic but playable DVDs. MP3, FLAC, MIDI, all of them fall into VLC and start becoming scrubbing-ready sound experiences. There are quibbles here and there about accuracy of reproduction (especially with older MOD-like formats like S3M or .XM) but these are code, and fixable in code. That VLC doesn’t immediately barf on the rug with the amount of crapola that can be thrown at it is enormous.

And completing this thought, by choosing something like VLC, with its top-down open source condition and universal approach, the “closing of the loop” from VLC being available in all browsers instantly will ideally cause people to find the time to improve and add formats that otherwise wouldn’t experience such advocacy. Images into Apple II floppy disk image? Oscilloscope captures? Morse code evaluation? Slow Scan Television? If those items have a future, it’s probably in VLC and it’s much more likely if the web uses a VLC that just appears in the browser, no fuss or muss.

vlc-media-player-dowload-for-windows

Fundamentally, I think my personal motivations are pretty transparent and clear. I help oversee a petabytes-big pile of data at the Internet Archive. A lot of it is very accessible; even more of it is not, or has to have clever “derivations” pulled out of it for access. You can listen to .FLACs that have been uploaded, for example, because we derive (noted) mp3 versions that go through the web easier. Same for the MPG files that become .mp4s and so on, and so on. A VLC that (optionally) can play off the originals, or which can access formats that currently sit as huge lumps in our archives, will be a fundamental world changer.

Imagine playing DVDs right there, in the browser. Or really old computer formats. Or doing a bunch of simple operations to incoming video and audio to improve it without having to make a pile of slight variations of the originals to stream. VLC.js will do this and do it very well. The millions of files that are currently without any status in the archive will join the millions that do have easy playability. Old or obscure ideas will rejoin the conversation. Forgotten aspects will return. And VLC itself, faced with such a large test sample, will get better at replaying these items in the process.

This is why this is being done. This is why I believe in it so strongly.

besthook

I don’t know what roadblocks or technical decisions the team has ahead of it, but they’re working very hard at it, and some sort of prototype seems imminent. The world with this happening will change slightly when it starts working. But as it refines, and as these secondary aspects begin, it will change even more. VLC will change. Maybe even browsers will change.

Access drives preservation. And that’s what’s driving this.

See you on the noisy and image-filled other side.


A Simple Request: VLC.js —

Almost five years ago to today, I made a simple proposal to the world: Port MAME/MESS to Javascript.

That happened.

I mean, it cost a dozen people hundreds of hours of their lives…. and there were tears, rage, crisis, drama, and broken hearts and feelings… but it did happen, and the elation and the world we live in now is quite amazing, with instantaneous emulated programs in the browser. And it’s gotten boring for people who know about it, except when they haven’t heard about it until now.

By the way: work continues earnestly on what was called JSMESS and is now called The Emularity. We’re doing experiments with putting it in WebAssembly and refining a bunch of UI concerns and generally making it better, faster, cooler with each iteration. Get involved – come to #jsmess on EFNet or contact me with questions.

In celebration of the five years, I’d like to suggest a new project, one of several candidates I’ve weighed but which I think has the best combination of effort to absolute game-changer in the world.

vlc-media-player-dowload-for-windows

Hey, come back!

It is my belief that a Javascript (later WebAssembly) port of VLC, the VideoLan Player, will fundamentally change our relationship to a mass of materials and files out there, ones which are played, viewed, or accessed. Just like we had a lot of software locked away in static formats that required extensive steps to even view or understand, so too do we have formats beyond the “usual” that are also frozen into a multi-step process. Making these instantaneously function in the browser, all browsers, would be a revolution.

A quick glance at the features list of VLC shows how many variant formats it handles, from audio and sound files through to encapsulations like DVD and VCDs. Files that now rest as hunks of ISOs and .ZIP files that could be turned into living, participatory parts of the online conversation. Also, formats like .MOD and .XM (trust me) would live again effectively.

Also, VLC has weathered years and years of existence, and the additional use case for it would help people contribute to it, much like there’s been some improvements in MAME/MESS over time as folks who normally didn’t dip in there added suggestions or feedback to make the project better in pretty obscure realms.

I firmly believe that this project, fundamentally, would change the relationship of audio/video to the web. 

I’ll write more about this in coming months, I’m sure, but if you’re interested, stop by #vlcjs on EFnet, or ping me on twitter at @textfiles, or write to me at vlcjs@textfiles.com with your thoughts and feedback.

See you.

 


The Festival Floppies —

In 2009, Josh Miller was walking through the Timonium Hamboree and Computer Festival in Baltimore, Maryland. Among the booths of equipment, sales, and demonstrations, he found a vendor was selling an old collection of 3.5″ floppy disks for DOS and Windows. He bought it, and kept it.

A few years later, he asked me if I wanted them, and I said sure, and he mailed them to me. They fell into the everlasting Project Pile, and waited for my focus and attention.

They looked like this:

cs6rforueaagbcg

I was particularly interested in the floppies that appeared to be someone’s compilation of DOS and Windows programs in the most straightforward form possible – custom laser-printed directories on the labels, and no obvious theme as to why this shareware existed on them. They looked like this, separated out:

cs6reiouaaaamnd

There were other floppies in the collection, as well:

cswfhv2xeaa4hrl

They’d sat around for a few years while I worked on other things, but the time finally came this week to spend some effort to extract data.

There’s debates on how to do this that are both boring and infuriating, and I’ve ended friendships over them, so let me just say that I used a USB 3.5″ floppy drive (still available for cheap on Amazon; please take advantage of that) and a program called WinImage that will pull out a disk image in the form of a .ima file from the floppy drive. Yes, I could do a flux imaging of these disks, but sorry, that’s incredibly insane overkill. These disks contain files put on there by a person and we want those files, along with the accurate creation dates and the filenames and contents. WinImage does it.

Sometimes, the floppies have some errors and require trying over to get the data off them. Sometimes it takes a LOT of tries. If after a mass of tries I am unable to do a full disk scan into a disk image, I try just mounting it as A: in Windows and pulling the files off – they sometimes are just fine but other parts of the disk are dead. I make this a .ZIP file instead of a .IMA file. This is not preferred, but the data gets off in some form.

Some of them (just a handful) were not even up for this – they’re sitting in a small plastic bin and I’ll try some other methods in the future. The ratio of Imaged-ZIPed-Dead were very good, like 40-3-3.

I dumped most of the imaged files (along with the ZIPs) into this item.

This is a useful item if you, yourself, want to download about 100 disk image files and “do stuff” with them. My estimation is that all of you can be transported from the first floor to the penthouse of a skyscraper with 4 elevator trips. Maybe 3. But there you go, folks. They’re dropped there and waiting for you. Internet Archive even has a link that means “give me everything at once“. It’s actually not that big at all, of course – about 260 megabytes, less than half of a standard CD-ROM.

I could do this all day. It’s really easy. It’s also something most people could do, and I would hope that people sitting on top of 3.5” floppies from DOS or Windows machines would be up for paying the money for that cheap USB drive and something like WinImage and keep making disk images of these, labeling them as best they can.

I think we can do better, though.

The Archive is running the Emularity, which includes a way to run EM-DOSBOX, which can not only play DOS programs but even play Windows 3.11 programs as well.

Therefore, it’s potentially possible for many of these programs, especially ones particularly suited as stand-alone “applications”, to be turned into in-your-browser experiences to try them out. As long as you’re willing to go through them and get them prepped for emulation.

Which I did.

floppo

The Festival Floppies collection is over 500 programs pulled from these floppies that were imaged earlier this week. The only thing they have in common was that they were sitting in a box on a vendor table in Baltimore in 2009, and I thought in a glance they might run and possibly be worth trying out. After I thought this (using a script to present them for consideration), the script did all the work of extracting the files off the original floppy images, putting the programs into an Internet Archive item, and then running a “screen shotgun” I devised with a lot of help a few years back that plays the emulations, takes the “good shots” and makes them part of a slideshow so you can get a rough idea of what you’re looking at.

00_coverscreenshot

You either like the DOS/Windows aesthetic, or you do not. I can’t really argue with you over whatever direction you go – it’s both ugly and brilliant, simple and complex, dated and futuristic. A lot of it depended on the authors and where their sensibilities lay. I will say that once things started moving to Windows, a bunch of things took on a somewhat bland sameness due to the “system calls” for setting up a window, making it clickable, and so on. Sometimes a brave and hearty soul would jazz things up, but they got rarer indeed. On the other hand, we didn’t have 1,000 hobbyist and professional programs re-inventing the wheel, spokes, horse, buggy, stick shift and gumball machine each time, either.

screenshot_05

Just browsing over the images, you probably can see cases where someone put real work into the whole endeavor – if they seem to be nicely arranged words, or have a particular flair with the graphics, you might be able to figure which ones have the actual programming flow and be useful as well. Maybe not a direct indicator, but certainly a flag. It depends on how much you want to crate-dig through these things.

Let’s keep going.

Using a “word cloud” script that showed up as part of an open source package, I rewrote it into something I call a “DOS Cloud”. It goes through these archives of shareware, finds all the textfiles in the .ZIP that came along for the ride (think README.TXT, READ.ME, FILEID.DIZ and so on) and then runs to see what the most frequent one and two word phrases are. This ends up being super informative, or not informative at all, but it’s automatic, and I like automatic. Some examples:

Mad Painterpaint, mad, painter, truck, joystick, drive, collision, press, cloud, recieve, mad painter, dos prompt

Screamer screamer, code, key, screen, program, press, command, memory, installed,activate, code key, memory resident, correct code, key combination, desired code

D4W20timberline, version, game, sinking, destroyer, gaming, smarter, software,popularity, timberline software, windows version, smarter computer, online help, high score

Certainly in the last case, those words are much more informative than the name D4W20 (which actually stands for “Destroyer for Windows Version 2.0”), and so the machine won the day. I’ve called this “bored intern” level before and I’d say it’s still true – the intern may be bored, but they never stop doing the process, either. I’m sure there’s some nascent class discussion here, but I’ll say that I don’t entirely think this is work for human beings anymore. It’s just more and more algorithms at this point. Reviews and contextual summaries not discernible from analysis of graphics and text are human work.

For now.

screenshot_00

These programs! There are a lot of them, and a good percentage solve problems we don’t have anymore or use entire other methods to deal with the information. Single-use programs to deal with Y2K issues, view process tasks better, configure your modem, add a DOS interface, or track a pregnancy. Utilities to put the stardate in the task bar, applications around coloring letters, and so it goes. I think the screenshots help make decisions, if you’re one of the people idly browsing these sets and have no personal connection to DOS or Windows 3.1 as a lived experience.

I and others will no doubt write more and more complicated methods for extracting or providing metadata for these items, and work I’m doing in other realms goes along with this nicely. At some point, the entries for each program will have a complication and depth that rivals most anything written about the subjects at the time, when they were the state of the art in computing experience. I know that time is coming, and it will be near-automatic (or heavily machine-assisted) and it will allow these legions of nearly-lost programs to live again as easily as a few mouse clicks.

But then what?

screenshot_03

But Then What is rapidly becoming the greatest percentage of my consideration and thought, far beyond the relatively tiny hurdles we now face in terms of emulation and presentation. It’s just math now with a lot of what’s left (making things look/work better on phones, speeding up the browser interactions, adding support for disk swapping or printer output or other aspects of what made a computer experience lasting to its original users). Math, while difficult, has a way of outing its problems over time. Energy yields results. Processing yields processing.

No, I want to know what’s going to happen beyond this situation, when the phones and browsers can play old everything pretty accurately, enough that you’d “get it” to any reasonable degree playing around with it.

Where do we go from there? What’s going to happen now? This is where I’m kind of floating these days, and there are ridiculously scant answers. It becomes very “journey of the mind” as you shake the trees and only nuts come out.

To be sure, there’s a sliver of interest in what could be called “old games” or “retrogaming” or “remixes/reissues” and so on. It’s pretty much only games, it’s pretty much roughly 100 titles, and it’s stuff that has seeped enough into pop culture or whose parent companies still make enough bank that a profit motive serves to ensure the “IP” will continue to thrive, in some way.

The Gold Fucking Standard is Nintendo, who have successfully moved into such a radical space of “protecting their IP” that they’ve really successfully started moving into wrecking some of the past – people who make “fan remixes” might be up for debate as to whether they should do something with old Nintendo stuff, but laying out threats for people recording how they experienced the games, and for any recording of the games for any purpose… and sending legal threats at anyone and everyone even slightly referencing their old stuff, as a core function.. well, I’m just saying perhaps ol’ Nintendo isn’t doing itself any favors but on the other hand they can apparently be the most history-distorting dicks in this space quadrant and the new games still have people buy them in boatloads. So let’s just set aside the Gold Fucking Standard for a bit when discussing this situation. Nobody even comes close.

There’s other companies sort of taking this hard-line approach: “Atari”, Sega, Capcom, Blizzard… but again, these are game companies markedly defending specific games that in many cases they end up making money on. In some situations, it’s only one or two games they care about and I’m not entirely convinced they even remember they made some of the others. They certainly don’t issue gameplay video takedowns and on the whole, historic overview of the companies thrives in the world.

But what a small keyhole of software history these games are! There’s entire other experiences related to software that are both available, and perhaps even of interest to someone who never saw this stuff the first time around. But that’s kind of an educated guess on my part. I could be entirely wrong on this. I’d like to find out!

Pursuing this line of thought has sent me hurtling into What are even musuems and what are even public spaces and all sorts of more general questions that I have extracted various answers for and which it turns out are kind of turmoil-y. It also has informed me that nobody kind of completely knows but holy shit do people without managerial authority have ideas about it. Reeling it over to the online experience of this offline debated environment just solves some problems (10,000 people look at something with the same closeness and all the time in the world to regard it) and adds others (roving packs of shitty consultant companies doing rough searches on a pocket list of “protected materials” and then sending out form letters towards anything that even roughly matches it, and calling it a ($800) day).

Luckily, I happen to work for an institution that is big on experiments and giving me a laughably long leash, and so the experiment of instant online emulated computer experience lives in a real way and can allow millions of people (it’s been millions, believe it or not) to instantly experience those digital historical items every second of every day.

So even though I don’t have the answers, at all, I am happy that the unanswered portions of the Big Questions haven’t stopped people from deriving a little joy, a little wonder, a little connection to this realm of human creation.

That’s not bad.

screenshot_00-1


Why the Apple II ProDOS 2.4 Release is the OS News of the Year —

prodos-2-4-splash

In September of 2016, a talented programmer released his own cooked update to a major company’s legacy operating system, purely because it needed to be done. A raft of new features, wrap-in programs, and bugfixes were included in this release, which I stress was done as a hobby project.

The project is understatement itself, simply called Prodos 2.4. It updates ProDOS, the last version of which, 2.0.3, was released in 1993.

You can download it, or boot it in an emulator on the webpage, here.

As an update unto itself, this item is a wonder – compatibility has been repaired for the entire Apple II line, from the first Apple II through to the Apple IIgs, as well as cases of various versions of 6502 CPUs (like the 65C02) or cases where newer cards have been installed in the Apple IIs for USB-connected/emulated drives. Important utilities related to disk transfer, disk inspection, and program selection have joined the image. The footprint is smaller, and it runs faster than its predecessor (a wonder in any case of OS upgrades).

The entire list of improvements, additions and fixes is on the Internet Archive page I put up.

prodos-2-4-bitsy-boot

The reason I call this the most important operating system update of the year is multi-fold.

First, the pure unique experience of a 23-year-gap between upgrades means that you can see a rare example of what happens when a computer environment just sits tight for decades, with many eyes on it and many notes about how the experience can be improved, followed by someone driven enough to go through methodically and implement all those requests. The inclusion of the utilities on the disk means we also have the benefit of all the after-market improvements in functionality that the continuing users of the environment needed, all time-tested, and all wrapped in without disturbing the size of the operating system programs itself. It’s like a gold-star hall of fame of Apple II utilities packed into the OS they were inspired by.

This choreographed waltz of new and old is unique in itself.

Next is that this is an operating system upgrade free of commercial and marketing constraints and drives. Compared with, say, an iOS upgrade that trumpets the addition of a search function or blares out a proud announcement that they broke maps because Google kissed another boy at recess. Or Windows 10, the 1968 Democratic Convention Riot of Operating Systems, which was designed from the ground up to be compatible with a variety of mobile/tablet products that are on the way out, and which were shoved down the throats of current users with a cajoling, insulting methodology with misleading opt-out routes and freakier and freakier fake-countdowns.

The current mainstream OS environment is, frankly, horrifying, and to see a pure note, a trumpet of clear-minded attention to efficiency, functionality and improvement, stands in testament to the fact that it is still possible to achieve this, albeit a smaller, slower-moving target. Either way, it’s an inspiration.

prodos-2-4-bitsy-bye

Last of all, this upgrade is a valentine not just to the community who makes use of this platform, but to the ideas of hacker improvement calling back decades before 1993. The amount of people this upgrade benefits is relatively small in the world – the number of folks still using Apple IIs is tiny enough that nearly everybody doing so either knows each other, or knows someone who knows everyone else. It is not a route to fame, or a resume point to get snapped up by a start-up, or a game of one-upsmanship shoddily slapped together to prove a point or drop a “beta” onto the end as a fig leaf against what could best be called a lab experiment gone off in the fridge. It is done for the sake of what it is – a tool that has been polished and made anew, so the near-extinct audience for it works to the best of their ability with a machine that, itself, is thought of as the last mass-marketed computer designed by a single individual.

That’s a very special day indeed, and I doubt the remainder of 2016 will top it, any more than I think the first 9 months have.

Thanks to John Brooks for the inspiration this release provides. 


Who’s Going to be the Hip Hop Hero —

People often ask me if there’s a way they can help. I think I have something.

So, the Internet Archive has had a wild hit on its hand with the Hip Hop Mixtapes collection, which I’ve been culling from multiple sources and then shoving into the Archive’s drives through a series of increasingly complicated scripts. When I run my set of scripts, they do a good job of yanking the latest and greatest from a selection of sources, doing all the cleanup work, verifying the new mixtapes aren’t already in the collection, and then uploading them. From there, the Archive’s processes do the work, and then we have ourselves the latest tapes available to the world.

Since I see some of these tapes get thousands of listens within hours of being added, I know this is something people want. So, it’s a success all around.

mixtape

With success, of course, comes the two flipside factors: My own interest in seeing the collection improved and expanded, and the complaints from people who know about this subject finding shortcomings in every little thing.

There is a grand complaint that this collection currently focuses on mixtapes from 2000 onwards (and really, 2005 onwards). Guilty. That’s what’s easiest to find. Let’s set that one aside for a moment, as I’ve got several endeavors to improve that.

What I need help with is that there are a mass of mixtapes that quickly fell off the radar in terms of being easily downloadable and I need someone to spend time grabbing them for the collection.

While impressive, the 8,000 tapes up on the archive are actually the ones that were able to be grabbed by scripts, without any hangups, like the tapes falling out of favor or the sites they were offering going down. If you use the global list I have, the total amount of tapes could be as high as 20,000.

Again, it’s a shame that a lot of pre-2000 mixtapes haven’t yet fallen into my lap, but it’s really a shame that mixtapes that existed, were uploaded to the internet, and were readily available just a couple years ago, have faded down into obscurity. I’d like someone (or a coordinated group of someones) help grab those disparate and at-risk mixtapes to get into the collection.

I have information on all these missing tapes – the song titles, the artist information, and even information on mp3 size and what was in the original package. I’ve gone out there and tried to do this work, and I can do it, but it’s not a good use of my time – I have a lot of things I have to do and dedicating my efforts in this particular direction means a lot of other items will suffer.

So I’m reaching out to you. Hit me up at mixtapes@textfiles.com and help me build a set of people who are grabbing this body of work before it falls into darkness.

Thanks.


Karateka: The Alpha and the Beta —

As I enter into a new phase of doing things and how I do things, let’s start with something pleasant.

karateka-beta

As part of the work with pulling Prince of Persia source code from a collection of disks a number of years back (the lion’s share of the work done by Tony Diaz), Jordan Mechner handed me an additional pile of floppies.

Many of these floppies have been imaged and preserved, but a set of them had not, mostly due to coming up with the time and “doing it right” and all the other accomplishment-blocking attributes of fractal self-analysis. That issue is now being fixed, and you are encouraged to enjoy the immediate result.

As Karateka (1985) became a huge title for Brøderbund Software, they wanted the program to run on as many platforms as possible. However, the code was not written to be portable; Brøderbund instead contracted with a number of teams to make Karateka versions on hardware other than the Apple II. The work by these teams, Jordan Mechner told me, often suffered from being ground-up rewrites of the original game idea – they would simply make it look like the game, without really spending time duplicating the internal timing or logic that Jordan had put into the original. Some came out fine on the other end; others did not.

Jordan’s opinion on the IBM port of Karateka was not positive. From his Making-of-Karateka journal (thanks to  for finding this entry):

CrN5pu9XYAIo4es

You can now see how it looked and played when he made these comments. I just pulled out multiple copies of Karateka from a variety of internally distributed floppies Jordan had in the set he gave me. I chose two representative versions and now you can play them both on the Internet Archive.

screenshot_02The first version is what would now be called the “Alpha”, but which in this collection is just called “Version 1986-01-30”, and was duplicated on February 4, 1986. It is a version which was obviously done as some sort of milestone – debugging information is everywhere, and it starts with a prompt of which levels to try, before starting the game.

Without going too much into the specific technical limitations of PC Compatibles of the time, I’ll instead just offer the following screenshot, which will connect you to an instantly playable-in-browser version of the Karateka Alpha. This has never been released before.

alpha-karateka

You can see all sorts of weird artifacts and performance issues with the Alpha – glitches in graphics and performance, and of course the ever-present debugging messages and system information. The contractors doing the work, the Connelly Group, have no presence on the internet in any obvious web searches – they may have just been internal employees, or a name given to some folks just to keep distance between “games” work and “real” work; maybe that information will come out.

The floppy this came on, as shown above, had all sorts of markings for Brøderbund to indicate what the build number was, who had the floppies (inventory control), and that the disk had no protection routines on it, which makes my life in the present day notably easier. Besides the playable version of the information in a ZIP file, there is an IMG file of the entire 360k floppy layout, usable by a number of emulators or viewers.

The next floppy in terms of time stamp is literally called BETA, from March 3, 1986. With over a month of effort into the project, a bunch of bugs have been fixed, screens added, and naturally all the debugging information has been stripped away. I’m assuming this was for playtesters to check out, or to be used by marketing/sales to begin the process of selling it in the PC world. Here is a link to an instantly playable-in-browser version of the Karateka Beta. This has also never been released before.

beta

For the less button-mashy of us, here are the keys and a “handing it over to you at the user group meeting” version of how Karateka works.

You’re a dude going from the left to the right. If you go too far left, you will fall off the cliff and die. To the right are a bunch of enemies. You can either move or fight. If you are not in a fighting stance, you will die instantly, but in a fighting stance, you will move really slowly.

You use the arrow keys (left and right) to move. Press SPACE to flip between “moving” and “fighting” modes. The Q, A, and Z keys are high, middle and low punches. The W, S and X keys are high middle and low kicks. The triangles on the bottom are life meters. Whoever runs out of triangles first in a fight will die.

It’s worthwhile to note that the games, being an Alpha and a Beta, are extremely rough. I wouldn’t suggest making them your first game of Karateka ever – that’s where you should play the original Apple II version

Karateka is a wealth of beginnings for understanding entertainment software – besides being a huge hit for Brøderbund, it’s an aesthetic masterwork, containing cinematic cutscenes and a clever pulling of cultural sources to combine into a multi-layered experience on a rather simple platform. From this groundwork, Jordan would go on to make Prince of Persia years later, and bring these efforts to another level entirely. He also endeavored to make the Prince of Persia code as portable and documented as possible, so different platforms would have similar experiences in terms of playing.

In 2012, Jordan released a new remake/reboot of Karateka, which is also cross-platform (the platforms now being PC, iOS, PS4, Xbox and so on) and is available at KARATEKA.COM. It is a very enjoyable remake. There are also ports of “Karateka Classic” for a world where your controls have to be onscreen, like this one.

In a larger sense, it’d be a wonderful world where a lot of old software was available for study, criticism, discussion and so on. We have scads of it, of course, but there’s so much more to track down. It’s been a driving effort of mine this year, and it continues.

But for now, let’s enjoy a really, really unpleasant but historically important version of Karateka.


MAME 0.175 Arrives and the Emularity Follows —

Just a quick update on the whole “JSMESS is now just something MAME does and so we should theoretically be able to update the Internet Archive’s emulation quickly” front.

It works.

MAME, that lovely thing, went from version 0.174 to 0.175 yesterday. It was pointed out to me pretty soon after the program dropped. Taking notes for later instructions, I began a process of compiling the new versions of the in-browser emulators for the Internet Archive. I used a “weighted library” approach, where the emulator with the most individual items in it (that would be the ZX Spectrum, at a svelte 20,000 items) gets compiled first, and then the next largest set of emulated items, and so on. There are roughly 700 emulator drivers on the Emularity at the Archive, but only roughly 30 have more than 1 item.

So, what this means is that within about 90 minutes of being informed about MAME 0.175, the Emularity on the Internet Archive was upgraded to match for 99 percent of all the items. 

The other hundreds of emulator drivers look to take about 12 hours in total to compile and install, but it’s pretty much 100% automatic, so it’s not taking a lot of time.

So consider that process doing extremely well.

But more than that, it’s definitely going to be a case of pushing for more and more people to contribute to the MAME project, because as proven here, if someone makes wholesale improvements to emulation support for something, the Archive will reflect it within 30 days. If the word comes down that the new thing is so amazing it can’t wait, it’s possible to declare that driver to be done immediately and updated to an in-beta version. Really, this is when I hope the advancements come. I hope it’s just a case that really talented programmers looking to make the world better just don’t know how easy it is to contribute to MAME now, and that it’s in github and open licensed, and once they find out, amazing optimizations and expanded work will make itself known.

We can hope! But until then… upgrading is definitely a snap now. Enjoy the new versions of everything.


A Showcase for a Terrible Monitor —

arcade_crtdemo_screenshot

Some time ago, I wrote about the amazing efforts to bring a realm of accuracy to emulation, especially arcade machine emulation. The work was (and is) amazing, but one problematic aspect was that the main components for this work in MAME were for the Windows version only, with plans to make it more cross-platform down the line.

The fact that it is down the line and work has been done and we’ve improved the turnaround on the compile time from “new version of MAME released” to “the new version of MAME is running at the Internet Archive” meaning that we can finally put up a nice example of this wrinkle in emulating the vintage experience.

So, by visiting this item on the Archive, you can boot up an arcade machine that is blurry, has ghosting, is slightly curved, and has bleed through some of the pixels and an obvious set of scanlines and reflection….

hooray

Seriously, though, this is incredibly important news. It’s a living demonstration of the hurdles and considerations of ’emulating’ older technological experiences. It’s not all ultra high-definition happiness and razor-sharp graphics over surround sound blasted through bluetooth speaker bars. It’s about capturing components of these games that are coming from a different realm than just what the code had to say about it. Between 50 and 80 milliseconds after an emulation is working, people come out to say “it’s not the same, it doesn’t have _____” where _____ is an ever-more-picky set of attributes that makes the experience of the game “real” to them and which they think ruins the entire emulation if the attribute is not right there.

Bear in mind that the potential additional settings for these monitors being emulated are many times more complicated than in this demo, and that the higher the resolution, the better – because now you’re not just emulating the pixels themselves, but the actions and effects around those pixels.

Welcome to the cutting edge of the cutting edge of the blurry edge.

crazykongMarquee2

The game I chose for this particular demo is its own nutty story: Crazy Kong. As recounted in The Secret History of Donkey Kong article, and not in Wikipedia any more, Donkey Kong was actually programmed by an outside party for Nintendo. (It was designed by Nintendo, for sure.) This same outside company went on to do other games you might know, like Zaxxon and Congo Bongo. Part of this is that Crazy Kong is not a bootleg of Donkey Kong but a legit license.

It’s also terrible, and wasn’t supposed to be in the US, but then with the skyrocketing success of Donkey Kong, it ended up here in bootleg form.

So, for me personally, Crazy Kong brings back memories of being one of those games shoved into bowling alleys, pizza places, and in shifty locations where the purchase of drugs heavily overrode the quality of the arcade game experience. It seems only right, then, that the slightly crappy monitor panorama be bestowed upon this emulated game, brought up from the depths. I know that’s how I experienced it so many years ago, and you can experience it now.

Some notes: The resolution on this web-based emulation is much higher than the usual games in the Internet Arcade, mostly to really bring out the qualities of the monitor properties. This might slow on some machines, or annoy with some different setups out there. But setting the whole thing to fullscreen will probably make it better for you. As is still the case, Firefox tends to run it better than Chrome and both better than Microsoft Edge (for now) and Safari. You can also always go back to the non-CRT emulation on the site to compare.

A huge amount of applause to the MAME development team for this years-long effort to emulate this important aspect of the vintage experience, and a thanks to everyone who helped make it run at the Internet Archive.

Now get blurry.