The Emularity — January 23, 2015

Last week, on the heels of the DOS emulation announcement, one of the JSMESS developers, James Baicoianu, got Windows 3.11 running in a window with Javascript.

That’s impressive enough on its own right – it’s running inside the EM-DOSBOX system, since Windows 3.x was essentially a very complicated program running inside DOS. (When Windows 95 came out, a big deal was made by Gates and Co. that it was the “end” of the DOS prompt, although they were seriously off by a number of years.)

It runs at a good clip, and it has the stuff you’d expect to be in there.

Bai, tinkerer that he is, was not quite content with that. He wanted this operating system, sitting inside of a browser and running in Javascript, to connect with the outside world.

That took him about 3 days.

That’s Netscape 1.0n, released in December of 1994, running inside Windows 3.11, released in August of 1993, running inside of Google Chrome 39.0.2171.99 m, released about a week ago, on a Windows 7 PC, released in 2009.

And it’s connected to TEXTFILES.COM.

Windows 3.11 definitely works, and all the icons in there click through to the actual programs actually working. You can open solitaire and minesweeper, you can fire up MS-DOS, you can play with the calculator or play audio, and you can definitely boot up Netscape and NSCA Mosaic, or mIRC 2.5a or ping/traceroute to your heart’s content.

The world these Mosaic and Netscape browsers wake up in is very, very different. Websites, on the whole, and due to the way this is being done, don’t work.

It turns out a number of fundamental aspects of The Web have changed since this time. There are modifications to the stream that can be done to get around some of this, and we’ll have screenshots when that happens. But for now, the connections are generally pretty sad looking.

To connect to the outside world, the Windows 3.11 instance is running Trumpet Winsock, one of the original TCP/IP conversions for Windows, and which uses a long-forgotten (but probably still in use here and there) protocol called PPP to “dial a modem” (actually, connect to a server), and transfer data to a PPP node (really just a standard web connection).

This means that somewhere, this instance needs to be connected to a proxy server, which assigns a 10.x.x.x address to the “Windows” machine, and then forwards the connections through. Basically, world’s weirdest, most hipster ISP on the face of the earth.

In other words, this is janky and imperfect and totally a hack.

But it works.

It took about three weeks after I decided we needed to go with EM-DOSBOX in addition to JSMESS to work with DOS programs, that we had it up on the archive.org site and going out to millions. It has taken two weeks after that for this situation to arrive.

Contrast with how it took poor Justin De Vesine, working hard with Justin Kerk and a host of other contributors, eight months to get JSMESS’s first machine (a colecovision) to run at 14% normal speed inside a browser, for one cartridge.

Welcome to the Emularity, where the tools, processes and techniques developed over the past few years means we’re going to be iteratively improving the whole process quicker, and quicker, and we’ll be absorbing more and more aspects of historical computer information.

Now the stage is set – the amount of programs that can be run inside the browser is going to increase heavily over time. The actions that can be done against these programs, like where they can be pulled from or pushed out to, will also increase.

What becomes the priority (as it has been for some time) is tracking down as much of the old software as possible, especially the stuff that doesn’t sell itself like games or graphics do. I’m talking about educational, business, and utility software that risks dropping down between the cracks. I’m talking about obscure operating systems and OS variants that fell out of maintenance and favor. And I’m most certainly talking about in-process versions of later released works, which could stand to be seen in their glory, halfway done, and full of possibilities.

Documentation for the software just skyrocketed in value – we had bai reading 1995 books on PPP troubleshooting to get things going. MS-DOS programs on the Internet Archive will need links to manuals to become more useful (this is coming). And just grabbing context will continue to be a full-time job, hopefully split among a group of people who are as passionate as the folks I’ve been lucky enough to come into contact with so far.

I can entertain debates about the worthiness of this whole endeavor as an abstract anytime anyone wants. But the flywheel’s in motion. It’s not going to slow down.

We’re there.

Update:

I should have known this was click-juice. Welcome everyone. To speak specifically to folks who “just want to try it”, I ask for patience in terms of this being available to try – it’s still so new and fragile, and frankly, it doesn’t help to have thousands of people hit on the thing, go crazy when it acts weird, and complain bitterly.

If you’re new to the Javascript Emulator party we’ve been throwing for the last year, may I humbly suggest visiting the Internet Archive’s Console Living Room, Software Library and Arcade? With over 25,000 items to try, there’s plenty to keep your attention before the next generation of stuff becomes playable.

19 Comments

The Haze of Possibility — January 22, 2015

The game Grand Theft Auto V will send you on a large variety of missions and quests, many of which will require travel between the southern, more urban locations and the northern, nature-filled expanses. A number of roads and paths connect these areas, including mountainous trails, single-lane routes and a number of highways. One of these is the Great Ocean Highway, a multi-lane affair that swings along the west side of the island and into the city.

The day in GTA V is shorter than “real time”, so it’s a good chance you will catch yourself on this highway in the sunset or sunrise, the twilight hours, the golden time. This popular highway is filled with cars making their way north and south, driving at various speeds into the city or away from it. Among them will be you, often caked with blood or carrying passengers to the next quest, or both.

In the mile or so before you get into the city, the view ahead of you stretches far into the distance, and besides the lights of buildings and skyscrapers intermingling with the stars, comes the boardwalk, bright and beautiful, a hazy line stretching into the ocean. At the end of this horizon line of glowing activity is the blur of the ferris wheel and roller coaster at the pier’s end, turning ever so slightly, twinkling into the darkness.

That view, that hazy view into the boardwalk, has a lot of special, internal meaning to me. It’s a view that promises, just over the horizon, the fun and excitement I might be wishing it was part and parcel of. I didn’t always have the free agency to find my own fun – sometimes I’d be somewhere because the family or the group I was with was on its way to another function, one not as fun, and seeing the glow and promise of things in the distance might be all I got, and I’d craft a dream and hope from it.

Grand Theft Auto V is a marvelous thing, technically. The world is full and real. When I run into the edges of it, the places the engine and the construction fall short, I am so far deep into a situation that my complaint seems ridiculous: Emergency services fail to arrive when I explode cars at busy intersection. Strip club appears to have no roof access to service air conditioning, requiring scaling from other nearby buildings. Specific style of taco truck seems unnecessarily prominent around city.

But far and away, the most brilliant, beautiful thing is the lighting, which ranges the gamut from sun showers to cold nights, and from summertime-quality days to buzzing, foggy evenings. To remember the process of generating a functional, single-color sprite on my Atari 800 by calculating the binary behind each set of 8-16 pixels, it has been a privilege to witness how far the technology has come in my lifetime.

That a moment in the game, a simple one of driving into the city on the Great Ocean Highway, music blasting on the radio, could call me back to a feeling and emotion that has been a part of my life since the beginning… well, that’s maybe the highest compliment I can pay.

2 Comments

The Digital Nostalgia Heat Differential — January 13, 2015

This post is, once again, my favorite kind of idea: a terrible one.

If you’re walking into the emulation-in-a-browser-thing cold, then this big massive essay from me is not going to be of interest. And if you’ve read my stuff before, it’s going to be a bunch of stuff you probably already knew, presented to you again, and concluding the same way.

But, this is the year I’m taking the in-browser stuff on the road, and I also intend to step up my game with the presentations, and so it’s going to be a few rounds of stating my philosophy and motivations as the Software Curator at the Internet Archive on this site, because it’s just going to be a year of that.

So let’s work it over again.

DCP01104

Here’s me in 2002, finding a pile of old game consoles in a room of technology and craziness at the Hackers on Planet Earth convention in NYC. And I spent some time doing what I like to do – arrange stuff for ease of use. So there’s a fully-realized demonstration station, with easy-to-find access of the cartridges, and a little do-it-yourself for controllers.

Naturally, this setup was doomed – it was only alive for 48 hours, and I’m glad someone grabbed a snap. Along with my short-lived luxurious flowing locks period. But the exercise, with its flaws, and its neatness, was worth my time. People played games they’d only heard of, I saw parents use it as a teaching tool (or change to wool-gather) for kids, and there were surprisingly few douchemobiles.

Even by then, Emulation was a Thing, a Thing that had caused no end of controversy, and had been a boom, a bust, a craziness. I contributed what I could where I could, although my technical chops are seriously not that good. Mostly I saved things, or wrote a few things here and there, and mostly I just enjoyed the magic of what emulation provided, and dreamed.

Both these situations have a place – emulation, growing and controversial, could do things that the game setup of a vintage system could not, and vice versa. They both had long-term issues to face, but don’t we all. And for that time, and even into now, both can continue on quite happily, and both approaches exist.

This will not always be the case.

In the beginning, was the software.

As The Narrative goes, Software was not thought of as a separate, meaningful product until IBM, realizing they were really getting themselves set up for an anti-trust lawsuit, started to unbundle software/programming from hardware to head off the interest of a government intervention. A gambit, by the way, which failed miserably; but in doing so, IBM had unwittingly birthed this new industry, one that has grown into the juggernaut of what it is now and which so many of the raised voices of the Internet World seem to dabble in directly or indirectly.

The Narrative is problematic out of the gate, because the whole point of Software as a concept is that it’s the manipulation of materials towards a goal, and if you’re willing to accept that definition of mine, then music, painting, poetry, writing… all software. Just focusing on the piano roll industry of the 19th-20th century will do to demolish the timeline of 1960s IBM creating “software”. We even have the old “disk images” of the piano rolls around, both as paper, MIDI files, and digital recreations that make 100-year-old player pianos do their thing.

But rest assured, the idea of there being at least some sort of definable, distinct and of course recompensable art in the endeavor of the manipulation of electronic signals has roots going back a very long time. There are the usual tales of exploitation, misappropriated credit and tribute, and the triumphs of will and talent. But it’s a real thing. In the beginning is the software, and the software persists.

My particular interests stem from the late 1970s to the early 1990s, but that’s just a personal bias. After all, it reaches from my childhood, a time of great turmoil and misery punctuated by the ecstasy of the promises of computers, leading up to my early 20s, when the Internet truly takes hold in a meaningful way. Maybe the Hunter S. Thompson quote applies here, but in the geekiest way possible. Regardless, the love continues, and I have never stopped adoring the memories of the time.

There’s two ways you can really go in the old computer biz, once you decide you like it. You become a collector, or you become an implementor. You can be both, but I find you don’t do one of them particularly well. So let’s keep it separate.

Collectors gather up all the materials, be they physical or digital, related to their area of interest. Motivation is variant, as is justification. Some people just like having a lot of stuff they like, within arm’s reach or a basement’s trip, and arranging or rearranging materials to suit presentation or a perceived permanent storage. Personalities range, as does how much time spent with them is enjoyable. Like anywhere else, infighting and competition occurs, along with breathtaking efforts to maintain their specific pile to the highest degree.

Implementors have a second layer of creation applied on top of collections, be they collections they maintain or by referencing the collections of others. In the very specific world of vintage computing, there’s been an explosion of newly created devices for reading, recreating, modifying and manipulating older hardware. Think “SD Card Reader for a Machine That Could Not Possibly Have Needed an SD Card Reader until now”, but there’s much more to it than that.

For the sake of this classification, I’m willing to lump academics into the Implementors camp, even though they’re primarily “implementing” classes, books and writings.

So into this area of study, which I keep calling “vintage computing” but should probably just be called “computer history”, comes the entrenched collectors, the implementors old and new, and the audience related to experiencing all of this, in whatever way they choose to.

Welcome.

Into all of that situation, then, comes the “implementation” I’ve been on for a few years – throwing bootable computer simulations into the browser. Nothing, going back to the beginning, was intended for this to happen. Executing code within the browser’s guts itself, especially exceedingly complicated code like an emulation system, is way out of bounds of original design parameters. But here we are.

At some point, every project, even one trying to do as much of the “right thing” as this one has, will hit the bedrock of dogma and architecture. I may be happy that the toolchain components (MESS, Emscripten, Most Browsers, Javascript’s Standard) are all variant amount of “open” instead of non-variant amounts of “proprietary/closed”, but eventually even that’s not good enough, or is irrelevant to the goals of someone coming in from the outside.

So file the following back at shouts in the darkness on choices.

Java was supposed to be the solution, but I don’t like any way that Java has worked out. It’s a mess on ownership, standards, and implementation. Also, it’s friction-filled – you not only have to install a plugin into the browser (not allowed in many environments), you have to make sure it’s updated (it’s a security nightmare and some browsers expire the Java plugin, forcing a re-install). I just didn’t like a “all you have to do is install [scary thing] and you get access to the old software” as an overall design. I also think “install this plugin” is on the way out as something browsers will do – it’s just plague-ridden as a situation. More on Java/Flash as a solution below.

Javascript has its own limits, but it works in all the browsers, and (assuming you didn’t run into a gotcha), when I take machines and run a fullscreen hit of an archive.org-hosted game, it’s quite breathtaking to realize this is Javascript blowing through a browser and no other trickery enacted. I still get a joy out of that.

Making it run in browsers as opposed to the desktop is also huge to me. Browsers boot everywhere, in everything. The more things that run in browsers, the more people it runs into. Desktops (and straight executable code) all have advantages, but they lose the factors that are important to the project.

So, let’s step back.

Already, baked into the choices, are the fundamental philosophies of the project and what I’ve been going for. It’s been a learning experience over the last decade or so to see the sparks between my belief system and an audience when not everything is in sync. Oversights become intentional sins. Advantages along one parameter seem like needless obtuseness over another. “Hate” is used as a descriptor. So are “broken”, “forgot”, and “stupid”.

Let’s cover the factors involved in the project thus far, then.

The belief that the originals can’t last. Hardware fades out and dies, disks demagnetize, cartridges break, tapes are lost, and documentation hits the trashbin or the trash fire. In the long run, saying “I’ve got a machine in my basement” isn’t going to cut it.
Belief that the software of the past should be accessible to as many people as possible, as quickly and as flexible as can be, and in a way that can grow and change to continue to be accessible for the future.
The idea that “friction” is a major problem in older software, that outside of the masochistic delight a small crowd takes in how much effort it is to set up, the action of clicking a single URL and getting, say, a 1983 Apple II program running in it, is a new level of frictionless experience.
As little manipulation of the original software as possible, or, barring that, having it easily seen what manipulation was done, so it can be undone going forward.
Function, after a fashion, as a sort of talisman-proof for a variety of additional arguments, by proving the data/software can be used in a contemporary sitting, and that it has continuing value. I.e. “Oh, we might want to look into providing a software archive if this can happen to it.” or the all-important “I guess I better contact this Jason guy about these old floppies I was going to throw out because I thought they were doomed.“

I’ve been happy with these factors – they’ve solved the project well.

There are also factors I don’t like, but are also a part of this:

100% volunteer. I technically get paid to do this, but it’s a sliver of my total duties at the Archive, believe me.
QA testing is “as best as possible” and “let’s see what happens when people meet it for the first time”. Again, no budget.
Multi-layer failure vectors that overlap in terms of performance. I.e. Archive.org infrastructure issue, Old Browser, Broken Browser, Slow Machine, Slow Network, Odd Expectations, Plugins That Block Javascript, Web Compatibility is a Flaming Viking Ship Burning In Your Swimming Pool And No Fire Extinguisher in Sight

Call me biased, bitter or merely observant, but I feel like the trade-off for no-budget has been that we’ve been able to move rather actively in doing proof-of-concept things in the main project. I have a very dim view of how institutions do this work, and especially how they tend to smother bizarre or off-ranch approaches like this so nobody gets “hurt” or nobody feels any lines were crossed. It has been a very strong indication that I’m working for just the right place that they’ve let me get things as far as they have, with some of the rocket ship being made of corrugated cardboard and a distinct smell of burning almonds just before we hit orbit.

There are other platforms to bring (Macintosh, Commodore), but before I think we bring those under the fold, I want to go back and really nail out the QA and documentation of what we’ve done so far, including making distributable versions of JSMESS and EM-DOSBOX so that archive.org isn’t the only place running this show. I want those kind of minds that want to iteratively improve the underlying programming, and I’ve discovered they’re rare indeed. Maybe 2015 is the year this whole emulation-in-browser thing gets the Big Cleanup, while I stomp the ground to talk about while it’s so important. That seems like a good blend.

Next: The World.

The MS-DOS section went big last week. Real big. We’re back up into the millions of new visitors to the Archive, which, when combined with the last year and understanding the overlaps, at least means the place is growing, in terms of the audience and awareness, which are important things in the standing and future of this Internet Archive thing.

Along with the weblog entries, the youtube videos, the radio and TV mentions, and the discussion board flare-ups, I’d estimate I’ve read roughly 4,000 comments related to the MS-DOS collection and this whole project.

So. now that I’m back from the eyewash station,… I have a whole bought of thoughts, reactions and obvious points in need of clarification.

As is usual, for everything ever in the history of internet communication, there’s a few phylum of responses that just always latch onto anything that gets thousands of eyes. They are:

This is all bullshit, I did this years ago in a better fashion and do it 10x better daily while wearing five hats
It did not work perfectly, it was like watching my dog die
Look at me, look at me, look at meeeeeeeeeeee

The first one’s got my interest as a topic of discussion. The other two don’t.

Regarding “This was all done better” posts, which usually make an appearance at the beginning or the end of a flood, I find that’s because there’s some distortion between the project intentions (stated above) and the perceived intentions.

For example, EM-DOSBOX is a port of DOSBOX into Javascript. Therefore it has some features that DOSBOX does not (the ability to run in a browser, detachment from a specifically-installed machine for your DOSBOX needs) and has dropped features that DOSBOX has (integration with serial ports, ability to mount local filesystems into the DOSBOX environment, etc.)

This means that I can say “Go here to try out MyMan, a Pac-Man clone made for DOS in 2000.”

The good news is you go there and it instantly happens.

The bad news is that if anything goes wrong, people instantly complain. Harshly.

They especially complain because it “works great” on the desktop version of the emulator. What they’re skipping is that as good or bad as it is now, improvement can happen, but you’re just never likely to get to a point where you say to someone “Try out MyMan – click here for a .exe file that will install DOSBOX on your machine, load MyMan, and allow you to start playing it.” EM-DOSBOX wins on Friction.

But if that friction is truly meaningless to you, you are one unhappy puppy and there’s a delicious, grape-flavored comment box just below the statement for you to fill with anger.

And anger! I thought the Internet was a machine for making copies. Turns out it’s a machine for facilitating anger.

A large amount of people have very poorly-maintained machines and browsers. That’s just always going to happen. Also, some people are running machines that are very, very old. So any experience that depends on their browser performance is probably going to be odd and possibly infuriating. (A note to myself is that we will probably make some “evaluate your system” process where you click on a URL and you get told about potential ways to make your system run better, or improvements that might need to be done.)

Some games run too fast due to a speed differential, however – that should be a shocker; the javascript implementation of the DOS emulation being too fast – and we have a solution coming very shortly.

The way that JSMESS ran in October of 2013 when we first announced it and the way it was running by the time of the Internet Arcade was night and day – we did so many improvements, revisions, updates and continuing efforts to jack up speed and responsiveness. That came from user response and improvements along the toolchain. I don’t see that trend changing going forward.

EM-DOSBOX improved markedly from last week to this week, and improvements continue. So, ultimately, playing the long game (and the Archive is nothing but Playing The Long Game). “It Just Works” is a definite goal.

Through the 4,000 comments, a lot of it is people arguing with each other, not putting forward any ideas, and obviously a microscopic amount of offers to help or assist. But there were some excellent pieces of feedback; the most marked being that people wished they could “save games” and come back to the site to continue them. I’m going to be honest – that never even occurred to me, that people would be intensely playing some of the longer games in the collection, and have to save them away for later. Looking back, of course, it’s obvious, just like a lot of the situations with “obvious” features. It turns out to be an involved solution, but one that’s being looked at.

There’s a little part of the mindset in the modern world that’s been interesting in all this, as well – kind of a perception that every single website, no matter who or what is behind it, is a Product.

Products have guarantees. Products are out to kill and murder competing or similar Products. And Products owe you something, because you walked in the door.

Maybe it’s related to the announcement and interest of the games aspect of all this. There’s certainly a rather bloody-knuckled approach to the millions who showed up, found some things working, some things rough, and then decided it was a Failure. I don’t think there’s a cure for that mindset.

Instead, as mentioned, I took what I could from it – improvements in the UI, changes in the options, and definitely improving the way the games can be tuned to work in the browser as expected. And that save game function!

I don’t know, at this start of the year, where we’ll be at the end of it. But it’ll be interesting to watch the journey. We’ve got years of work behind this now, and a year of effort ahead.

Let’s see where this takes us.

4 Comments

Love and Noir in the Time of Ebooks — January 11, 2015

I first met Chris Orcutt at the age of 11, and we both remember it exactly the same way – the wall at the condominium village we both lived in, who was hanging around, even who was standing where. 2015 is 34 years later. 34! That is a very long friendship.

We’re now in our forties. I’m a filmmaker, archivist, annoyance. Chris is a writer.

Here’s a good one of Chris these days, at his writing desk:

It’s a still from from a film I did about his insistence on writing his first drafts in pencil. (The film is here.)

Chris photographs well.

However, we usually look like this when we’re together:

We’ve both moved around a lot, but for the last few years, we’ve lived within a few dozen miles of each other, and getting together on a semi-regular basis is the easiest it’s been in over a decade, so we’re making the most of it. Good lunches, good walks, good conversation.

In his early 30s, Chris became a full-time writer. He’d been a writer for years before that, including for a newspaper, as a speechwriter, and as a copywriter, but after a certain point, he and his wife made the decision that he would dedicate himself, full-time, to writing. Writing novels and short stories and plays. Nothing else.

They’ve kept at that choice, with various amounts of hardship attendant, since then. Chris writes. He gets up in the morning and he writes the entire morning, then either does some reading or other research. Pretty much every day. For years.

∴

Back in his twenties, when we were both going to colleges near each other in Boston, my friend started taking a poor turn. Drinking got amusing, then prodigious, then alarming. It got to a point I simply assumed my very good friend, my best friend, was going to die, so I tried to enjoy what time I had with him.

As he has discussed in the open more recently, Chris had an un-diagnosed bipolar disorder. With medication and medical supervision, it’s under control, the drinking is rare, and my friend is alive to this day. With this control came the ability to focus on tasks and quality of work that shows in his efforts since then.

And that effort, I will say one more time, is writing. Pretty much every day. For years.

∴

The result of this writing has been a collection of novels. They include a three-novel detective series, the Dakota Stevens Mysteries, and two short-story collections.

They’re good. They’re unquestionably good.

He has not wasted his time, his years, poured into these projects. I’ve not had to look away and make excuses for them, or call a glue-ready horse a racing champion. He’s done fantastic work here, and will continue to do so.

∴

Here’s why I’m talking about this.

The third novel in the detective series just came out. Of his going-full-time novels, this is the fifth overall. He and I both do these long-term projects (he the books, I the movies) and we don’t look at each other’s stuff, in-process – we wait until the end and enjoy it with everyone else. So on the day it came out, I bought and downloaded the third novel, A Truth Stranger Than Fiction, and damn if I didn’t devour that thing in a day. I read very fast and vociferously, but I take the book in along the way. And the book’s good. Damn good.

While I didn’t read the in-process novel, I went to a lot of lunches with Chris during the year he’d worked on this book (and he’d written drafts before the intense year) and I know how many times he read the thing, how many drafts he pounded out to get it where it was, and how much editing and polish he put on it, across months, to get the book where it is. This thing’s a craftwork wordsmithing of the highest order for Chris. You may diverge from opinions stated by characters or you may be surprised at which way things go, but none of his books, none, have the sour feeling of a good idea burdened by crap writing. He wrote these for the ages.

This turns out to be a problem.

If you haven’t observed this already, I’m a whirlwind of input and output. I splash through a hundred hangouts, online and off, a week. Tracking down the most minute of factoids just to prove a point in a web-based conversation I wasn’t even in two hours ago is my idea of a good time. Shit howdy do I like splashing through life’s intellectual wading pools and swinging a bazooka of manic lyrical fireworks. Pump me full of the hottest images from imgur, the latest insanity out of hackernews, the twitter tornado of the moment. My tolerance for the always-on glut of intellectually stimulating-and-stultifying river of human experience is legion. Bring it.

Not so much, with Chris.

The rough years of his twenties and the writing years of his thirties and forties have diversified the two of us. In our teens, we were both pretty hyper and bring-it kids. I’ve turned that up to dangerous, slap-a-warning-label-on-it levels. Chris has instead focused on his craft, his skills, in the art of writing. It’s made him different. Not less of a person, mind you… far from that. It’s his years of focus that’s made him really great at the writing he’s aimed his life around. He is the monk, waking to the cold water of the stream and practicing, and I am the insane brother living deep in the city, buried under twelve crazy deals and all kinds of trouble. Different choices.

All said, we still get along.

But it does mean that each time Chris emerges from this thing that means so much to him, this novel that he’s poured hundreds, thousands of hours into… he just hates the way things are. It’s like he’s brought the best pie he could bake to a fair where folks are gleefully handing out Hostess Fruit Pies and crowing at how the flaky crust and the cherry-flavored-filling is a meal fit for a king.

He gets really sad about it, sometimes. Books were, and continue to be, special. But the modern world is working very hard to make the kind of effort Chris puts into his part of the writing landscape a waste of time.

What he wants is for the books to be available in 1975, making a splash with the effort he’s put into them, meeting up with others doing equivalently hard work, and taking his residuals while taking his wife to exotic locations to do research for further writing.

What he gets books are available on Amazon. Amazon takes away the problems of a warehouse of rotting copies that your publisher isn’t promoting, and enables to-the-moment direct control by the author of the sales and pricing of their book. For that advantage, Amazon has progressively, and I do mean progressively, destroyed writing books as a career for any writers who are not in some way self-promotion machines or blowing three quarters of their time that should be spent on their books, on blog entries meant to drag in readers.

Amazon’s structure favors people shooting out a bunch of crap, in serial form, to constantly stay in the promotional frame. Amazon’s routines disfavor older books, with “older” being a month or two ago. And ever-present “sales”, “prime” and “deluxe” situations are existing where Amazon is heading towards making books into an “all you can read” Netflix-like situation, where people pay Amazon $10 a month and they get all the books they can slam down their gullet.

This is not what he signed up for.

I’ve done this rodeo before.

Here, I’ve told you my friend’s craft and pain, and the more passionate reactions will tend to be along two lines of thought, with a bonus track.

The two lines are

Fuck your buddy. Sorry to hear he came down with the born-too-lates. It’s a world of self-promotion and staying-with-the-times and he better learn those skills as well as he’s apparently learned to write. Or get a second job.
Writing has never been a self-paying job – congratulations on learning what we all knew.

(The bonus track is, “oh, a book series to read, I’ll check it out.”)

So, both of these are wrong for what I’m talking about here.

First, I think the world is seriously suffering when everyone is writing garbage teenage supernatural romance with a chapter’s worth a month. I realize there has always been crap and Sturgeon’s Law reigns supreme, but that doesn’t mean Sturgeon’s is an end goal. I make the argument that there’s got to be some middle ground – that as we’ve been working so hard to make it easy to transport electronic books in seconds and share them among ourselves in seconds more, that we lost some of that imprint feeling, of knowing a house of publishing was going to give us winners and it was worth the effort to seek out the newest titles. I think it’s a problem, and a problem that needs solutions.

Second, there have been writers who have made it a career, after they put in their dues. Chris has, trust me, put in his dues. After saving for years, and with a boost from sales from a book, husband and wife took a trip to Paris (which was then used for research for a later novel), and this was the first time they’d done any vacation in a decade of marriage. Dues? Paid.

I can’t wrap this entry in a bow. I can’t even bring you the place the plot is going to go. I don’t know what’s going to happen. My friend has skills that outstrip mine in many ways, and I get lots of goofy press for my efforts, while he works in relative shadow.

I thought, maybe, I’d shine a little light on a friend’s lifetime effort, since it deserves a beam many times brighter than it has gotten.

And maybe, just maybe, send them to Paris again, one day.

6 Comments

That Whole Thing With Sound in In-Browser Emulation — January 10, 2015

We’re past 4 million verified MS-DOS players on the Archive this week.

So that’s something.

It’s time to really lay out the floating elephanty-thing related to this whole in-browser emulation kick – sound.

Sound on the in-browser emulators (be they JSMESS, EM-DOSBOX, or others) suffer from sound strangeness. In JSMESS, it works fine except when it doesn’t. In EM-DOSBOX, it works, but it does this kind of strange fluttery-ness that, if you’ve got experience with the originals, you notice right away.

Both are the results of a specific situation with browser audio, and the work that the Archive has been doing to bring this all to a mass audience is bringing the situation to the forefront. I just needed somewhere to send people who go ‘sound seems weird’, and so here we are.

One of the side effects of this project is that I had to spend years following the week-by-week advances and activities of the current lead Web Browsers in how they handled things like Javascript engine improvements and other features.

As might be expected, I’m not a fan.

The rest of this is insider baseball, but if you want to know why audio has an issue, here it is.

Currently, there’s four active large-scale browsers with any traction and attention. They are Chrome, Firefox, Internet Explorer, and Safari. There’s others, just like there are other land-line telephone companies and there’s other network hardware companies and there’s other search engines, but come on. As part of the testing procedures for JSMESS I took a sample of the other browsers that were hitting the Internet Archive and installed all those things. Many of them are Chrome and Firefox, recompiled with features added in or features ripped out as the tastes/anger of the maintainers dictate. This just adds more lint to the Big Four.

(Also, as regards Safari, I’m being really nice including them.)

With four browsers to consider, it then comes into play about how they will handle certain standards so that something that appears on one browser will appear some level of the same on the others. Examples include rendering of fonts, tables, adding buttons, allowing control of keyboard, and so on. On the whole, generally, the world has gotten by.

Here, at the beginning of 2015, the big drag is Internet Explorer. Forget the future and the present; the past has been rolling updates for browsers, so that Mozilla/Firefox and Google/Chrome are basically making major, wholesale changes to the operations of their browsers a few times a month, as it suits them. Safari sort of follows suit with much less frequent work, but the system is in place to allow them to do it when it pleases them. Meanwhile, Internet Explorer drags along, basically doing a major scary X.0 release every year… except 2014, when they didn’t do a single release.

Here’s where we start to run into the issues with stretched supply lines, a la warfare, except here the war is to bring runtime-quality items on webpages.

Firefox and Chrome both have in-house speed-up flav-o-paks for making their stuff run quicker. Firefox has ASM.JS, and Chrome has NaCL. ASM.JS will work in all the major browsers, and NaCL is kind of a sleight of hand trick. They both want the same thing – to cheat out certain programming actions and runtimes so that they run very, very fast in the browser. ASM.JS, since it’s basically Javascript, will run in the other browsers, but Firefox has gone ahead and written in-program routines for noticing ASM.JS is in place and then optimizes it further. Therefore, programs like JSMESS, that use Mozilla’s Emscripten compiler to make their run-times, run significantly faster in Firefox then Chrome. In an ideal world, the end-user doesn’t notice because once you get to a certain speed with, say, emulating DOS or a video game, you just have it. It’s done. But it’s not an ideal world.

So, essentially, JSMESS and EM-DOSBOX run pretty fast in Firefox, run a little slower but fast enough in Chrome, run notably slower in IE, and run OK in Safari. Tiny-share browsers using Firefox’s engine therefore get the speed boost, and the Chrome-based browsers get that amount of speed.

IE could release another version soon, and blow everyone away, and then another year will happen. For better or worse, they really should be on the same ridiculous twice-a-month schedule the other browsers are rocking.

Let’s step back.

All of this back-room crapola is part and parcel of the entire software industry as a whole -there’s nothing special about it related to browsers. The only reason it even gets attention is because the nature of the web has, for the moment (a moment lasting 20 years) forced natural adversaries into cooperation, because initial attempts to be “THE” browser for the web by both Netscape and Microsoft fell on their face. In other words, because nobody could really “own” the way webpages were being made, all the browsers had to do work to make themselves at least superficially coherent when it came to standards.

Along the last 15 years, there’s been weird little deals and tricks employed to make one browser seem “better” than others. For example, IIS webservers (made by Microsoft), if they detected they were dealing with Internet Explorer, went into this kind-of-not-a-standard mode that basically wrecks TCP/IP but ensures really fast data transport. (In high-level terms, it stops waiting for acknowledgement and just blasts the data.) Google added some really persistent programs running in some operating systems (Google Crash Handler, among others) that were “helpers” that gave their browser some resilience and options. And now we have ASM.JS and Firefox’s both providing the new idea and the others not really wanting to endorse it in any meaningful way.

Most of this is very boring.

But let’s proceed, in this fogged landscape of standards and trickery, to sound.

So, you start up a game of Smurfs: Rescue in Gargamel’s Castle, and it either works for you, or it doesn’t. Generally, it will run at full speed in all browsers. Sound, however, is another situation entirely. When the game starts and the music begins, it either sounds clear, or it does not.

The reason for how it sounds is multi-fold.

First, when you listen to an audio track (or a video track with audio) or you play a javascript-native videogame, or anything that’s using audio in a webpage, in general these items have a set of pre-made, rendered, finished audio files in them. An audio track is obviously that – a multi-megabyte .wav or .mp3 file, and a video has an audio track that is same. A videogame will have a little pile of samples. All the audio routines in browsers assume this is how the world is, and what the world will be. And generally, that has been the case and all has been good.

JSMESS breaks that, as does EM-DOSBOX.

Instead, these items are generating chunks of data, live, and mixing them as they go. They’re not preformed samples being pushed out – every millisecond is brand new, and won’t repeat.

The current standards don’t like this situation at all.

Mozilla had cooked up an audio standard that sort of dealt with mixed audio, but that standard has been removed in favor of a shared standard called the Web Audio API. Here’s the current version of that standard. As you can see, Google and Mozilla have people working on it. Microsoft isn’t evident, but who knows. Apple is MIA as well.

I’ve been told that if this standard gets formalized and gets implemented, it will just “fix” the audio problems. It accounts for programs like JSMESS/EM-DOSBOX doing dynamically created audio. Emscripten will then observe this standard, and then compiling JSMESS will push it through and we’re in happy audio land.

I want nothing more than this. I have nothing I can contribute directly to the standards bodies other than to say it’s obvious millions will benefit.

So that, in a nutshell, is why audio currently has problems in the in-browser emulators.

But let’s go one step deeper.

At the Chaos Communications Congress this past December, there was a talk by Olia Lialina called The Only Thing We Know About Cyberspace is that it’s 640×480. it’s a wonderful walk back through the Geocities pages of yore, which Archive Team was one of the contributing groups to get the data.

It’s a great talk. You should watch it.

Fundamentally, there’s a theme in Olia’s speech (and the speech of others in that space, like Dragan Espenschied, Ben Fino-Radin, and so on) bemoaning the move away from a space on a website being the province of the users, and being turned into a homogenized, commodified breeder farm of similar-looking websites with only surface implementations, like WordPress, Facebook Pages, and so on.

Well, if you’re wondering how the good goddamn that happened, look no further than what I’ve just been talking about with the Web Audio standards and the endlessly shifting goalposts of the browsers.

There was a time when a person who was not particularly technical, or whose technical acumen was sufficient to get applications running on a machine and not much more, could code a webpage. The tags were pretty straightforward, the uses of them clear, and the behavior pretty dependable. Much how one could, in a weekend, learn sufficiently how to pilot a sailboat… such was that a few weekends of study could allow a person to craft a fun little webpage, with their voice, their stamp, and the idiosyncrasies of their personality shining through.

Those days are gone. Long gone.

Instead, we have this (as my buddy Ted Nelson calls it) nightmare honky-tonk of interloping, shifting standards soirees that ensure, step by step, bylaw by beta, that anybody who isn’t willing to go full native will be shut out forever. The Web’s underpinnings, at least on the basic HTML level, have been given over to the wonks and the engineers, making it an impenetrable layer of abstraction, not worth your time to learn unless you were looking to buff up your resume, or if some programmer pride resided in this whole mess being in your job description.

As I’ve said before, the idea behind JSMESS is meant to be greater than Browsers, Javascript and the MESS emulator. I use all three of these components because they reach the widest and have the greatest chance of survival on all sides. If one fails the project, it’s out.

In the world of Browsers, right now, there’s a group (Emscripten) which is making the porting of C++ programs (like MESS) pretty simple to compile. They’ve gotten so good at it in the last few years, but we should never lose track of the original fact, which is that the entire project is botard insane and the result of an insane and terrible situation. I appreciate everyone’s efforts, but that fact remains.

What we’re left with, and this is the god-awful truth, is a world where people have either gone full immersion into this bizarro world and can talk, with a straight face, about coding Ruby or Python for The Web, and people who have long ago given up understanding it and who just slam the side of their computers like a TV that isn’t getting the local news station properly. And both sides have contempt for each other, and it didn’t have to be this way, but here we are.

My hope is that the Web Audio API will get the overhaul from the new standards that it needs, Emscripten will reflect these sooner rather than later, we will tell the shut-out masses to “upgrade your browsers to the newest version”, and we’ll all just have nice little games and programs in our web pages and we can forget how much of the landscape of Brazil is lurking underneath our browsers’ clean, crisp buttons.

16 Comments

The MS-DOS Flood (And the High Flight of V2) — January 7, 2015

A couple days later from my entry above.

This has already significantly dwarfed the Internet Arcade. Thousands of people are hitting the site a minute, playing MS-DOS programs, writing reviews or keymappings or just freaking out about seeing this whole collection available and playable.

The choice to send everyone to the Version 2 beta interface has also been nice – lots of useful, interesting feedback, good comments, and people really working out the rough spots. The system got a fix-up during this and runs notably faster. The whole Internet Archive infrastructure is getting the workout it deserves.

A long time ago, I wrote about Opinion Spectrum Collapse Disorder, where if enough online attention is aimed at a thing, whatever that thing may be, the resulting flood of commentary will quickly expand out, like a gas, into every conceivable tangent, subject or opinion, until the whole thing is kind of meaningless for “this is what is being talked about”. Celebrity gossip stories seem to attract this particularly: talk about a celebrity buying coffee at a supermarket when they are anti-coffee, and the comments will quickly blow out into the celebrity, the notion of celebrity, something the celebrity said or did 10 years ago, coffee, supermarkets, Ayn Rand, and cat ownership. It’s just the way things are, now. I don’t know if there’s a vaccine for that. OH NO I SAID VACCINE

So yes, people are coming to the Archive in droves to get their MS-DOS on. I think the number is either over a million already or will be soon, and it may be more. The Archive (very admirably) does not log users in any meaningful way, so it’s not like I’m going to be able to run Webalizer on the stats directory. I suspect that, like the arcade, we’ll be into multiple millions before the week is over. Enjoy.

And I think the feedback from the Beta/Version 2 interface has been a spectacular boon – it’s a direction the archive is going in, and the massive stacks of information it maintains will be that easier to go through. The journey to the upgrade isn’t over – but it’s bringing a lot of people into contact with the Archive’s information that might have missed out on it before. The bounty was always there, and folks inside and out are happy to enjoy it.

The rest of this entry is me just answering a bunch of questions for people and reporters, because both camps just kind of make stuff up when they’re not confronted with actual facts. The answering will be in the form of an essay, not a FAQ.

Actually, speaking of a FAQ, here is the Internet Archive FAQ for the MS-DOS emulator. That answers a lot of the technical questions, or what seems to slow people up. So if you’re just here because SPACE BLORTS isn’t working for you, I think that’ll help. Otherwise, let’s just throw out some thoughts I have and some information people seem to want.

It is adorable that press and some bloggers have turned the narrative into “Jason Scott did this”. I am a top-hatted loudmouth for a very, very large amount of people who have poured decades of human years into what you see when you start the emulator and it goes “boop”. This includes the dozens of employees of the Internet Archive who keep things running, the developers at the Archive who have made the emulator work in the infrastructure, the developers and volunteers working on the JSMESS and EM-DOSBOX emulators and their porting to Javascript, the Emscripten developers (many) who have made the system work incredibly in the last couple of years, and the emulator authors (now we’re up into the hundreds) who have slaved, tirelessly and with scant recognition, for coming up on two decades to save the software by saving the running of it in simulated/emulated hardware.

I have a strong presence and personality, and a very strong will, so keeping my eyes on the focused goal over the past three years has been something of a contribution, but I’ve written a minuscule amount of code, mostly in the form of scripts. I’ve watched people max their brains out trying to make a system do something it was never designed to do, and watched others figure out how to wend deep into code to make completely indifferent and goliath codebases interact.

I am fearful to write out a list of names right now and miss people, but maybe sometime soon I’ll do an entry where I just talk about all the people and the worlds that have collided to do all this. It’ll be quite long.

Some quick technical things. This weblog is absolutely packed with entries describing the JSMESS system that the vast majority of systems on the Internet Archive use to emulate consoles, computers and arcade games. Just use the search box and look for “JSMESS”; you’ll be rewarded with the years of entries on the subject, including pleas, triumphs and tribulations. I don’t feel the need to repeat that here – if you want it, it’s there.

What I do want to say is a few words about the differences between JSMESS and EM-DOSBOX, which is what the MS-DOS systems work on.

I was not happy about adding an exception-based emulation system into the mix, but JSMESS isn’t quite ready for MS-DOS yet, and I think it’s a ways away from it. When it works, it’ll work really nice, and maybe we’ll even switch over to it, but for now, it started to become a real problem in the timeline of the Archive’s software collection that we were emulating the ZX Spectrum and the Atari 800 and Apple II, but not the massive juggernaut that has been decades of MS-DOS compatible programs. So I made that decision and a bunch of us put our faces to the grindstone and that’s what got announced – EM-DOSBOX.

A random slashdot anonymous poster claims they did critical work before the implementation of EM-DOSBOX, but the EM-DOSBOX Repository appears to be a straight fork from DOSBOX, so jury’s out, but I didn’t have any direct involvement in DOSBOX or EM-DOSBOX, but I’m fairly sure the lineage is direct (although there have been other javascript ports of DOSBOX in the past). If so, step forward and take your glory, soldier.

EM-DOSBOX is a very fast, very slick porting of DOSBOX, using the Emscripten compiler to bring the code into the browser. It implements some tricks which even JSMESS can’t do quite yet, and which might lead to speedups in the future along that line. I can run some EM-DOSBOX emulations at full-speed on my phone, which is sufficient for me to throw up my hands and say the future’s here.

The biggest difference between EM-DOSBOX and JSMESS is that JSMESS wants disk images, gathered magnetic copies of the original booting medium. (An example on the Apple II would be SPACESHIPS.DSK, where this is a “Spaceships” Apple II floppy portrayed as sectors that JSMESS reads like a disk). EM-DOSBOX is quite happy with a .ZIP file of all the files needed to run. It’s rather convenient because the vast majority of MS-DOS materials are sitting in .ZIP files (or .ISO files if they’re CD-ROMs, but we’re not doing those at the moment). The only downside to this is that you have to tell EM-DOSBOX what the “starter file” is in a collection. That’s something that can’t easily be automated, so I have to go into each .zip and do a best guess of what probably makes it go. AUTOEXEC.BAT’s a good guess, as well as PROGRAM.EXE where PROGRAM is the name of the program being run. But I get caught out all the time, and with thousands of programs, how the hell did I deal with that?

The answer was the Screen Shotgun, the automatic player-and-screenshotter script pile I’ve spent a while putting together over the last year. I’ve written lots of entries about THAT process as well, stalwart inquisitors – just search for “shotgun” in the weblog search box.

These days, I have a version of the script that can start up on an Amazon EC2 instance and I can run any amount of systems at 14 cents an hour to generate screenshots as fast as possible. These emulator-playing robots serve multiple functions, but the big ones are that they can take a bunch of screenshots of the programs and remove the doubles, and they can serve as a basic Quality Assurance team, because if the game doesn’t work, the screenshots are going to show it.

Hey, look, 2000 MS-DOS Screenshots taken by robots. That’ll hold you for a while.

The most annoying difference this time was that the MS-DOS programs switch up their resolution, with VGA vs CGA and so on, and there’s not a locked specific “size” of the canvas. A few kind folks on twitter took me under their wing, with various solutions, and I chose one that was very, very flexible, removing the need to hand-type-in resolutions going forward for future machines. The power of the crowd, they say.

A number of people to pop in among both this and the other major software release events have said something along the line of “what’s the point”, or “this is a terrible thing”, and like anyone working on a project, it’s kind of a waste to engage people who are stopping by with that position. Obviously, if I spent years on it, I thought it was something worth doing. I’ve tried to articulate why on multiple occasions. It either sticks or it doesn’t.

But the fundamentals are pretty straightforward:

Software is history. It is culture, it is process, it is a signpost.
Software also has a ridiculous half-life. A year or two, with few exceptions.
Software, unlike many other media, has a massive barrier to using it.
This barrier does nothing but grow over time. Over enough time, insurmountable.
Coming up with solutions like in-browser emulators promises to lower barriers.
If people can’t see or interact with software, it disappears. Utterly.
This project is an attempt to lower that barrier, ideally to nothing.

After these statements comes a lot of endless debate and discussion, and I’d rather be sitting in some room with a microphone arguing with someone than dealing with e-mails and weblog entries, but I think, much like my work with Archive Team, that once the core subject being debated is deleted or gone (or effectively gone), the conversation essentially ends.

What this army of volunteers and collaborators have done with javascript emulators is we’ve stopped the conversation from ending. We’ve essentially jammed the door open. It’s here now – we’re in a world where computer and software history runs in browsers, and that’s that. It’s refinement and iteration here on out. If my time or priority takes me away, the work will continue. That’s done, that’s a fact. Nothing will take that away, now.

Going forward, I expect debate in places where this effort collides with commercial interests, but I can tell you one thing I’ve learned, maybe something intuitive to people who work in the space of preservation or curation, but still worth mentioning:

In the realm of historical objects that have started their cultural/contextual journey as products, there are a specific subset that have gotten the attention, effort, quality preservation and presentation befitting a worthy project to do so. But that’s it. The vast, vast, vast majority of commercial products, including hardware and software, sink without a trace, or end up in attics or collections with only glancing references made. Maybe a completist providing a text listing of “what there was”, or a wrap-in of a few more-obscure B-list examples, but then it’s this giant, yawning abyss.

I figured, in the example of Arcade Games, that The Nerds would have long ago gutted, garrotted and filleted the history of all arcade machines enough to tell us who worked on it, what it consists of in terms of gameplay, or the usual love one sees showered on a Pac-Man or a Mario. No. No, not at all – there were (and are) games for which we have the ability to emulate a ROM dump someone did as long as 20 years ago, and a photo of a machine sitting neglected in a warehouse. Nothing else. And this is arcade games, where one could argue there was a perfect storm of subject interest, intensity of fanbase, and overlap with the most curatorial personalities in terms of maintaining information. Nope. Sometimes the games exist as a single found circuit board and everything else was imparted from that for the purpose of an emulator.

Now extend that to software, and it’s much, much worse. Now extend it to non-gaming software, especially custom software, and it’s even worse than that.

While people wring their hands over the preservation of Doom, a hundred thousand programs are at risk of being locked away only to people willing to burn hours to spellcast them into function in private geek caves.

This is a real and functional problem, and it has real consequences. This set of projects is, at some level, a strike back against that. It’s worth the doubt and the derision.

Finally, the little prayer I always say.

I’m very lucky – I’m working for a wonderful non-profit library, a dream of a lifetime. It allows me to work on projects like this, to maintain the effort to keep the project focused, and to have a platform to make the project available. That’s due to the forward thinking of Brewster Kahle and my various managers at the Archive, that they let this maniac onboard to work on things. I never forget to be grateful.

Comments Off on The MS-DOS Flood (And the High Flight of V2)

The MS-DOS Showcase (And The Ascension of Version 2) — January 5, 2015

2,400 MS-DOS games and programs have gone live on the Internet Archive. The full collection is here and the games-specific section (basically all of them, for now) is here. But wait before you click on those two links.

That’s the headline, and based on the thousands of plays that this collection has already gotten, a pretty well-known thing if you were looking for it. Considering the Internet Arcade has dragged in over 5 million people, this new collection will probably bring in a flood of its own. Welcome.

But.

Let’s talk about what makes this different than the other playable software libraries I’ve been putting up.

First, I really worked hard to have only fully-functioning programs up, or at least, programs that gave viable, useful feedback. Some of them will still fall over and die, and many of them might be weird to play in a browser window, and of course you can’t really save things off for later, and that will limit things too. But on the whole, you will experience some analogue of the MS-DOS program, in your browser, instantly.

Next, this is the first time I’ve successfully bridged metadata with Mobygames, the game information database which has been running for many years and recently returned to private management. My scripts yank in metadata from the entries, and place them in a relatively sane way into the descriptions. It’s not perfect, but it’s pretty good. Best of all, if someone overhauls the entry on the Mobygames site, I can see about pulling in the improved data into the Internet Archive item over again. So it’s still a one-way pipe, but it’s a fast one-way pipe, and the games make much more contextual and informational sense now.

Anyway, here’s the next big deal – I want you to start really getting into the Version 2 interface of the Internet Archive. So, go here:

https://archive.org/details/softwarelibrary_msdos_games/v2

This is a way to go directly to the Version 2 of the Internet Archive’s interface. In the upper right, there’s an exit beta button, so you can always pull the rip cord.

The Archive introduced v2, or “the Beta Interface” late last year. It was slow, stocky, and freaked people out. But folks got the idea, mostly – it was taking a site that had only incremental changes for 13 years, shaking the whole story up, and re-imagining the whole thing as a visual and browsing collection, as well as a way to dig deep into the materials.

Since last year, it’s gotten faster, slimmer, has added a bunch of features, and continues to become better to use. But it needs feedback, which is why I’m pushing you at it.

Enjoy the games, and check out all those beautiful screenshots! Play a few programs, note how you get around to things, and talk about what works and what doesn’t work for you. There’s a feedback button – use it. The goal is that you will be able to do everything you can do with the old interface with the new, but that you’ll have so much more happening on the new one. And remember, v2 works across all of the Archive… all the collections are out there, be they movies, books or audio, and the new interface has cool ways to interact with them as well.

As for the software library itself… many titles will stick around – some will go, or change, or be replaced. This is still brand new territory, and with the addition of the MS-DOS programs so far, this no-plugin in-browser experience is one of a kind in the world. I’m working just as hard as the V2 devs are to make everything work, and it wouldn’t be worth my time if I didn’t have people trying it out, commenting on it, and making use of it.

So please, let’s start 2015 right – in CGA and with a brand new way to use the Internet Archive.

11 Comments

Each New Boot a Miracle — December 23, 2014

This week, we got the emulation of MS-DOS programs (mostly) working on the Internet Archive.

Up to now, every single emulated system has been via the JSMESS project, which supports well over 600 systems and which does a hell of a bang-up job on it. The fact that one system (MESS) emulates everything from an Atari 2600 to an Apple II to a PDP-1 and so on… that’s pretty amazing stuff. And when we had well over a dozen people intensely involved in porting MESS to Javascript, it made sense to use this one emulator for everything, since once we got the work done, it just worked, for many many systems. It was good, economical effort.

There are, of course, many other emulators out there, some of them little hacks meant for one machine and some which handled multiple systems. Some of them, over the years, simply got absorbed into MAME/MESS, and others died out.

The MESS developers have biases and preferences, like anyone else, so there are systems that have gotten a ton of attention, and systems which have gotten some rough scaffolding, some proof of concept, and then little else. This isn’t to say they’re slacking off, of course – it’s just that there’s a lot of work to do in a lot of directions and they move along at the speed they’re comfortable with. Also, they’re more focused on getting it accurate and hardware compliant than fast and easy. This is, absolutely, the best long-term solution – just hacking things to “sort of work” is the province of forgotten half-functional messes that don’t function with an OS upgrade or an app change.

Also, there are cases where a system gets a burst of love from a developer and it just sings. Recently, the Apple II driver within MESS (and therefore JSMESS) got a massive boost in cleanup, rewrite, and expansion – and the result was a fast-moving (nearly 10 times increase in speed) wonderbomb that can handle graphic modes and settings that previously dropped everything. MESS continues to be the best choice in emulation, long term.

However, projects that focus, intensely, on one platform for emulation will do amazing work in support and accuracy. They’re not 100% flexible and they certainly take shortcuts here and there, but they work really well for what they are.

One such project is DOSBOX, the DOS emulation system that has been around for over 10 years, and which does one thing, the x86 architecture and specifically running of DOS-compatible programs, pretty well. It has different goals and approaches than MAME/MESS, but it a very respectable and well-run project, and we’ve all benefited from their work. Except for the fact it only does one platform instead of 600, it would behoove the JSMESS team to attempt a port of it.

Luckily, someone already did that.

Dreamlayers ported DOSBOX via Emscripten into a browser-functional emulator. He did it all by himself, and he did it very well, all things considered. His name for it is em-dosbox.

I’m just going to lay it out and say that Dreamlayers is a software engineering genius, one of those people with a gift for coding and making things work not just better, but understanding what things have to be left tied down and waiting for later improvements. Most of his em-dosbox notes are where Emscripten falls down as a compiling and conversion platform, with indications of how they can be improved. And buried in the code of his is an alien artifact that makes the generated javascript from the process run extremely fast. It’s called extractfun.py and let’s not go too deep into it, other than to say that if we ever figure out how to plug it into JSMESS, fireworks will result.

Now, em-dosbox is not perfect – far from it. It can freeze a browser, and it runs into problems of all sorts when running some of the oddball machinery that the DOS world produced between 1980 and (roughly) 2003.

But when it works, man, it just works. Clear as a bell, right as rain, a DOS window boots into a browser. If you go fullscreen, the mouse even works. (It works in not-fullscreen, but then you see both the desktop and the emulator cursors and your brain is sad.)

It’ll be a little while before the collection is really public, as there’s now the process of throwing thousands of DOS programs into the system, verifying they boot, making screenshots, and all the rest of that delightful curation the world hopes is waiting for them.

But it’s happening. It’s really happening.

And now more fundamentally, the personal take, and why this three year journey has been so worth it.

MS-DOS was always my main thing; I coveted the Apple II world, really got into the Amiga when it came out, danced among Ataris as I could, but for the vast majority of my “work” or whatever a teenager’s activities count as, I did it on MS-DOS.

MS-DOS was what my primary computing machine was from 1983 to 1988 – I hacked on the exact same machine, with minor improvements, all that time. I grew used to the clack of the keyboard, the arrangement of the fonts, and I never really used a windowing system on it for any time in high school. I called BBSes and online services with the internal 2400 baud modem – all the files I’d downloaded for my textfile collection, and which ended up on TEXTFILES.COM, passed through that machine.

Work under any system for long enough and you buy into its processes, whether you want to or not. And I bought in big. The grey on black text, the directory structure, the FILENAME.EXT system, the all-around aesthetic. I feel DOS systems glorify text in a way that other computers didn’t, due to IBM’s long, long history at providing the cleanest, crispest fonts for typewriters and correspondence. I was an IBM kid, after all, and my father’s acquisitions of company-made computers had me fully involved in it for years and years.

So, MS-DOS is my thing, and getting this working was my thing.

A lot of people poured themselves into the original DOSBOX and EM-DOSBOX. But two people, Daniel Brooks (freelance) and Hank Bromley (Internet Archive Dev) worked very hard for a significant amount of time to jam the EM-DOSBOX system into the Internet Archive environment, which is much more difficult than one might expect. Imagine playing a chess game by keeping the board in your head and dropping coins with move descriptions written on them down a grate and you start to get it. A lot can jam up in the pipes, and the whole thing wasn’t designed for what I’ve been making it do. They worked for days on end, bouncing iterative changes, seeing the results, and at the end, it worked.

The functionality of the MS-DOS/EM-DOSBOX emulator within the Internet Archive environment has more caveats than a gym membership, to be sure – dropping to the prompt currently freezes it. You have to tell it what exact file is the “boot” file, the one that will be executed on startup. Too big a file, and it blows out your stack. It doesn’t write out to the emscripten filesystem, causing some to blow up because they can’t write to “disk”. And the sound… well, sound continues to be an issue until the nerds-that-be fix them in the browser world.

But…

When it works, it’s glorious. The speed is really clippy, the prompts blink like I remember, and as it plays CGA, EGA and VGA graphics out and hits the notes, I can sit back and enjoy that clean, crisp little world I spent so much of my childhood in.

And more than that…

The Arcade, while fun, is not a technologically advanced project, and the systems are beyond simple – they work or they don’t. They boot up, they play music, they take your quarters, they play a game. I knew people would love them, and they certainly provide entertainment, and they’re historically important and culturally relevant. (And the effort to accurately reflect them in the emulator is a Herculean effort, I rush to say.)

But they’re also very simple affairs, and designed to impart 3 to 10 minute experiences. You get Pac-Man in a few minutes – you understand nearly all to know about Defender in a day or two. Granted, the skills to keep past that 3-10 minute period are legion and deep, but that’s not what most people would experience anyway.

But with a computer program – especially one designed to be sold for a good amount of money, or which represents a lot of work by someone, there’s a depth in there, and an intensity of the product, that can’t be ignored. A strategy game that doesn’t care how long you take to enter the next move, or a simulator that carefully gives you dozens of knobs to twiddle or points to tweak… that’s the harbinger of the computer experience. Not to mention the applications, those tools of productivity that have permeated computers since the beginning. It’s rich and lovely, and it’s my favorite thing.

Getting the Apple II and ZX Spectrum and Atari 800 collections running at the Archive have been a pleasure, and the resulting screenshots are beauty to behold. But MS-DOS… that’s my neighborhood.

The neighborhood looks good.

The system uses a .ZIP file that contains all the files that will be used inside the emulator as the “drive”. So if you have 5 files, including some batch file and an executable, accompanied by some data files, those will all be loaded in at the emulator’s boot. It will then require a designated “start” file, one of the collection, that it will attempt to run and execute against.

Sometimes this doesn’t work. There’s a lot going on there and my sources are variant.

But sometimes, it just works.

And when it works, I see smooth as glass graphics, beautiful pixel art, and those beautiful words, and the whole thing is alive again, in the emulator’s box, infinitely reproducible, instantly referenced, ultimately preserved in a pretty useful fashion.

Folks who are the sort of folks who professionally quibble will professionally quibble, but the now-years of work I’ve been involved in this (and others have been there right with me, by the way) are utterly justified every time I run the ingestor and push a .zip file of a gathered-away program online. Within seconds, it’s a single click away from beeping, booping and playing music, or standing stoically waiting for my input, unaware of the science-fiction universe it now lives in, destined to never really die, and to stand and be counted by history.

It’s like watching a miracle with every boot.

It’s a great feeling. It’s the best I get.

7 Comments

The Tourist, The Researcher, The Maniac — December 3, 2014

This year was primarily about software, specifically collecting and emulating it.

That went well.

Asked what my goals or projects are for work for the next year (2015), I have resolved that it will be cleaning up and making it easier to go through the mountain of material in the Internet Archive’s stacks.

Naturally, I’ll be continuing to help with Archive Team, improving JSMESS, and acquiring tons of crap to throw into the hard drives as well, but I really want to take advantage of the tools and time to really dig deep into the Internet Archive’s wonder as a collection. It is a huge, huge set of materials and I’m positive people would have minds blown if they really came into contact with the true size and breadth of the material.

So that’s how it’s going to go in 2015.

But, as usual, I can’t wait and I’m starting now.

First, understanding the mass of material already in the Internet Archive that has come flooding in in the past few years is a task in itself. We don’t have a few monks chanting – we have 4,700 chants by a group of monks in France. Or maybe you want your sermons in english. Well, that’s fine, here’s a set of 21,000 of them. Or religion bums you out – that’s OK, take your mind outside and look at 8,000 books from the library of the New York Botanical Garden. Or go back in time to see a hundred episodes of Starcade, the game show for video gaming.

I can do this all day. There are thousands of groupings of thousands of items down there. It goes in many directions, and it goes deep as your tolerance can go. Books? The Internet Archive’s contributors added 1,000 books yesterday, and 1,000 the day before that as well. (It’s been a good couple of months.) Audio, don’t get me started. Newspapers, tens of thousands of issues of newspapers!

Stuff. We have it.

Before I talk further, let me also say I’m not the only one thinking about this, especially within the organization. There is a very intense effort underway to completely redo the Internet Archive’s interface. If you visit http://archive.org/v2 you will go into the Beta Interface, which you can see is coming along swimmingly. (There’s a button in the upper right to turn it off and return to the current interface.) A team of people spent a year dealing with both the backend and the frontend aspects of this new design. It has a way to go, but it actually brings this whole issue to the forefront, since it’s going to become a lot easier to browse the collection.

But for my little bit, I want to find ways to get a better hold on the millions of items, move aside spam and broken items, and group up really cool and similar things without me spending every waking hour doing it.

But before I even do that, it’s time to talk about who I think the audience for the Internet Archive’s media riches are in the first place.

Hence, the Tourist, the Researcher, the Maniac.

The Tourist washes up on land, pushed in by the general tools of contemporary discovery – popular links and search result. The Tourist can be very compelling, because whatever dragged a Tourist to your site probably dragged a lot of other Tourists. They can almost seem like a coherent movement, or a congregation. But they’re not, any more than one or two methods out of a building results in an actual orderly line. They’re a mob, a gawking fog of incoherent attendance, a group who has come in because there was a blinking sign somewhere, and having found something similar to what they want, have smiled and gone away, or they’ve stuck around just enough to pass judgement on somewhere they’ve never been and will likely never return again.

You can tell a Tourist because they go back to comment at the place they heard about the link, and the commentary they make tends to be along the lines of “I have a lot to say about the way about how nothing impresses me.”

Tourists can definitely become recurring users, of course. But their introduction to the place will almost always be “here was a thing you might like”, and they run in and out, because these are busy people, and there’s a lot of things to like.

All is fine and good with Tourists, unless your site or the site attracting Tourists thinks Tourists are the best people to acquire, what with their bloating numbers and all. They’re not – they’re basically sugar. Quick high, bad aftereffects, no long-term benefit.

The Researcher has come into your life and your site because they’re Looking For Something. Hopefully you have it.

Their requests are usually somewhat strange, but they’re also really flexible about what they find. They browse, they look up similar items, and they try to do some of your sorting job for you. In a good day, they send some of the results of their work over – “I have a list of items that need to go into a different collection” is a common one Internet Archive gets in the inbox.

Researchers often want something that is not quite what search engines can provide. A query can be something like “I want all the magazines that reviewed the Apple III”, or “I want to look at movies that feature skateboards or skateboard parks.” You can feel like everything is an incomplete patchwork of metadata and fairy dust after a Researcher walks around. But they’re not trying to be mean or disruptive – they just really appreciate the place and want to get the most out of it.

Researchers hunt in solitary or in small packs.

The Maniac sounds like who you’re doing this all for.

You are not.

The Maniac comes in, and never leaves.

The Maniac has a dream inside their head of how the world should be and you are failing at it, right now, even while you’re not doing anything.

The Maniac has lots of ideas for the site, how things should be done, but they’re really a bundle of unfunded mandates and issues that affect a very, very small number of people. As a bonus, they become anti-ambassadors, posting frequently and viciously in other realms about how your place wouldn’t suck if you only decided to use the sdfjkhsfsdjfh standard or you would just allow unicode in usernames. I’d say The Maniac hasn’t done anything, but they probably have done something and by Ramthar’s Crowbar you will hear about it, a lot, as explanation of why they get to come in and take a dump in the potbelly stove.

DPI matters a lot to the Maniac, as does impeccable presentation and completeness. These, in themselves, are excellent things to have, if you can get them. But Maniacs are very, very quick to flip out if any of an internal checklist isn’t met. Maniacs flip out a lot. It’s what they do.

Maniacs are almost always solitary, although they can give the impression they travel in packs – but to be honest, every Maniac thinks all the rest of the Maniacs are tourists.

So, there’s a fourth group, called The Future – but going too insanely over the top to appease people that don’t exist with goals and ideals you can’t possibly anticipate is not today’s exercise. So you keep out there in the hazy realms of Tomorrow, you crazy Futures.

I think the sweet spot are the Researchers. You want to push Tourists to consider becoming Researchers if the mood takes them, and you want Maniacs to either tamp down into becoming Researchers, or go spill their hot oil of hate out from another castle.

Make the site make sense, start culling out items that are spam into places that won’t discourage browsing, and help build collections that sing. That’s 2015 for me – researchers welcome, Tourists a delight but sorry about the sdfjkhsfsdjfh being sub-par – I’m sure we’ll get to it soon.

Now, let’s get some sorting done.

3 Comments

Pain — December 2, 2014

Somewhere around the age of 26, I woke up and couldn’t really get out of bed.

The reason, in the immediate, was that any attempt to put weight on my legs sent shooting pains through them. Real, intense, shooting pain like there were rods lodged inside the legs and they sent out shocks when they detected any movement or effort. I have forgotten what I thought at the time, but I do remember how much it hurt.

The mattress was on the floor, so I was able to roll and crawl to a phone and call for help. After getting clothes on (somehow) and getting down three flights of stairs (somehow), I waited outside and was taken to the hospital. During this hour or two before I was picked up, I learned which positions hurt the most, which the least, and tricks for getting around, somewhat. I also stopped being shocked at how much it hurt and just winced or yelled an awful lot as I discovered a new angle, a new approach that would make it all hurt anew.

At the hospital, after some battery of tests, I was given a bunch of over the counter pain relief, some crutches, and no real idea of what was wrong. I do remember one young doctor saying “You know, this really acts like gout. But you wouldn’t have gout – you’re 26.”

I hobbled on crutches for a couple days, the pain went away, I lived on.

•

A couple years later I was arguing with my girlfriend while she was driving her car. I remember this because in the middle of yelling, I felt something very unpleasant and I told her to stop arguing because something felt very unpleasant.

The thing about pain is that you both remember and don’t remember it. I can, these many years later, describe it as a tightness, a strain deep inside my gut, like shoelaces of nerves had been pulled very intensely, and were now knotted in a bow, and my movement in most any direction was pulling at them. That’s what I remember made me go from thinking I’d had a bruise or a swallowed knick-knack in there – its that there was no logical sense to my ratio of pain to the motion. Moving and trying to get into a good painless angle did nothing, and then it would stop, and then it’d come back harder, and I was rapidly losing brain function. I asked and then really begged my girlfriend to take us back to her house.

On her couch, it just got worse, we didn’t know if I had a burst appendix, a knife I’d somehow fallen on, or if something inside me was just going necrotic and taking out my organs. But boy, did it hurt. It hurt until I ran out of ways to think it was hurt. I wanted desperately to take something to knock myself out but then I simply assumed that I’d die in my sleep and that seemed needlessly unpleasant and surreal. So I lay there, looking up at rafters, overcome with pain, waiting for either healing or death. Some measure of healing happened, and ultimately I fell asleep, as if I’d run a marathon.

•

Let’s move ahead to the diagnosis, that came in my mid-30s. Kidney stones.

There’s two kinds of kidney stones – the calcium ones and the acid ones. I have the acid ones, which are notably rarer and as a bonus, the treatments, which I had many, many, many meddling Dr. Google-Readers give me over the years, are entirely opposite. Drinking cranberry juice doesn’t work, for example – a lot of people told me to drink cranberry juice. No.

Due to a set of circumstances where I didn’t seek out the best doctor but just fell into doctors, I didn’t really have a doctor look at me, properly, with competence and awareness, until somewhat recently. This doctor has done wonders. So there’s your happy ending.

Now, let’s go back.

•

By my early 30s, I was passing 100 kidney stones a year. That is not an exotic allegorical statement – I was passing some sort of stone every few days. For months, and then years. On the whole, over time, I didn’t even notice them. In fact, I almost got worried when one didn’t show up in a given week. It was just another fact of life, like waking up feeling tired or occasionally having a joint feel sore. Stuff happened, and some of it involved the fact that I ceaselessly created stones and passed them frequently, with small pinpricks of pain involved in the process.

By the time this was all happening, pain was rapidly not becoming a problem for me. The kind of pain one might have from hitting your foot or shin against something, or from grabbing something sharp, I could notice, but it was more like hearing a distant yell, like someone cupping hands and shouting that you should turn back, the woods are not safe. It wasn’t immediate, and it certainly wan’t concerning. By this time, most pain was being given a level of interest far beyond what it probably should have been.

The reason for this is because of the times when it got my attention.

•

Imagine, if you can, a very large meatball inside a bowl that is only a little larger than said meatball. Imagine that meatball is kind of sloshing back and forth, kind of randomly, in the bowl. That’s how it felt, and how it started.

I’d feel some overarching dullness inside myself, and I started to get really good at recognizing it. The feeling, you see, was a largish stone, and one that had gotten in some way stuck. I got really, really good at noticing this, because I knew what was coming next.

If I was lucky, I could run in some direction and get my hands on water. Lots and lots of water, like quarts of it. I’m really fast at drinking water, and I’d stop mid-conversation and drink everyone’s water at the table, or run towards a store and buy a gallon of water and start drinking it at the cash register. Anything, you see, to get as much water flushing through my system as absolutely possible.

If I was lucky, and a few times I was lucky, the meatball would evaporate. It’d kind of roll into a cloud version of itself, and waves of discomfort would subside, like a pool minutes after someone jumps in it.

I wasn’t often that lucky.

•

I am a weaver of metaphors. I am a person who tries to find the middle ground between the listener and the moon, and concoct a pointing finger that will lead you to the moon. But where do you even begin with describing pain like a pain of a stone that’s lodged in your system?

First, it’s often in your gut, deep inside, so it doesn’t feel like the sharp sting of a skin burn or a radiating line of a strained muscle. It feels like something deep inside you is twisting apart, and has no recourse to stop, and probably will never stop. After a few minutes, you forget when the pain started and you definitely stop assuming it will ever end. Oh, you definitely hope it will end, but it never seems to.

Sometimes I was just two eyes staring at something in the room and pain. One time I was a flattened person on the floor of an airport, screaming, everyone looking at me, while a stranger held my hand. Another moment, I was in the softest bed ever, in a visit to my father’s home, with my siblings watching me appear to be dying for no good reason. They scrambled through medicine cabinets looking for something, anything, but ultimately all they could do was sit elsewhere in the house until the screaming stopped and I fell asleep.

This happened a lot. I’m not telling anyone who has lived with or spent a lot of time with me something new – they probably have at least one memory of me either having to leave very quickly to go be possessed elsewhere, or they’ve sat nearby, wondering what you do with a body when that happens, when your friend is gone. They know. Things people shouldn’t have to consider with guests, my occasional attacks gave them time to consider.

•

I only got operated on once, by a doctor I’ve called Doctor Coldfront since the time he operated on me. He had the bedside manner of a mailman to a mailbox. I’m sure he was competent, but ironically, he made me hurt the most, although with that pain came some interesting revelation.

He inserted a stent into my left Ureter, which is the tube from a kidney to the bladder, while I was under. I didn’t mind that – I was asleep.

Some time later, and I’ll spare you too many details, he removed the stent under local anaesthesia, as an outpatient procedure in his office. I had requested a full knockout but he wasn’t interested in doing that – I assume out of 10% concern about putting someone under and 90% of the inconvenience for what he perceived as a minor operation.

I had nightmares about that day for a decade.

•

But as I said, some revelation came from that – which was that there are doctors and professionals all throughout the world, some even here, who use some level of medical procedure to torture out information. Some of them are paid by governments, some by private organizations, to do so. Back in the hazy glance of youth’s cobbled knowledge and self-assuredness, I thought that a reasonably strong-willed person could resist giving whatever information was being extracted, under torture.

Considering how I was treated by a doctor who was merely inattentive to my wishes, I gained the knowledge and the understanding that no – information is probably not worth keeping from someone who is intentionally, unendingly, aggressively intent on you staying alert and bathed in pain for as long as is necessary for you to give up whatever pearl of knowledge they wish to extract.

That’s an interesting lesson to have gotten.

I never spoke to that doctor again. I hope someone broke his heart.

•

So, why go into all this? Well, two reasons.

First, my weblog has been suffering a little from a case of the Nerd, and has way too much technical information without much self-revelation and consideration. So there’s a little self-revelation.

Second, I wanted to make clear to a certain subset of fans and followers as to what my secret is, since I’ve had people literally say that they are trying to “Jason Scott” something, i.e. throw a lot of energy at it, or never stop.

My secret is that I am all too cognizant of how quickly life changes, and how one can go from a moment of relaxing in the sun to feeling pain so great, so intense and miserable, that you will cry tears of anger at your previous self for wasting precious minutes and hours in the paradise of existence you had been granted. It is why I don’t like lying down for too long unless I am very tired, and why I often (but not always) get up with a burst of intensity, worried about what I missed and what needs to be done.

It’s also why I tend not to wear coats, and underdress for the weather. It’s why I can hold my hand over a candle for a stupid amount of time. And it’s why I hand people things and am confused at the look on their faces, only to realize it’s because I handed them something blistering hot or bitterly cold.

It’s because pain’s kind of an abstraction now. When I go to the hospital these days, I have to exaggerate what pain I’m feeling, because I realized after a few times that they were asking if it hurt, and my definition of hurt and pain is not theirs anymore. I can see clearly, and I can think about things, and I call that not being hurt, even with a hole in my arm or a knife cut that needs stitches.

I’d call it a superpower, if I didn’t make an effort to remember how I got it, and how truly useless it is, and how dangerous. But I have it.

In the meantime, here I am, working very hard, doing a lot of things, and counting the missed opportunities and the upcoming chances to do even more things. Because of the pills I got from the right doctor, three years ago, I have not had a single kidney stone, or a single day in true screaming pain that defined my life for so long.

And if that’s what I have now, a day without that pain, then that is a day it would be a crime to waste, to not pull the most out of, to not work on things until I barely can stand up and then fall asleep until the next round of things to do.

I do not recommend the journey to this situation to anyone.

Have a great day.

Comments Off on Pain