ASCII by Jason Scott

Jason Scott's Weblog

That Whole Thing With Sound in In-Browser Emulation —

We’re past 4 million verified MS-DOS players on the Archive this week.

So that’s something.

It’s time to really lay out the floating elephanty-thing related to this whole in-browser emulation kick – sound.

Sound on the in-browser emulators (be they JSMESS, EM-DOSBOX, or others) suffer from sound strangeness. In JSMESS, it works fine except when it doesn’t. In EM-DOSBOX, it works, but it does this kind of strange fluttery-ness that, if you’ve got experience with the originals, you notice right away.

Both are the results of a specific situation with browser audio, and the work that the Archive has been doing to bring this all to a mass audience is bringing the situation to the forefront. I just needed somewhere to send people who go ‘sound seems weird’, and so here we are.

tumblr_mdg63iqTL81qkmqyho1_400

One of the side effects of this project is that I had to spend years following the week-by-week advances and activities of the current lead Web Browsers in how they handled things like Javascript engine improvements and other features.

As might be expected, I’m not a fan.

The rest of this is insider baseball, but if you want to know why audio has an issue, here it is.

Currently, there’s four active large-scale browsers with any traction and attention. They are Chrome, Firefox, Internet Explorer, and Safari. There’s others, just like there are other land-line telephone companies and there’s other network hardware companies and there’s other search engines, but come on. As part of the testing procedures for JSMESS I took a sample of the other browsers that were hitting the Internet Archive and installed all those things. Many of them are Chrome and Firefox, recompiled with features added in or features ripped out as the tastes/anger of the maintainers dictate. This just adds more lint to the Big Four.

(Also, as regards Safari, I’m being really nice including them.)

With four browsers to consider, it then comes into play about how they will handle certain standards so that something that appears on one browser will appear some level of the same on the others. Examples include rendering of fonts, tables, adding buttons, allowing control of keyboard, and so on. On the whole, generally, the world has gotten by.

Here, at the beginning of 2015, the big drag is Internet Explorer. Forget the future and the present; the past has been rolling updates for browsers, so that Mozilla/Firefox and Google/Chrome are basically making major, wholesale changes to the operations of their browsers a few times a month, as it suits them. Safari sort of follows suit with much less frequent work, but the system is in place to allow them to do it when it pleases them. Meanwhile, Internet Explorer drags along, basically doing a major scary X.0 release every year… except 2014, when they didn’t do a single release.

Here’s where we start to run into the issues with stretched supply lines, a la warfare, except here the war is to bring runtime-quality items on webpages.

Firefox and Chrome both have in-house speed-up flav-o-paks for making their stuff run quicker. Firefox has ASM.JS, and Chrome has NaCL. ASM.JS will work in all the major browsers, and NaCL is kind of a sleight of hand trick. They both want the same thing – to cheat out certain programming actions and runtimes so that they run very, very fast in the browser. ASM.JS, since it’s basically Javascript, will run in the other browsers, but Firefox has gone ahead and written in-program routines for noticing ASM.JS is in place and then optimizes it further. Therefore, programs like JSMESS, that use Mozilla’s Emscripten compiler to make their run-times, run significantly faster in Firefox then Chrome. In an ideal world, the end-user doesn’t notice because once you get to a certain speed with, say, emulating DOS or a video game, you just have it. It’s done. But it’s not an ideal world.

So, essentially, JSMESS and EM-DOSBOX run pretty fast in Firefox, run a little slower but fast enough in Chrome, run notably slower in IE, and run OK in Safari. Tiny-share browsers using Firefox’s engine therefore get the speed boost, and the Chrome-based browsers get that amount of speed.

IE could release another version soon, and blow everyone away, and then another year will happen. For better or worse, they really should be on the same ridiculous twice-a-month schedule the other browsers are rocking.

Let’s step back.

00_coverscreenshot (11)

All of this back-room crapola is part and parcel of the entire software industry as a whole -there’s nothing special about it related to browsers. The only reason it even gets attention is because the nature of the web has, for the moment (a moment lasting 20 years) forced natural adversaries into cooperation, because initial attempts to be “THE” browser for the web by both Netscape and Microsoft fell on their face. In other words, because nobody could really “own” the way webpages were being made, all the browsers had to do work to make themselves at least superficially coherent when it came to standards.

Along the last 15 years, there’s been weird little deals and tricks employed to make one browser seem “better” than others. For example, IIS webservers (made by Microsoft), if they detected they were dealing with Internet Explorer, went into this kind-of-not-a-standard mode that basically wrecks TCP/IP but ensures really fast data transport. (In high-level terms, it stops waiting for acknowledgement and just blasts the data.) Google added some really persistent programs running in some operating systems (Google Crash Handler, among others) that were “helpers” that gave their browser some resilience and options. And now we have ASM.JS and Firefox’s both providing the new idea and the others not really wanting to endorse it in any meaningful way.

Most of this is very boring.

But let’s proceed, in this fogged landscape of standards and trickery, to sound.

smurf

So, you start up a game of Smurfs: Rescue in Gargamel’s Castle, and it either works for you, or it doesn’t. Generally, it will run at full speed in all browsers. Sound, however, is another situation entirely. When the game starts and the music begins, it either sounds clear, or it does not.

The reason for how it sounds is multi-fold.

First, when you listen to an audio track (or a video track with audio) or you play a javascript-native videogame, or anything that’s using audio in a webpage, in general these items have a set of pre-made, rendered, finished audio files in them. An audio track is obviously that – a multi-megabyte .wav or .mp3 file, and a video has an audio track that is same. A videogame will have a little pile of samples. All the audio routines in browsers assume this is how the world is, and what the world will be. And generally, that has been the case and all has been good.

JSMESS breaks that, as does EM-DOSBOX.

Instead, these items are generating chunks of data, live, and mixing them as they go. They’re not preformed samples being pushed out – every millisecond is brand new, and won’t repeat.

The current standards don’t like this situation at all.

Mozilla had cooked up an audio standard that sort of dealt with mixed audio, but that standard has been removed in favor of a shared standard called the Web Audio API. Here’s the current version of that standard. As you can see, Google and Mozilla have people working on it. Microsoft isn’t evident, but who knows. Apple is MIA as well.

I’ve been told that if this standard gets formalized and gets implemented, it will just “fix” the audio problems. It accounts for programs like JSMESS/EM-DOSBOX doing dynamically created audio. Emscripten will then observe this standard, and then compiling JSMESS will push it through and we’re in happy audio land.

I want nothing more than this. I have nothing I can contribute directly to the standards bodies other than to say it’s obvious millions will benefit.

So that, in a nutshell, is why audio currently has problems in the in-browser emulators.

But let’s go one step deeper.

geocities-1996

At the Chaos Communications Congress this past December, there was a talk by Olia Lialina called The Only Thing We Know About Cyberspace is that it’s 640×480. it’s a wonderful walk back through the Geocities pages of yore, which Archive Team was one of the contributing groups to get the data.

It’s a great talk. You should watch it.

Fundamentally, there’s a theme in Olia’s speech (and the speech of others in that space, like Dragan Espenschied, Ben Fino-Radin, and so on) bemoaning the move away from a space on a website being the province of the users, and being turned into a homogenized, commodified breeder farm of similar-looking websites with only surface implementations, like WordPress, Facebook Pages, and so on.

Well, if you’re wondering how the good goddamn that happened, look no further than what I’ve just been talking about with the Web Audio standards and the endlessly shifting goalposts of the browsers.

There was a time when a person who was not particularly technical, or whose technical acumen was sufficient to get applications running on a machine and not much more, could code a webpage. The tags were pretty straightforward, the uses of them clear, and the behavior pretty dependable. Much how one could, in a weekend, learn sufficiently how to pilot a sailboat… such was that a few weekends of study could allow a person to craft a fun little webpage, with their voice, their stamp, and the idiosyncrasies of their personality shining through.

Those days are gone. Long gone.

Instead, we have this (as my buddy Ted Nelson calls it) nightmare honky-tonk of interloping, shifting standards soirees that ensure, step by step, bylaw by beta, that anybody who isn’t willing to go full native will be shut out forever.  The Web’s underpinnings, at least on the basic HTML level, have been given over to the wonks and the engineers, making it an impenetrable layer of abstraction, not worth your time to learn unless you were looking to buff up your resume, or if some programmer pride resided in this whole mess being in your job description.

As I’ve said before, the idea behind JSMESS is meant to be greater than Browsers, Javascript and the MESS emulator. I use all three of these components because they reach the widest and have the greatest chance of survival on all sides. If one fails the project, it’s out.

In the world of Browsers, right now, there’s a group (Emscripten) which is making the porting of C++ programs (like MESS) pretty simple to compile. They’ve gotten so good at it in the last few years, but we should never lose track of the original fact, which is that the entire project is botard insane and the result of an insane and terrible situation. I appreciate everyone’s efforts, but that fact remains.

IMG_0218What we’re left with, and this is the god-awful truth, is a world where people have either gone full immersion into this bizarro world and can talk, with a straight face, about coding Ruby or Python for The Web, and people who have long ago given up understanding it and who just slam the side of their computers like a TV that isn’t getting the local news station properly. And both sides have contempt for each other, and it didn’t have to be this way, but here we are.

My hope is that the Web Audio API will get the overhaul from the new standards that it needs, Emscripten will reflect these sooner rather than later, we will tell the shut-out masses to “upgrade your browsers to the newest version”, and we’ll all just have nice little games and programs in our web pages and we can forget how much of the landscape of Brazil is lurking underneath our browsers’ clean, crisp buttons.


Categorised as: computer history | Internet Archive | jason his own self

Comments are disabled on this post


16 Comments

  1. Elu Sive says:

    There is a solution to these problems, Jason. It is called launch a native-like java window from browser, not inside the browser.
    Making things run inside the browser window has never been and never will be a good idea. I believe you have listed many of the reasons for this but there may be more.
    The major strenght of java is not that it lets you put things in browser windows but that it lets you launch apps on any platform supporting java, preferably from native-looking windows. If only software developers would understand this and end users everywhere in all sorts of corporations and at home would have a much better life.
    All of this being said I greatly appreciate and support your work. It is just this little detail I think you need to change 90 degrees.

    • Jason Scott says:

      In an ideal world, sir, yes. Yes, Java would be the solution, or a Java-like environment that simply existed outside the browser space, even being executable on desktops or off of servers.

      One problem.

      Java is fucking ruined. RUINED.

      One merely has to see how Oracle has used Java as a pawn to try and destroy Google, to see how the Java plugin itself, which you download from Sun/Oracle, is BY DEFAULT attempting to install the ask.com toolbar, and of course the way Sun never truly allowed Java to be an open standard… and then you realize that Java can’t be used, even if engineering-wise, it is the smarter solution.

      I wrote about this years ago at this point. Javascript makes the most sense for the moment merely because it comes “pre-installed” in all major browsers, is nominally an open standard, and it isn’t a series of weapons being used by deranged billionaires to extract blood out of rivals. It is a messy, foolish environment, bereft of the advantages that engineers can do with a clean slate, but it does work. If it didn’t work, we wouldn’t use it.

      In the future, I suspect some group will do it “right” and create a wonderful plugin or default wrap-in for all browsers that will simply function as the native environment all this material needs. At that point, MESS and DOSBOX will be ported to this, likely overnight, and the process will continue.

      Until then, I’m holding firm on this solution. It’s working.

      • The problem with low level solutions like asm.js and nacl is the lack of a standard higher level interop standard. .net and java have this, and both enable nearly seamless interop between different programming languages. However, as managed runtimes, they are slightly too high level for some applications.

        In my opinion, the right balance would be a low level intermediate code representation like nacl, asm.js, or something else, but with a standard high level interop mechanism. Something like WinRT on windows 8, which let’s c++, .net, and JavaScript code talk to each other without the pain of pinvoke or ffi.

        • Interesting thing about asm.js is that even though Chrome has no explicit support for it, it does compile to really fast code. The article makes it sound a lot slower than it really is. At one point V8 produced faster asm.js code than SpiderMonkey did (although that is different now). For something like DOS emulation you probably don’t care about that performance ceiling.

          > code representation like nacl, asm.js, or something else, but with a standard high level interop mechanism.

          asm.js *is* a standard because it’s strict subset of bog-standard js. It’s weird to use js as an IL, but hey, no matter how ugly it is, it works across all browsers. The problem with a new dedicated IL is that there would be spotty support and we would be in a worse position than we are now.

          http://xkcd.com/927/

    • Mark Entingh says:

      I disagree, Elu… running Java applets is as much of a poor excuse as running a Flash applet. Jason is right about the web being a complicated place where only passionate advocates of web design can create and engineer things, but making Java a solution to this problem is lazy and careless. Javascript has a very bright future, and it is the responsibility of Microsoft, Google, Mozilla, and Apple to create this future.

  2. Has anybody out there ever tried to run a Kickstarter to raise funds to create Open Source Software that does the same job as Java? If there was a free Java-clone out there, maybe you could try to do things the way that Elu suggested. You could maybe even modify it to make a special Wayback Machine custom version.

    Would a Java-like environment have allowed people to run things like this offline (instead of putting a load onto the Wayback Machine servers)? Would that be “better”? Or would it stop you getting information from beta testing? (I’m assuming the entire beta test phase is supposed to encourage people to identify things that are broken, so that bugs can be ironed out.)

    • Such things do already exist. For the compiler, Gnu has one, Eclipse IDE has one called ECJ built in. The runtime is also an issue, for which you have Dalvik and Android Runtime (ART), both free, and some other options such as Apache Harmony (retired now). But I agree with other commenters, Java is a poor solution for the browser. Remember also that plugins are being phased out of most browsers these days anyway, and Java won’t be available except through Javascript or NaCl witchery.

  3. John says:

    I really enjoyed this. A succinct explanation of how the Web is a total clusterfuck of mismatched “standards”, complete with concrete examples. As a web developer, bitching about that never gets old. Thanks for a good read.

  4. greggman says:

    MS already said they’re implementing the Web Audio API so other than that what issues are there? You didn’t really give any specifics or I missed them.

    • Jason Scott says:

      Oh, nothing. Remember all those other times Microsoft implemented a standard, once, following all the aspects of it, and not forcing everyone to have to make a special “for IE” shim in their code to deal with decided-upon idiosyncratic retardations? And then not update or change for a year or more?

      Welcome to the Web!

      • Jeff Walden says:

        Microsoft’s actually gotten pretty good these days about implementing standards correctly, more or less comprehensively, from the start, without the myriad quirks of days of yore. And, (setting aside the elephant in the room of their slower release cycle) about implementing new standards in short order, not just playing catchup on existing ones. But a bad rep is hard to change. (And I say that as a JS engine hacker at Mozilla, who’s been around in one capacity or another since 2003.)

        • aphid says:

          Aside from pushing their own RTC spec over webRTC. I guess we can at least appreciate that they’re pushing *a* standard even if it isn’t *the*.

          • Jeff Walden says:

            Interesting, I was unaware of their historical (the first hit I found suggested this changed end of October) non-participation in WebRTC. A bit like WebGL, to which they proposed no alternative and eventually caved, as well. (I was pretty surprised at how thoroughly they ignored 3D graphics in the browser til OpenGL had dominated the field.) Still, with respect to things mostly agreed upon by other browsers, they’re usually on board these days, with the other cases as exceptional.

            I should also say, the closer you get to bleeding edge, the more everyone tends to disagree at varying levels, push their own proposals, etc. To a large extent that’s life in a world where (happily) no one has unfettered monopoly power any more. When Microsoft disagrees in this particular stage of standardization, they’re no better or worse than anyone else.

  5. Phoebe says:

    > Has anybody out there ever tried to run a Kickstarter to raise funds to create Open Source Software that does the same job as Java?

    This already exists: Sun released Java under GPLv2 in 2006 (with praise from rms, even). I’m not sure whether Oracle is still releasing stuff that way, but I guess so given that Debian has an openjdk-8-jdk package in main. All you need from Kickstarter, then, is to have someone build proper Windows/Mac/Android/etc. packages without the shit.

    • Nathan Myers says:

      And of course there is Mono, implementing C#, which is pretty much the same thing except even less likely to be where you need it. But neither Java nor C# will ever run code as fast as asm.js on Mozilla and, eventually, Chrome and IE, and maybe someday even Safari (until somebody comes up with an asm.java AND Oracle and MS feel obliged to recognize it). Maybe you will miss Java’s “business objects”, but you get everything you need to implement a modern browser, which is quite a lot. And, you can code in a powerful, expressive language like C++ (or Rust, or what-have-you) without compromise, choosing on your own schedule and without hanging on every MS, Oracle, and Apple press release. That’s a lot like freedom. Eventually support for non asm.js features will wither and fall off, unnoticed.

  6. Lennie says:

    Microsoft said it’s new browser, codename ‘Spartan’, for Windows 10, will be an ‘evergreen browser’. Always the latest version: http://blogs.msdn.com/b/ie/archive/2015/01/22/project-spartan-and-the-windows-10-january-preview-build.aspx

    That is a bit of a game changer.

    Also notice asm.js is ‘Under consideration’ by the Microsoft team if you look on status.modern.ie.

    When 2 browsers have an implementation of a specification, then W3C will consider making it a standard. So this could all still happen in our lifetimes.