D Tetris running on Webassembly

Posted 2020-08-10

Over the weekend, I took last week's Tetris source code and compiled it for the browser, doing minimal ports of required libraries to make that work.

Core D Development Statistics

In the community

Community announcements

See more at the announce forum.

WebAssembly in D

Playing with it

For now at least, I have it running live at: http://webassembly.arsdnet.net/

You can see there's only about 3 KB of Javascript bridge and about 40 KB of webassembly on the site running the little game.

The source code is on github: https://github.com/adamdruppe/webassembly

To compile the tetris source from last week, put it in that downloaded webassembly directory and run a command like this:

ldc2 -i=. -i=std -Iarsd-webassembly/ -L-allow-undefined -ofserver/omg.wasm -mtriple=wasm32-unknown-unknown-wasm tetris.d arsd-webassembly/object.d

If you want to use my web server, you can grab cgi.d from the full arsd lib and compile dmd serve.d cgi.d -version=embedded_httpd. As you can see from the serve.d source though, it is trivial and you can use any other server just as well. Just make sure the other files from that directory are served: you will need the html and javascript to actually run the thing.

What works

Tetris, of course! I also did a few other tests in the way, like class inheritance and even accessing dynamic objects:

NativeHandle body = eval!NativeHandle("return document.body");
body.methods.insertAdjacentHTML!void("beforeend", "<span>hello world</span>");
eval(`console.log($0)`, body.properties.innerHTML!string);

However, just about anything else will not work. I implemented the bare minimum of druntime and library modules to get my game to run. Tetris uses rectangles, so I implemented drawRectangle.. but it doesn't use circles, so drawCircle is not there. I needed new SimpleWindow to work, so I did minimal class support, but it doesn't actually free memory or support dynamic casts.

There's an import std.stdio; but you'll see it has exactly one function: just barely enough to compile tetris.d.

In my defense, I wrote it from scratch over the weekend, but going forward we can just port one function at a time, as needed, to expand the capabilities. Perhaps at some point it will cross the threshold to use some of the libraries directly too.

But as I often like to say, the language is easy; D can do it. The libraries are the hard part.

What sucks (for now)

The debugging experience is awful. I wanted the libraries to remain modularized in D, without needing separate javascript files to be available. I did this through the eval function - it sends a string over to javascript and it creates a function on that side you can call. The browser debugger thus often just shows the generated function. It sometimes showed the source anyway but wasn't consistent. I believe this can be improved with more work.

I don't know how well it performs. I imagine it is not great right now but it might not be awful: it passes slices of memory over to Javascript but each call over the bridge is still a decent amount of work. We'll see as time goes on.

D's current TypeInfo implementation is a huge pain. It is required for things it shouldn't be required for (druntime's generics essentially work through type erasure coupled with RTTI!) and it needs to be written out by hand. Andrei Alexandrescu is working on templating it at this very moment, which would make this a lot easier, but since it isn't merged yet, still have to add nonsense to object.d by hand today.

Lastly, HTML5's canvas is not the same as the Xlib and GDI functions simpledisplay assumes, so you might see some pixel artifacts in the tetris game. Blargh, but it is good enough.

How it works

Let's dive into the code. Again, you can grab it online at https://github.com/adamdruppe/webassembly and I will probably be pushing updates to that too if you are reading this after initial publication.

There's four main parts I want to look at: the Javascript side of the bridge (server/webassembly-core.js with a little bit in webassembly-skeleton.html), the D side of the bridge (arsd-webassembly/arsd/webassembly.d), the druntime implementation (arsd-webassembly/object.d), and the library (the rest in arsd-webassembly/).

The bridge

Compiling the code to webassembly was easy: ldc supports that out of the box. Just use the right mtriple argument in the build.

The hard part is webassembly is essentially a new operating system. This OS has a pretty extensive API, but there's no defined way to reach it from webassembly. All it provides are shared memory and basic function calls which can receive and pass numbers.

Thus, we need some bridge code on both sides, Javascript and D. In Javascript, you create an object that defines functions that D can see as extern(C) things. This goes the other way too: D can export functions Javascript can see, as long as it follows this numeric abi.

To write a minimum amount of Javascript, I made a handful of basic functions: memorySize and growMemory to help malloc, retain and release to reference count Javascript object handles in D, and one super-function called acquire that takes a slice of D's memory containing Javascript code, compiles it to a function on-demand, and then calls it with an array of arguments also found in D's memory.

If you look at the file, you will also see abort and _Unwind_Resume, which don't do anything yet but will help flesh out the druntime later.

Anyway, the D file arsd/webassembly.d provides the other side of this bridge. It has the definitions of the Javascript functions, then one big function, eval, and a NativeHandle struct to help use them.

NativeHandle takes an integer handle which an index into an array on the Javascript side, and refcounts using those retain and release functions. It also has two helper objects (alas, we cannot overload on @property, sigh) to make using these objects a little more convenient in D. You can ad-hoc define accessors and types which eval code in Javascript. The JS object in D is an opaque handle, but thanks to the eval function and opDispatch magic, it feels almost like you have access to the object directly. Almost.

So let's get into that eval function. You can request a particular return type and pass arguments. This function will massage the D arguments into an array Javascript can process, then send it over.

In Javascript, it slices the memory to get this data, converts it back into Javascript types, then creates the function and calls it.

The eval'd code looks something like this:

eval!int("return $0 + $1", 10, 20);

The arguments are set to $0, $1, and so on. The Javascript code passed to eval should NOT be constructed. In fact, I was tempted - and still am, this might change later - to make it a template argument to kinda force this on you, and perhaps even compile those Javascript functions ahead of time instead of on-demand, would also make it easier to reuse it later.

But anyway, just pass the arguments and it works. $0, etc, are just plain Javascript variables, no different than if it was a, b, c, etc. Since it is living inside a function, you must use the return keyword if you actually want a value to be returned.

In the code, you will also see __MODULE__ argument. That's there because the JS bridge sets a this object for module data.

eval("this.foo = $0", "this is stored");
eval!string("return this.foo"); // would be "this is stored"

I'm not sure I love this yet, but the idea is that D modules can define their own variables on the Javascript side for whatever purposes it needs without worrying about name conflicts in the JS global variable namespace.

With these pieces, I can start building up libraries. First one: the D runtime library.

Druntime

object.d in the repo has a bare-basics druntime implementation. You'll see it imports arsd.webassembly since it depends on that bridge. But otherwise, you'll also see there's not much to it: a simple bump-the-pointer malloc (the __heap_base symbol is created by ldc and tells me where I can start using the memory), a start symbol that calls the D main function, and then just a few functions to support the parts of the language tetris.d needed like a homemade memset, a class allocator, and just enough typeinfo so the compiler lets it pass. The definitions in there are required at this time - it is hardcoded into the compiler. But that is slowly changing upstream.

Just 135 lines at the time I write this - not a lot of code really needed for these basic functions, and we have the option to add more as needed. This is why I see no need to limit ourselves to -betterC!

Library code

tetris.d also imported std.stdio, std.random, and arsd.color|simpledisplay|simpleaudio. These needed implementations - minimally - to get the game to compile here.

Instead of modifying the original sources, I just made a new directory that has replacement modules and the compile command uses these instead of the original modules by changing the search path (see the next section for details).

Crack open stdio.d and see writeln just forwarding to eval... yes, it simply asks Javascript to append the text to the page. Similarly, my std.random just forwards to Javascript's Math.random.

arsd/simpleaudio.d is a do-nothing stub. In the future though, it could wrap the html5 audio api and actually work.

arsd/color.d doesn't require any operating system functionality, so it could perhaps try to use the original module, but since I implemented so little of druntime, the whole module there wouldn't work yet either. So I made a minimal version here too. The most notable one is toTempString, which the original color.d doesn't have... but perhaps should. The original has toCssString that would do the same job, but it allocates a new string. Here, knowing I didn't have a working GC, I didn't want to waste memory throwing it away on temporaries every draw command. So instead it writes out the string to a caller-provided reusable buffer which then forwards it to Javascript.

(And actually, I also could have just used opDispatch to return the color name as a string, since tetris.d only uses the named static colors and html can understand those just as well. But truthfully that simply didn't come to mind until right now lol! It's ok, this implementation is more flexible and precise anyway.)

Finally, arsd/simpledisplay.d was the real challenge. See, the real simpledisplay.d is quite a large chunk of code and is pretty intertwined with desktop operating system APIs. It is also in no way optional: the game is still playable without sound, but without the display, it is unusable.

simpledisplay is written with an eventual port to html5 in mind, in fact, for a while, I had the version(html5) stubs in there, though I deleted them a couple years ago since it felt like it would never actually happen. It even has flags in there to indicate only one window is supported, timers aren't supported, etc.

But actually porting it would take far too long to finish in one day. (I wrote the bridge on Saturday and the libraries on Sunday, in between doing other things, which is why they are so short.)

So instead of even trying, I just wrote a new module that defines the few symbols tetris actually needed. Then, using eval, I could implement them by forwarding calls back to Javascript. The HTML5 canvas api, used here, isn't exactly the same, thus the small visual bugs you may notice, but it is close enough.

simpledisplay does more than just display though, even here: it also has an event loop and can take input. This also needed for the game, so I couldn't just skip it.

For user input, I needed a callback into D from Javascript. Using extern(C), defining one isn't too hard, then I could eval a setup function to register an event listener in Javascript and forward it to the exported function. Then, the D function forwards it back to the handler delegate.

I cheated here: the implementation in there forwards to just one global delegate, whereas simpledisplay is supposed to have separate handlers for each window. I could probably fix this in the future by having a pointer to the window embedded in the closure. Heck, if I do that right, it shouldn't need the global at all, would be like passing a void* userData to a C callback!

The next major difference relates to blocking. See, in the real library, window.eventLoop blocks until the program is ready to exit. But in webassembly, I am not sure this is possible, at least not while keeping the direct(ish) JS access I'm using here. Perhaps instead I could pass the messages across threads and simulate blocking functions in D.

But as it is now, window.eventLoop actually just sets up the handlers then immediately returns! That's why the "0" displays on the web immediately. That's actually writeln(score), which in the original ran after the game ended, telling you your final score. But here, it runs immediately since eventLoop doesn't actually wait for the game to end to return.

This challenge would also affect most I/O functions, so it probably does need to be changed. Probably a web worker thread running the native code that message passes over the bridge and can afford to be suspended as needed.

Bringing it together

Let's look at this compile command in more detail:

ldc2 -i=. -i=std -Iarsd-webassembly/ -L-allow-undefined -ofserver/omg.wasm -mtriple=wasm32-unknown-unknown-wasm tetris.d arsd-webassembly/object.d

First, the -i and -I commands change the search path to my replacement modules and overrides the default behavior of skipping the std namespace when looking for custom modules. This allows it to use the custom library instead of the real thing.

Next -L-allow-undefined tells the linker to allow references to functions not present in the code itself. This is because the Javascript side of the bridge functions are unknown to the linker. On the downside, this defers ALL link errors to the browser's console instead of the build process, but on the upside, it lets the bridge work.

-of is just the output file, in my test rig, I called it omg.wasm because I'm silly like that.

-mtriple is how you tell ldc to target a different architecture/OS than the default. This actually generates webassembly code.

Then, finally, tetris.d is the name of the file containing main, and the last argument is also very important: you must specify my replacement object.d explicitly on the command line to opt into my replacement druntime.

Previous webassembly demos have used -betterC instead of object.d. You do NOT want to do that here since that would disable D classes! But since the full druntime port is not yet ready (Sebastiaan is still working on it btw, he has a test version working but still a ways to go), and even if it was, it is over 200 KB and here our whole game is only 40 KB!

I'm pretty tempted to make serve.d automatically compile modules on demand to make a generic little server that can run a variety of programs without even thinking about it. Just drop a D file in the directory and surf to it in your browser! I might do that later.