This Week in D

Browsers in D

Blog
Posted 2023-12-18

I made my own browsers in D: a mainline one using webviews (Microsoft's WebView2 on Windows, and Chromium Embedded Framework on Linux, with secondary webkit impls that need more work to go mainline), and a little homemade toy with 100% custom engine. More after the stats.

Core D Development Statistics

In the community

Community announcements

See more at the announce forum.

Browsers in D

Chrome skins

About two years ago, Firefox drove me sufficiently nuts with their constant changes and breakages, coupled with websites I cared about not working anymore in the locked version I had, that I decided to take the dive into making my own browser for daily use. Not exclusive use - I still have other browsers installed and actually still use them pretty regularly for certain tasks - but primary use.

I dusted off an old file, arsd.webview, which started out as a port to D of a C "webview" library, which was meant to be a thin wrapper around the OS libraries, but didn't do much useful to me. Almost all of that original ported code is obsolete now; it used GTK and older Microsoft webview, which weren't very useful to me due to lack of features and integrations into the rest of my system (in other words, I hate GTK, part of my problem with firefox is it using those absolutely awful gtk file dialog boxes, so if I can't easily replace those, I'm not going to go far with it).

I wrote my own..... almost totally unusable browser engine, 100% from scratch, back in about 2013, but pretty obviously that's not going to be useful as a daily driver replacement for Firefox, even if I did start devoting serious time to it. So chromium was the most practical option.

My main browser UI code is NOT public (though it may be at some point, I wrote it just for me anyway), but the minigui widget that makes its core is public, but also not especially stable, since it is pretty transparently just a helper for my own use case, and mostly not even documented (at the time I write this, WebViewWidget_CEF is documented, but the more generic WebViewWidget is not!), but that said, my use is at least proving it works... for me.

However, one goal of the module is to have a webview widget available for other minigui users, perhaps including cases where people want to do their main ui inside the webview, or something in the middle, just providing a little view inside another application (it might be easier to put out a html file with a <video> instead of using another video display library in some cases, for example), but I have not actually done any of that support yet.

That said, let's take a look at what you can do with it right now.

Using the minigui webview

The dub package name is arsd-official:minigui-webview, though, as always, I recommend you just git clone the arsd repo and use dmd -i so things just work without extra config.

On Linux, you also most download and set up the Chromium Embedded Framework library, a very specific version, since they do ABI breakage on a regular basis. Frankly, I don't think it is worth it for something you intend to distribute, but fine for something that is for your own personal use. (If you're going to do a web ui, maybe instead spin up a local web server and open the user's browser to it. I might make a subclass implementation that does this for you too. Also embedding a window of an existing browser using something like XEMBED might be doable, but Firefox isn't terribly compatible with this either, still, another thing maybe worth exploring at some point too.) The documentation of arsd.webview (the module with the bindings itself, separate from the minigui-webview wrapper) lists the compatible version.

On Windows, however, the situation is much better, because Microsoft maintains their ABI-compatible wrapper, and installs the component as part of Windows Update. Thus, you can run an old program against a new lib, or a new program against an old program (assuming you don't require any of the new features to run), giving you a much easier time as an application developer. All you have to do there is bundle a WebView2Loader.dll file with your exe (or you could static link it in theory, but my library dynamic loads the dll to issue a runtime error with download instructions if it is missing). This redistributable file comes from Microsoft and can also trigger a download of the main runtime if it has not already been installed on the user's computer.

It is perhaps possible in the future to make the library download WebView2Loader.dll from the internet when it is not already present, but right now, it is your responsibility to download it and bundle it with your exe. It is less than 150 KB, so not much of a burden for you to bundle, in stark contrast to the 1.4 GB of CEF on Linux! (That said, maybe I could make the library auto-download it when needed too someday, but it is still huge and not likely shared with anything, unlike on Windows where it is shared.)

The loader dll is found in this package: https://www.nuget.org/packages/Microsoft.Web.WebView2

It also includes the C bindings and IDL files, etc., but you don't need those since I embed them in the D file.

Anyway, once you grab the dll, you can try out the module:

module embedded_webview_test;

import arsd.minigui;
import arsd.minigui_addons.webview;
import arsd.webview; // also need the base module for the app, for now.

void main() {
        // this is required to initialize the underlying library
        // and should be created before doing other work in main,
        // then live the whole duration of main.
        Wv2App appConfig = Wv2App(null);

        // note that the library has separate Wv2App and CefApp
        // objects. I will make a common interface to them in a future
        // version.

        // then make your ordinary minigui window
        auto window = new Window("arsd webview demo");

        auto webview = new WebViewWidget(
                "https://dpldocs.info/", // starting url
                null, // delegate to open a new window
                BrowserSettings(), // some settings,
                window // the parent widget
        );

        window.loop();
}

Compile with dmd -i embedded_webview_test when the arsd libs are in your import path, and you can run the exe to see a little web window.

screenshot of the webpage in a basic window

If you use it for a while, you'll find a few things: a lot actually works! This is thanks to the shared browser engine. But you'll also find some things don't work: right click a link and try to open in new window. It will crash the program, because it will try to call that null delegate we passed when creating the widget. This is just a simple bug; I forgot a null check in the Windows version (passing null to that in the Linux version prohibits opening new windows entirely, which is what is supposed to happen). I've already fixed it on my computer, but it isn't pushed up to github yet and I'm deliberately not going to until later in the week to illustrate the point that this library is not ready for general use.

In addition to bugs and some outstanding platform differences, you'll notice most the documentation is not written, but more importantly for use cases other than my own (and in fact, even for my uses, it is good enough for a lot, but I want it to do more), you'll quickly find that the API is also a bit lacking.

For example, there is a function widget.executeJavascript you can use run some script in the context of the page. But, there is no way for the page to communicate back to the host at this time, aside, of course, from the HTTP facilities provided by javascript. WebView2 (Microsoft's API) provides methods to pass messages between the two environments, but CEF doesn't (it lets you add a function to the global JS object though, so it is possible) and I haven't wrapped either facility for my api.

Similarly, the constructor takes a URL, which is fine for loading some web content, but what if you wanted to load some resource embedded in your application? I have no methods for this yet.

What if you want to filter external URLs, and ensure the webview *only* uses your embedded content instead of opening up to the main web? No methods for this.

What if you want to inject custom script or other content into web content? You could use widget.executeJavascript in response to url changes (webview.url_changed = u => webview.executeJavascript(stuff) kinda does it), but there is a race condition between load complete and url changing, and there is a progress indicator... on the CEF version, that's not implemented in the WebView2 version, so this isn't actually going to work well.

Instead, we will want a method to inject some custom script into all pages on the library end. And I don't have this yet (again, WebView2 has one, but CEF doesn't... but it does let you load anonymous browser extensions, which is close enough but must be done at a certain part of the load, so it will have to be a constructor argument instead of a method. or something).

I also don't have methods to control media capture (they're just automatically denied) or sound muting (playing sound is automatically allowed), or a few other things like that.

But I think if I were to implement these things, it'd be fairly usable. Once you can inject script, you can use that to add other content and stylesheets to individual pages. If you can do message passing between the two environments, you can use that to implement all kinds of other apis.

What else would you want at that point? Well, I guess it'd be up to the user! But I suppose using a webview with some canned functionality message passing to the local D app might be able to find some value in those single-page app techniques or something. Maybe someday I'll do the rest of it, but it will probably be a little while; my own browser is a mid-priority project of mine, but other uses cases are low priority.

A pure-D, all custom browser engine

A long time ago, I think around 2013, I took my arsd.dom, arsd.simpledisplay, and a few other modules, to make an all-D browser. Over the weekend, I recompiled it to see if it still builds.... and it does!

And it still sucks. lol, see for yourself:

the same dpldocs website but barely usable

The code for this:

1 import arsd.htmlwidget;
2 
3 void main() {
4         auto window = new SimpleWindow(800, 800, "Browser", Resizeability.allowResizing);
5 
6         BrowsingContext browsingContext = new BrowsingContext();
7         _contextHack = browsingContext;
8         Document document = new Document();
9 
10         void recalculateLayout() {
11                 layout(document.root, window.width, window.height, 0, 0, true);
12                 auto p = window.draw;
13                 drawElement(p, document.mainBody, 0, 0);
14                 LayoutData.someRepaintRequired = false;
15         }
16 
17         window.windowResized = (int width, int height) {
18                 recalculateLayout();
19         };
20 
21         void ael() {
22                 addEventListener("mousedown", document.querySelectorAll("*"), (Element ele, Event event) {
23                         auto l = LayoutData.get(ele);
24 
25                         if(event.button == 3 && ele !is document.mainBody)
26                                 ele.parentNode.removeChild(ele);
27                         if(event.button == 1 && ele.tagName == "a" && ele.href.length) {
28                                 scrollTop = 0;
29                                 document = gotoSite(window, browsingContext, ele.href);
30                                 ael;
31                         }
32 
33                         l.invalidate;
34 
35                         recalculateLayout;
36 
37                         event.stopPropagation;
38                 });
39 
40                 addEventListener("mousedown", document.querySelectorAll("input[type=text],textarea"), (Element ele, Event event) {
41                         browsingContext.focusedElement = ele;
42                 });
43                 addEventListener("mousedown", document.querySelectorAll("input[type=submit]"), (Element ele, Event event) {
44                         auto f = ele.getParent!Form;
45 
46 
47                         browsingContext.focusedElement = null;
48                         // FIXME: only if action == POST
49                         document = gotoSite(window, browsingContext, f.action, f.getPostableData);
50 
51                         recalculateLayout;
52 
53                         ael;
54                 });
55         }
56 
57 
58         document = window.gotoSite(browsingContext, "https://dpldocs.info/");
59         recalculateLayout();
60         ael();
61         window.runHtmlWidget(browsingContext);
62 }

You could try using it yourself, again, building with dmd -i when the full arsd repo is available to you (can just git clone it to your working directory), and you might find some if it ... kinda works. But it obviously is not usable, even for simple websites. It also has some annoying lag, due to the old curl code loading the sites in a blocking manner, and it just being generally slow.

But on the other hand, you can see some image, form, and css support. The main problem it has is the word wrap is pretty horribly broken. In 2013, I didn't have the tools to handle this. In 2023, I do - the new arsd.textlayouter module. I also now have a better Uri.basedOn method, which would fix some links that are broken there.

Perhaps some day I might revisit this and make it slightly less awful. It'd be kinda fun to have a working browser, even if it is barely capable of reading a basic Wikipedia article.

Blog Articles
Page generated by adrdox