automatic web interface discussion, reflection tips and tricks

Posted 2019-02-11

I got some work done on new web interface generation this week. A new dmd bug fix release came out, among many other community announcements, too.

Core D Development Statistics

In the community

Community announcements

See more at the announce forum.

What Adam is working on

I finally got working on something I have been wanting to do a long time: a new version of my automatic website and http api generation code. It is now integrated with cgi.d. Though not all features are fully operational yet, you can play with this immediately by grabbing the ~master branch of my arsd repo, however, note that I am still working on it and it is still going to change and totally break any code written with it.

Here's one of my test files, ncgi.d:

1 import arsd.cgi;
2 
3 struct Person {
4         string name;
5         int age;
6         string[] skills;
7 }
8 
9 class MyClass : WebObject!() {
10         Person[] table(Person[] people) {
11                 return people;
12         }
13 
14 	Person[] tableWithDefaultArgs(Person[] people = [
15 		Person("Adam", 32, ["D", "Apotheosis"]),
16 		Person("Alice", 29, ["Software Engineering"]),
17 		Person("Bob", 50, ["Mathematics", "Technical Writing"]),
18 		Person("Chanara", 46, ["Project Management", "Business Administration", "Python"])
19 	]) {
20 		return people;
21 	}
22 }
23 
24 void handler(Cgi cgi) {
25         if(cgi.dispatcher!(
26                 "/api/".serveApi!MyClass,
27                 "/ncgi.d".serveStaticFile,
28         )) return;
29 
30         cgi.write("404\n");
31 }
32 
33 mixin GenericMain!handler;

You can see the results of that code running live here: http://arsdnet.net/cgi-bin/ncgi/api/tableWithDefaultArgs and http://arsdnet.net/cgi-bin/ncgi/api/table and, for the output format demo, see: http://arsdnet.net/cgi-bin/ncgi/api/tableWithDefaultArgs?format=json

The program takes about 1.4 seconds to compile on my computer, which is a little slow for my likes, but not bad.

As you can see in the demo links (if they are still live when you are reading this), that simple function generates some basic html code for the functions and for the return value, along side automatic serialization into json, if requested.

That's just what I got done so far, there's more coming!

Background and history

cgi.d is one of my oldest, still-maintained D libraries, but it, by itself anyway, is not what most my D web code is built on. Instead of using cgi.d directly, I would use a separate library called my web.d which builds on top of the basic library to provide a lot of convenient features via D's reflection and code generation.

I wrote the first version of web.d in 2010. D's reflection was very buggy and fairly limited compared to what we had today (I actually did a dmd PR myself to add one feature I wanted for it: __traits(getProtection)... though all of us were lacking in experience back then and it was misdesigned, giving trouble with private members due to that design flaw :( See, it takes a symbol as an argument, but you cannot access a private symbol in the first place... except allMembers still returns them. Meaning you need a hack to filter private members out before you even check the protection, blargh.

Moreover, my experience in web APIs was limited, too. My way of writing web code back then was mostly switch($_GET["operation"]) { .... } , more or less. web.d reflects this - you had to write a gigantic class (well, in later versions, by 2012, it could be multiple classes, but I'd still mostly use one gigantic one) with all the functions and it dispatched them by name. It would generate the initial UI automatically - a form with the function parameters, the return values turned into HTML.

It was great for whipping out first versions, and for making minimal, but functional, UIs for internal-use code with barely any work. But, it wasn't great for end-user sites, which tended to combine features from multiple functions, need custom design and formatting, etc. In practice, much of the code I'd write with it would actually take little to few arguments and just return hand-written html template instantiations; very little of the reflection-based automation would actually make it into the final, live version.

I still like it better than most web frameworks I have used! Those automatic features are still great, even if I write 80% of the site, it still saves me that 20% of the work. (And, in the parts I hand write, I still get to enjoy my arsd.dom based templating system, which I like a lot. You write progressively enhanced HTML and CSS!)

But, that said, I knew I could to a lot better. D gained user-defined annotations after web.d's first version, and I added some support to them to later versions, but never really got to embrace them because it just didn't fit in existing code. And dmd didn't allow UDAs on function parameters, which really disappointed me.

However, UDAs on functional params are now supported! And dmd's reflection and code generation is a lot nicer to use than it was in 2010. And I have a lot more experience in various web frameworks and better know what I do and do not like about them.

We can definitely do better now. And while I have wanted to do web.d 2.0 for years now, I didn't get around to it... until now.

Implementation

The implementation is realized using - you guessed it - D's compile time reflection capabilities, static if, and templates. Most of this is fairly straightforward now, though one trick is still there: I'll pass the method as a compile-time alias for introspection purposes, AND as a runtime delegate to be called on the specific object:

// first is the CT alias, second is the runtime delegate
auto ret = callFromCgi!(__traits(getMember, obj, methodName))(&__traits(getMember, obj, methodName), cgi);

Trying to call the CT alias directly will cause a "need this for function whatever" error. But trying to get full reflection information off a delegate will fail since much of it is not passed to runtime arguments. Thus, I needed to pass both. (Or maybe there is a better way, if so, someone email me and let me know!)

The inside is mostly easy. Very few work-around hacks needed now, but let me point out a few trickier parts:

__traits(isDataMember)?

There is no __traits(isDataMember), but you can find them fairly easily with __traits(compiles, member.offsetof). Thanks to the IRC folks - tbh I don't remember who exactly - for reminding me of this.

How to get UDA on a function param

If you have tried to use last year's feature of putting UDAs on function params, you probably noticed it does NOT work to get them through Phobos. std.traits.getUDAs and std.traits.Parameters loses them at some point in the chain.

Even the most straight-forward way of using the is expression and __traits to get param UDAs doesn't work!

        static if(is(typeof(method) P == __parameters))
        static foreach(param; P) {{
		__traits(getAttributes, param); // doesn't work!

(BTW, did you know about is(whatever P == __parameters)? It is very poorly documented in the spec, but it is a magic incantation that aliases P to the function parameters. Can only be used inside a static if, but I'm OK with that - that also works as a convenient test for isMethod while we're at it!)

So, what is the answer? You have to actually slice the tuple returned by is(... == __parameters):

1         static if(is(typeof(method) P == __parameters))
2         static foreach(idx, _; P) {{
3 		// alias for convenience later
4                 alias param = P[idx .. idx + 1]; // this slice is the trick
5 		// can get the identifier now
6                 string ident = __traits(identifier, param);
7 		// and the attributes
8                 static foreach(attr; __traits(getAttributes, param))
9 			{}
10 	}}

I don't know why the compiler insists on that, but it does. A little awkward, but fully functional once you know the trick. (I got clued into this by reading the Phobos source code, so I do not know to whom to give the credit.)

Future Direction

Well, as you can see from the sample above, the only part of it that is working right now is basically the same ideas as the old one: write a class with a bunch of functions. Get auto-generated forms and return values out of them.

I know I can do better (and wow, this new implementation is a LOT nicer than the old one), but I don't want to throw out the wins of the old web.d - I actually do like this functionality. I want to keep it as an option, but not tie myself down.

As such, I wanted to explicitly build on cgi.d itself, with a starting point. That's why there is that dispatcher function. I plan on making various functions for it to handle, and probably build up a routes object for compile-checked linking. (You'd give it a function name, which the compiler checks, and it then uses the route object to turn it to a link string.)

The dispatcher gets several entry points, so you can mix and match paradigms as you like. Easy serving of files or data literals, calling a handler function with the Cgi object, or using the fancier code generation functions.

The function-based code generator is basically working. I also want to do an object-based code generator. And that's part of what held me back historically from making web.d 2.0 - I could never think of a design I was happy with for "REST" APIs. And part of that was I because I wasn't really happy with REST APIs.

Well, I did a fair chunk of reading about REST over the weekend to better understand the underlying theories (I would sum it up as "hyperlinks!" rather than what most people talk about in their "RESTful" documentation.) and wrote several design drafts for what the user code would look like and the theoretical backing for it.

And I think I have something. But, I haven't had time to implement it yet. You can see some evolution in comments near the end of cgi.d right now, and hopefully, I will have it up next week. (And then I need to return to the add-on server stuff I wrote about last month, and implement that for the Windows platform! So still lots of work to do to finish all the stuff I have been writing about.)

I'll probably write about the REST code when it is usable next week, so stay tuned.

Downsides

If you use these new features, compile time increases. On my computer, it went from 1.1s to 1.4s to build the libraries. Alas. I might be able to improve that later though.

This new stuff also adds a dependency on arsd.dom and arsd.jsvar, but it is templated so you don't pay for it if you don't use it.

Otherwise though, it allows fallback to cgi.d's core strength - its flexibility - so I don't think this will significantly limit anything.