automatic web interface discussion, reflection tips and tricks
Posted 2019-02-11
Blog
Articles- terminal.d gets clipboard functions, ldc 1.20 out.
- DConf keynote speaker announced: Lua architect Roberto Ierusalimschy, Named args DIP discussed
- February 3, 2020
- Adam's terminal suite explained
- Understanding mixin templates, terminal.d improvements
- My attribute-by-default proposal. Also dmd 2.090 came out.
- DConf 2020 announced: June 17-20 in London. @safe by default debated. Adam did: Android, JNI, WebSocket in arsd libs
- tar.xz, --DRT tip, dom bug fixes, more Android and JNI, link to old phobos docs
- LDC 1.19 - Android, AVR. My rant on tests, update on JNI and COM.
- Walter's string interpolation proposal is OK but not great. My Android thing nearing beta release. dub downtime explained.
- Android project update, introduction to arsd.jni
- New pattern about interface contracts
- Adam shares Windows console secrets - DO NOT USE chcp!!
- Adam's rant on benchmarks
- Socket tutorial
- November 4, 2019
- October 28, 2019
- arsd package updates, forum nonsense
- Update on Android
- Adam does iOS "goodbye world"
- September 30, 2019
- D turns 20, Adam rants on software freedom
- Named arg DIPs and my thoughts on code organization
- September 9, 2019
- I wrote about mixin templates vs string mixins on Stack Overflow
- August 26, 2019
- Bug bounty in D again - my hot take, on reusing code, a fun picture, my tentative plan for the next month
- Time invested is worth a lot
- cgi.d's new scheduler, static this tricks
- July 29, 2019
- July 22, 2019
- Solving vs managing problems
- A big week in the arsd repo
- July 1, 2019
- June 24, 2019
- June 17, 2019
- CRTP thoughts, named arguments DIP review, DConf videos now on youtube
- musings on hybrid CT/RT tests, some more progress on new web framework
- a little more webassembly
- May 20, 2019
- Adam's string interpolation proposal
- DMD 2.086 live, GCC 9 with D support formally released, DConf coming soon, links to posts on builder pattern and disallowing implicit conversions with templates, and 2d array op overloads
- template constraint error improvements coming?
- dmd 2.086 beta, dstep 1.0 released, Adam works on memory usage
- obj-c and webassembly report, tips on is expressions linked.
- new ldc, new dmd, dpp on the blog
- D's future discussed in forums
- LDC beta, DConf blog link, Adam introduces gamehelpers.d
- March 18, 2019
- LDC 1.15.0-beta1, responsive design rant
- dmd 2.085.0 released
- Obj-C interop and D without druntime code to copy/paste
- dmd beta, more info coming next time, demo of new web framework initial prototype
- automatic web interface discussion, reflection tips and tricks
- Adam busy with weather and a move, lots of community announcements
- January 28, 2019
- Working on official blog 2018 retro, C++ new wrapped, dmd reading zips?
- dmd obj-c growing, Adam static foreaches an interface to RPC
- dmd 2.084, hope for future, but busy non-D week for me
- IDE tools released, my cgi.d gets new features
- DConf announced, tip, Adam rants: mouse trap
- This Week in D is back!
I got some work done on new web interface generation this week. A new dmd bug fix release came out, among many other community announcements, too.
Core D Development Statistics
In the community
Community announcements
See more at the announce forum.
What Adam is working on
I finally got working on something I have been wanting to do a long time: a new version of my automatic website and http api generation code. It is now integrated with cgi.d. Though not all features are fully operational yet, you can play with this immediately by grabbing the ~master branch of my arsd repo, however, note that I am still working on it and it is still going to change and totally break any code written with it.
Here's one of my test files, ncgi.d:
You can see the results of that code running live here: http://arsdnet.net/cgi-bin/ncgi/api/tableWithDefaultArgs and http://arsdnet.net/cgi-bin/ncgi/api/table and, for the output format demo, see: http://arsdnet.net/cgi-bin/ncgi/api/tableWithDefaultArgs?format=json
The program takes about 1.4 seconds to compile on my computer, which is a little slow for my likes, but not bad.
As you can see in the demo links (if they are still live when you are reading this), that simple function generates some basic html code for the functions and for the return value, along side automatic serialization into json, if requested.
That's just what I got done so far, there's more coming!
Background and history
cgi.d is one of my oldest, still-maintained D libraries, but it, by itself anyway, is not what most my D web code is built on. Instead of using cgi.d directly, I would use a separate library called my web.d which builds on top of the basic library to provide a lot of convenient features via D's reflection and code generation.
I wrote the first version of web.d in 2010. D's reflection was very buggy and fairly limited compared to what we had today (I actually did a dmd PR myself to add one feature I wanted for it: __traits(getProtection)... though all of us were lacking in experience back then and it was misdesigned, giving trouble with private members due to that design flaw :( See, it takes a symbol as an argument, but you cannot access a private symbol in the first place... except allMembers still returns them. Meaning you need a hack to filter private members out before you even check the protection, blargh.
Moreover, my experience in web APIs was limited, too. My way of writing web code back then was mostly switch($_GET["operation"]) { .... } , more or less. web.d reflects this - you had to write a gigantic class (well, in later versions, by 2012, it could be multiple classes, but I'd still mostly use one gigantic one) with all the functions and it dispatched them by name. It would generate the initial UI automatically - a form with the function parameters, the return values turned into HTML.
It was great for whipping out first versions, and for making minimal, but functional, UIs for internal-use code with barely any work. But, it wasn't great for end-user sites, which tended to combine features from multiple functions, need custom design and formatting, etc. In practice, much of the code I'd write with it would actually take little to few arguments and just return hand-written html template instantiations; very little of the reflection-based automation would actually make it into the final, live version.
I still like it better than most web frameworks I have used! Those automatic features are still great, even if I write 80% of the site, it still saves me that 20% of the work. (And, in the parts I hand write, I still get to enjoy my arsd.dom based templating system, which I like a lot. You write progressively enhanced HTML and CSS!)
But, that said, I knew I could to a lot better. D gained user-defined annotations after web.d's first version, and I added some support to them to later versions, but never really got to embrace them because it just didn't fit in existing code. And dmd didn't allow UDAs on function parameters, which really disappointed me.
However, UDAs on functional params are now supported! And dmd's reflection and code generation is a lot nicer to use than it was in 2010. And I have a lot more experience in various web frameworks and better know what I do and do not like about them.
We can definitely do better now. And while I have wanted to do web.d 2.0 for years now, I didn't get around to it... until now.
Implementation
The implementation is realized using - you guessed it - D's compile time reflection capabilities, static if, and templates. Most of this is fairly straightforward now, though one trick is still there: I'll pass the method as a compile-time alias for introspection purposes, AND as a runtime delegate to be called on the specific object:
Trying to call the CT alias directly will cause a "need this for function whatever" error. But trying to get full reflection information off a delegate will fail since much of it is not passed to runtime arguments. Thus, I needed to pass both. (Or maybe there is a better way, if so, someone email me and let me know!)
The inside is mostly easy. Very few work-around hacks needed now, but let me point out a few trickier parts:
__traits(isDataMember)?
There is no __traits(isDataMember), but you can find them fairly easily with __traits(compiles, member.offsetof). Thanks to the IRC folks - tbh I don't remember who exactly - for reminding me of this.
How to get UDA on a function param
If you have tried to use last year's feature of putting UDAs on function params, you probably noticed it does NOT work to get them through Phobos. std.traits.getUDAs and std.traits.Parameters loses them at some point in the chain.
Even the most straight-forward way of using the is expression and __traits to get param UDAs doesn't work!
(BTW, did you know about is(whatever P == __parameters)? It is very poorly documented in the spec, but it is a magic incantation that aliases P to the function parameters. Can only be used inside a static if, but I'm OK with that - that also works as a convenient test for isMethod while we're at it!)
So, what is the answer? You have to actually slice the tuple returned by is(... == __parameters):
I don't know why the compiler insists on that, but it does. A little awkward, but fully functional once you know the trick. (I got clued into this by reading the Phobos source code, so I do not know to whom to give the credit.)
Future Direction
Well, as you can see from the sample above, the only part of it that is working right now is basically the same ideas as the old one: write a class with a bunch of functions. Get auto-generated forms and return values out of them.
I know I can do better (and wow, this new implementation is a LOT nicer than the old one), but I don't want to throw out the wins of the old web.d - I actually do like this functionality. I want to keep it as an option, but not tie myself down.
As such, I wanted to explicitly build on cgi.d itself, with a starting point. That's why there is that dispatcher function. I plan on making various functions for it to handle, and probably build up a routes object for compile-checked linking. (You'd give it a function name, which the compiler checks, and it then uses the route object to turn it to a link string.)
The dispatcher gets several entry points, so you can mix and match paradigms as you like. Easy serving of files or data literals, calling a handler function with the Cgi object, or using the fancier code generation functions.
The function-based code generator is basically working. I also want to do an object-based code generator. And that's part of what held me back historically from making web.d 2.0 - I could never think of a design I was happy with for "REST" APIs. And part of that was I because I wasn't really happy with REST APIs.
Well, I did a fair chunk of reading about REST over the weekend to better understand the underlying theories (I would sum it up as "hyperlinks!" rather than what most people talk about in their "RESTful" documentation.) and wrote several design drafts for what the user code would look like and the theoretical backing for it.
And I think I have something. But, I haven't had time to implement it yet. You can see some evolution in comments near the end of cgi.d right now, and hopefully, I will have it up next week. (And then I need to return to the add-on server stuff I wrote about last month, and implement that for the Windows platform! So still lots of work to do to finish all the stuff I have been writing about.)
I'll probably write about the REST code when it is usable next week, so stay tuned.
Downsides
If you use these new features, compile time increases. On my computer, it went from 1.1s to 1.4s to build the libraries. Alas. I might be able to improve that later though.
This new stuff also adds a dependency on arsd.dom and arsd.jsvar, but it is templated so you don't pay for it if you don't use it.
Otherwise though, it allows fallback to cgi.d's core strength - its flexibility - so I don't think this will significantly limit anything.