See more at the announce forum.
I got back to working on cgi.d's new features this weekend. I wanted to implement more of the add-on server APIs, and in accordance to my zero-dependency policy, and at first thinking the api would be very simple, I started off implementing it in a simple style: structs with static buffers.
However, this quickly got tedious and annoying, especially as I was adding more than the initial two functions (which had to be written by hand anyway, because they use special features, but 95% of functions I had planned don't need that).
I wanted to do something better! So, I wrote a little bit of code generation. (Such can be done in D in just a few dozen lines, so no need to modularize it. And besides, by making it private, I can do the simplest implementation that works for me and just call it done.)
And you know, D has come a long way in fixing bugs since when I wrote my first versions of reflection and code generation. Let's take a look!
1 interface BasicDataServer { 2 void createSession(string sessionId, int lifetime); 3 void setSessionData(string sessionId, string dataKey, string dataValue); 4 string getSessionData(string sessionId, string dataKey); 5 }
That's the basic idea: I'll define an interface and then have two classes: one for the client and one for the server. The client one will be mostly auto-generated, simply calling functions on the interface through the remote server. The server will implement the interface, and use an auto-generated function to communicate calls to and from the other processes.
I feel that the interface and class keywords are underrated by the most vocal parts of the D community. Yes, it is true that you can build the same features from other blocks... but I think it is quite difficult to get all the little details right that just work with interfaces, and at the same time, the costs of classes/interfaces is often overstated, and/or can be made to disappear, often easily, by just templating on a final class or emplacing on the stack, etc.
(I agree that there are some wasteful bits in D's object runtime that I would like to remove, but I don't want to digress too far and regardless, for my purposes here, those wasteful bits aren't important to me.)
Here, the interface gives a syntax-light way to write various details that D's reflection can very easily work with - the generated implementation classes will automatically inherit most attributes, for example! - and will offer compiler-checked guarantees for return values and arguments on both sides to ensure a compatible interface.
Anyway, let's move on to the code generation. Let me show you (an excerpt from) the client code generator first:
1 mixin template ImplementRpcClientInterface(T, string serverPath) { 2 static import std.traits; 3 4 static foreach(idx, member; __traits(derivedMembers, T)) { 5 static if(__traits(isVirtualFunction, __traits(getMember, T, member))) 6 mixin( q{ 7 std.traits.ReturnType!(__traits(getMember, T, member)) 8 } ~ member ~ q{(std.traits.Parameters!(__traits(getMember, T, member)) params) 9 { 10 SerializationBuffer buffer; 11 auto i = cast(ushort) idx; 12 serialize(&buffer.sink, i); 13 serialize(&buffer.sink, __traits(getMember, T, member).mangleof); 14 foreach(param; params) 15 serialize(&buffer.sink, param); 16 17 auto sent = send(buffer.sendable); 18 assert(sent); 19 } 20 }); 21 } 22 }
Yes, I defined the serialize function before the buffer struct... for now. I'll probably refactor that soon enough. But regardless, the point is here: I wrote a simple serializer (that only works on a limited set of types, but that is plenty - in 14 lines, it supports integral types, strings, structs of the above, and arrays of the above - a LOT of useful types!) and a loop over the interface members, which does a mixin string of the function body.
I'd like to spend some time on a few subtle details of this: first off, the only reason this is a mixin string at all is that it is the only way for static foreach to declare something with a unique name! I consider this to be a near-fatal design flaw in static foreach; it makes it only a very mild improvement over what we had before, since it must use almost exactly the same code we'd use without it anyway! That said though, the mild improvements we have are still nice.
Secondly, as I have mentioned in tips of the week before, you will notice that the vast majority of the body is simply in strings - the only use of concatenation (or interpolation btw) is for the function name. Everything else is inside the string - .stringof in mixin strings is usually a mistake!
Thirdly, I want to point out something I spent a bit of time on: all the newlines happen inside the mixin string! Why did I do that? It is so the assert line number matches up to the original source.
See, the runtime errors will give the line number as line of the mixin + the interior line offset of the string.
1 void main() { 2 mixin( 3 "assert(0);" 4 ); 5 }
core.exception.AssertError@ooooo.d-mixin-2(2): Assertion failure ---------------- ??:? _d_assert [0x806a81e] ??:? _Dmain [0x806a795]
Note the error claiming line 2 (in the parenthesis - the mixin-2 thing is the line where the mixin appears), whereas the assert itself is on line 3 of the original source.
By making sure all the newlines after the mixin keyword are inside the string itself though, this lines up nicely.
This brings me to one nice advantage of static foreach over the old way though: with the old way, you'd have to write a helper function to generate the string, then mix it in. It'd have to be careful to refer to loop indexes to template arguments in the generated string, so it still compiles when mixed in back at declaration scope. With static foreach, the string can simply refer to the regular loop variables directly, no awkward ~to!string(idx)~ in there! This makes the q{} strings a LOT easier to use. Note in my code above, the only thing outside the mixin string (non-italic in the sample) is the name - everything else is written quite plainly.
And thanks to all that, the code is quite simple! No interpolation necessary :P And, given the limited scope here, I could also skip out on most declaration identification (is it a function? or enum? or what) code, and since I am using a local pipe, I just binary dump stuff into the pipe, simplifying that code too.
Best part? So far, the compile time impact is negligible. Yay! (Cgi.d is kinda slow to build though, at about 1 second on my box. But it is still acceptable.)
I'm pretty happy with it. I want to do version 2.0 of my web.d sometime soon, and when I first did that (way back in like 2010!), compiler bugs led to a lot of painful hacks. I made it work, but it is ugly and took a lot of code. I think the new version will be a lot cleaner, and this makes me think I can do it far more compactly. (And UDAs are a thing now! oh my.)
Lastly, I take this generated content, and put it in a class:
class BasicDataServerConnection : BasicDataServer { mixin ImplementRpcClientInterface!(BasicDataServer, "arsd_session_server"); }
I am thus free to write additional methods by hand, if I desire, and the connection class itself has a reasonably readable name. All while the compiler still enforces the interfaces - including putting user calls through this instead of calling the server class directly (I can keep those private).
The server starts off about as simple as it gets:
1 final class BasicDataServerImplementation : BasicDataServer, EventIoServer { 2 void createSession(string sessionId, int lifetime) { 3 sessions[sessionId.idup] = Session(lifetime); 4 } 5 void setSessionData(string sessionId, string dataKey, string dataValue) { 6 sessions[sessionId].values[dataKey.idup] = dataValue.idup; 7 } 8 string getSessionData(string sessionId, string dataKey) { 9 return sessions[sessionId].values[dataKey]; 10 } 11 12 private struct Session { 13 int lifetime; 14 string[string] values; 15 } 16 private Session[string] sessions; 17 18 protected: 19 /* snip some irrelevant I/O implementation */ 20 21 void handleReceivedMessage(ubyte[] message) { 22 dispatchRpcServer!BasicDataServer(this, message); 23 } 24 }
No magic here, it is a very straightforward implementation class, except for the final method there - what is dispatchRpcServer? That calls the function from the pipe. Here's the code:
1 2 void dispatchRpcServer(Interface, Class)(Class this_, ubyte[] data, int fd) if(is(Class : Interface)) { 3 ushort calledIdx; 4 string calledFunction; 5 6 deserialize(data, calledIdx); 7 deserialize(data, calledFunction); 8 9 import std.traits; 10 11 sw: switch(calledIdx) { 12 static foreach(idx, memberName; __traits(derivedMembers, Interface)) 13 static if(__traits(isVirtualFunction, __traits(getMember, Interface, memberName))) { 14 case idx: 15 assert(calledFunction == __traits(getMember, Interface, memberName).mangleof); 16 17 Parameters!(__traits(getMember, Interface, memberName)) params; 18 foreach(ref param; params) 19 deserialize(data, param); 20 21 static if(is(ReturnType!(__traits(getMember, Interface, memberName)) == void)) { 22 __traits(getMember, this_, memberName)(params); 23 } else { 24 auto ret = __traits(getMember, this_, memberName)(params); 25 send_back_to_client(ret); 26 } 27 break sw; 28 } 29 default: assert(0); 30 } 31 }
This is the inverse of the client impl: take the index off the interface, assert mangleof matches just for a sanity check in case it was compiled separately against different versions (note the mangle includes arg and return types, so these are checked this way too), then pick off the params and call the function.
Here, I kept in the code for the return value just because I really wish D could declare void variables. That'd avoid the static if check above. (Or, perhaps, would move it below, but still a bit less wordy). Still, one check isn't too bad to get the function called!
And when I add new functions to the interface, the compiler will complain if I forget them in the server, but all the RPC glue is generated. Yay!
With this automating some of the tedium, I can now implement the remaining low-level functions and probably write a higher level API of top of it to finish the session server and the delayed jobs server. The event source server may also be extended with these. (I am thinking about making an admin/debugging interface too, which can query current state, dump to files, restore, etc., but that will come later. Still, it will be very easy with these auto-generated things! And I can even make them use multiple interfaces and OOP it all.)
I still have the websocket server framework to finish, as well as lots of refactorings now that this is all coming together, but I am getting to be fairly happy with it.
In the dmd land, the Objective-C binding keeps getting new stuff pulled, so that is progressing quickly right now, and in my world, I am writing new from-scratch reflection using dmd's new features.