This Week in D

Adam's dynamic link transition

Blog
Posted 2020-06-22

Been busy work work but will tell a brief tale about static to dynamic link transitioning in D code.

Core D Development Statistics

In the community

Community announcements

See more at the announce forum.

Long time without writing, I know, been busy with a variety of things.

What Adam is working on

One of the small things I did in the last few weeks is convert some more static bindings do dynamic ones. Doing this purely by hand is a bit of a pain, since D's syntaxes are pretty different and the loading is a little repetitive:

With static bindings:

// before
extern(C) /* etc other attribute */ {
	int printf(const char*, ...);
	int atoi(const char*);
	// etc etc etc
}

With dynamic bindings:

// after
extern(C) {
	int function(const char*, ...) printf;
	int function(const char*) atoi;

	void load() {
		printf = cast(typeof(printf)) GetProcAddress(lib, "printf");
		atoi = cast(typeof(atoi)) GetProcAddress(lib, "atoi");
		// plus possible error checking etc.
	}

It isn't the biggest of problems, but it is a little obnoxious since you need to move the name in the declaration, then repeat it thrice in the loader. (Actually, and this might be heresy, but I kinda like the C function pointer syntax at times like this.) Moreover, I extern(C)'d the whole thing, whereas technically I really only want the function pointer types to have the extern, not the variables themselves, but that takes even more lines to separate out and I'm the laziest. Isn't there any simpler way to expedite this migration? Well, I have an idea:

1 // the interface is really just to group the functions together
2 interface A {
3 	// but then inside here, the actual definitions do not need to change!
4 	extern(C) /* etc*/ {
5 		int printf(const char*, ...);
6 		int atoi(const char*);
7 		// etc etc etc
8 	}
9 }
10 
11 // Then a little mixin action over the interface takes care of the rest.
12 private mixin template DynamicLoad(Iface, string library) {
13 	// define the variables to hold the pointers
14 	static foreach(name; __traits(derivedMembers, Iface))
15 		mixin("typeof(&__traits(getMember, Iface, name)) " ~ name ~ ";");
16 
17 	private void* libHandle;
18 
19 	// and load them up
20 	void load() {
21 		version(Posix) {
22 			import core.sys.posix.dlfcn;
23 			libHandle = dlopen("lib" ~ library ~ ".so", RTLD_NOW);
24 		} else version(Windows) {
25 			import core.sys.windows.windows;
26 			libHandle = LoadLibrary(library ~ ".dll");
27 			alias dlsym = GetProcAddress;
28 		}
29 		if(libHandle is null)
30 			throw new Exception("load failure of library " ~ library);
31 		foreach(name; __traits(derivedMembers, Iface)) {
32 			alias tmp = mixin(name);
33 			tmp = cast(typeof(tmp)) dlsym(libHandle, name);
34 			if(tmp is null) throw new Exception("load failure of function " ~ name ~ " from " ~ library);
35 		}
36 	}
37 
38 	void unload() {
39 		version(Posix) {
40 			import core.sys.posix.dlfcn;
41 			dlclose(libHandle);
42 		} else version(Windows) {
43 			import core.sys.windows.windows;
44 			FreeLibrary(libHandle);
45 		}
46 		foreach(name; __traits(derivedMembers, Iface))
47 			mixin(name ~ " = null;");
48 	}}
49 
50 mixin DynamicLoad!(A, "c") libc;
51 
52 void main() {
53 	import core.stdc.stdio;
54 	libc.load();
55 	// old usage code continues to work!
56 	printf("hello %d\n", 45);
57 }

Now, it doesn't handle lazy loading - you need to use a module ctor or an explicit call to the load function. But the rest of your "before" code cotinues to work just as well after the migration (well, aside from possible thread issues, but you could take the pointers out of TLS and initialize ahead of time and should be fine like that).

I considered trying to lazy load all the pointers on demand, initializing it to a thunk that actually loads the library then retries the call transparently to the user. But I haven't made it work to actually do that *reliably* yet without actually generating a fair amount of code and hurting compile times. I imagine it should be possible to write a generic naked asm procedure that does the trick, including thread safety, but I need to spend more time on it than I have right now and just being like shared static this() { load; } is so much simpler lol.

Anyway, I haven't used this in the real world yet, but I plan to try soon: I want to change simpledisplay.d to dynamic link xlib and friends on Linux. I figure its static ctor will try to load the libs and if it fails, just set a flag you can explicitly check and fall back if needed. Then you also see if the X connection succeeds and if so, use it, but if not, you can carry on without it (the get connection function will also check that flag and throw if necessary). I'll also make terminal.d's embedded emulator option use this check so it can gracefully degrade back to a normal terminal if needed (this will prolly be configurable in a flag setting too). A fair bit of work to bring it all together but I anticipate no serious problems - and this little interface loader trick makes it less tedious to do across all the various functions simpledisplay.d uses.

(sometimes I am tempted to move from xlib to speaking the protocol directly (think: xcb), which would also accomplish this goal, among others actually, it is just a lot of legacy code to transition. Again, laziness.

The one somewhat scary thing is this would be a potentially scary change: your code might compile on a system then segfault at runtime because of the unloaded library. I think I'll at least put in a default abort thunk to give a nicer error message to the user in the event of this happening. But while it is a change in a sense, it is really just a move of one type of error to another.

BTW people have asked me "why not use derelict/bindbc?" and... I actually don't understand what value they are supposed to provide. Maybe I'm just not reading the code and docs right but it seems to me those libs are really just a very thin wrapper around the OS apis and a set of conventions on how to copy/paste names. Doesn't look like it would help with transitioning existing code and tbh I'm not even sure if it would help with all fresh code either.

Well, I'm hoping to get back to writing next week, maybe talking about module-based versioning or something like that, or a how-to on stdout redirect code, but no promises... been pretty busy with work and nothing much to share in public about it in the near term.

Blog Articles
Page generated by adrdox