extern(Objective-C) lets you use and write classes in a familiar D style that compile down to the same code as native Objective-C classes, written in Obj-C or Swift.
Here's an example from simpledisplay's Cocoa implementation:
extern(Objective-C) class SDWindowDelegate : NSObject, NSWindowDelegate { override static SDWindowDelegate alloc() @selector("alloc"); override SDWindowDelegate init() @selector("init"); SimpleWindow simpleWindow; override void windowWillClose(NSNotification notification) @selector("windowWillClose:") { auto window = cast(void*) notification.object; SimpleWindow.nativeMapping.remove(window); } override NSSize windowWillResize(NSWindow sender, NSSize frameSize) @selector("windowWillResize:toSize:") { if(simpleWindow.windowResized) { simpleWindow._width = cast(int) frameSize.width; simpleWindow._height = cast(int) frameSize.height; // this is an Obj-C method call! simpleWindow.view.setFrameSize(frameSize); simpleWindow.windowResized(simpleWindow._width, simpleWindow._height); } return frameSize; } }
Looks like slightly annotated D code, but compiles to something directly usable in the world of Objective-C.
D often tries to interface with existing code, being able to call most C and some C++ code directly. It even has some built-in support for Microsoft's COM interfaces. This gives good compatibility to Windows and Linux system apis, but is not ideal for Mac, iOS, and Android systems.
There's library solutions for those, like my own arsd.jni for interfacing with Android systems and other Java APIs, and you can do similarly for Objective-C too - you can interface with its runtime through extern(C) functions and libraries can wrap this up reasonably well to make it ergonomic too. If you have custom mangles and linker section support in the language, libraries can even be made to use with things like the xcode interface builder.
The library solutions can be solidly OK. I'm pretty proud of what arsd.jni achieved. But, thanks primarily to the work of Michel Fortin and Jacob Carlborg, D tries to go a step beyond that. See this blog post by Michel Fortin from 2007 to see an early attempt at a library solution: https://michelf.ca/blog/2007/d-objc-bridge/ and this DIP from 2013 to learn more about the history and aspiration of language support: https://wiki.dlang.org/DIP43
If you looked at the DIP, many features listed are marked unimplemented; it aspired to go further than has ever been implemented, even today. Until about 2019, very little of it worked at all. Jacob did the first PR in 2014, but it was not merged. He then did some piecemeal things, but it was not until about 2019 that the implementation really started to work upstream. Jacob put a lot of work into it: in 2018, and more in 2019 and even some more in 2020, getting huge chunks of it implemented. I did a blog post about it in October 2019 and did some bug fixes in it myself then too. It worked pretty well in dmd at this point - I got my D class using extern(Objective-C) working in the iphone simulator back then, 5 years ago!
However, the ldc implementation saw very little improvement since its original implementation by Dan Olson in 2016. Without the enhancements dmd saw in the years since then, it was essentially useless; if you wanted cross-compiler Objective-C calls, you had to use a library solution. After looking at the dmd implementation to fix that bug in 2019, I felt like it shouldn't be too hard to do in theory, but in practice I didn't know ldc and I don't use Apple products personally anyway, so I never actually got around to even trying until now.
extern(Objective-C)'s implementation on ldc is primarily in this commit: ba2147c
And while that worked locally, it triggered an assert error on ci (probably due to different llvm versions), so this follow-up commit fixed that: a8b2a88
Finally, one more bug was discovered while writing this blog post, which I fixed with this: 821b947ce1
I went into it somewhat familiar with dmd's implementation, by Jacob Carlborg, thanks to digging into it to fix that little bug five years ago, but with virtually zero ldc experience. Before forking it, I never even managed to get ldc to compile!
So, my implementation may not be very good. To be honest, this is the first time I've ever looked at ldc's actual code - just the build process was hard enough for me to get working for me in the past. Indeed, it is my first time looking at much of anything LLVM related.
The Objective-C runtime can be accessed through extern(C) functions with strings and pointers, and your classes are made available by registering their metadata with the runtime, either through those functions before first use, or by putting that data in a particular place with the linker so the runtime loader can do it for you automatically at function load.
Objective-C's object orientation traces its lineage back to the Smalltalk tradition, which is a bit different than the C++ / Java tradition D follows. When you "call a method" in D, Objective-C calls that "sending a message". They're very similar concepts, but the implementation is a bit different - in D's model, the compiler looks up the function and you jump to it. In Obj-C's model, you call one entry point function provided by the Obj-C runtime and *it* jumps to the implementation, if there is one.
Thus, calling an Obj-C method means looking up a symbol populated by the linker for the method name called a "selector" (basically an interned string) and passing it to the objc_msgSend function, along with the this pointer and its arguments. There's a variant, objc_msgSend_stret for methods returning structs, and a few others for x86 floating point returns, but they're all basically the same concept.
For a static method, in D, you'd simply call the function. There is no this object associated; the class' relationship to the static method exists purely at compile time. By contrast, in Obj-C, "static" methods are actually instead class methods; the this pointer is one representing the class itself, not your specific instance, which must be either dynamically looked up via objc_ runtime calls or available with a specific name via the linker for the loader to populate when starting up the program. (An Obj-C class is itself an runtime object as well as a compile time concept.) This necessitated a new AST object in the compiler, ObjcClassReferenceExp, to represent this runtime representation of the class.
Calling a super class implementation is similar, but a different function, objc_msgSendSuper is called, and instead of just taking this, it takes a struct that takes both this and the specific class object you want the method lookup to start on (it can still do inheritance, just it starts at the specified point instead of starting on the dynamic type of the passed this object). This also uses that ObjcClassReferenceExp in the compiler.
When defining your own class, you must put out all the data necessary to build those runtime objects to represent the class, as well as all the metadata necessary for the objc_msgSend implementation to match selector strings to your method definitions. It also has some type information for dynamic checking among other things. All this must be either registered at runtime via function calls or put in the right place for the loader to find it when starting your program.
The compiler needs to handle all this, and a few more things. For additional details, see the documentation Jacob Carlborg wrote upstream: https://github.com/dlang/dmd/blob/master/compiler/docs/objective-c_abi.md and search the web, Apple has some documentation as well (though it is not as detailed as I'd like!)
Thanks to OpenD, I had a merged dmd and ldc makefile I can run on my Linux box and successfully build either compiler from the same codebase, and I had a working github action that can build the same for the Mac and provide a download.
The CI is horribly slow, so I decided to instead cross compile. No real effort required here for the x64 linux over to x64 mac - just run opend ldc -mtriple x86_64-apple-darwin -c test.o on the linux box, then scp that file over to the mac I had setup for testing arsd and let my downloaded ldc link it into the rest of the test program for running or debugging. llvm-objdump could inspect the file, even right on Linux, so I wouldn't have to get on my mac box much at all. I could also compile the same program with dmd (only on the mac though, dmd's little cross compiler only sometimes works and failed with the objc bridge) and link in its object file instead to compare with everything else the same.
I could test an iteration in about 30 seconds - ldc's incremental compiles are not fast, but this process was ok. Then, when all done, I could download the CI build to do the final steps.
I had two test programs: one a simpledisplay hello world to use as an integration test (simpledisplay defines classes with instance vars that must receive events from the OS as well as calling out to several of them, so if its timer, drawing, key events, etc. worked for a basic program, that meant I had pretty good odds of more things working too), and the other a more stripped down basic test to use as a unittest of individual features as I went.
It was time to get started.
Knowing the goal and being familiar with dmd's implementation (mostly the bulk of it in src/dmd/objc_glue.d and some in src/dmd/objc.d) and having that starting point from Dan Olson in ldc, I figured this shouldn't be all that hard. Emboldened by having ldc building for OpenD and motivated by making the arsd.simpledisplay Cocoa implementation work (which now used this feature in dmd, helping it expand greatly in functionality compared to the old extern(C) based approach (which was contributed by kennytm back in 2011 and seeing that code also helped me a lot in understanding how these things are supposed to work), something I made big progress in last year thanks to help from Steven Schveighoffer) in both supported OpenD compilers, I got started.
My first goal was to make static methods work. Instance methods were already implemented in ldc, so I figured static methods are the next logical step as they work very similarly - remember, what we call a static method in D is actually just an instance on the class object in Obj-C, they work the same way, just being passed a different this.
Even this first goal was harder than I anticipated, though. It compiled, but the compiler died with an assertion failure saying the AST visitor was incomplete, missing type "class". Well, that's lovely, I wasn't even sure where to start! But I had to do something, so I first grepped the assert fail message and led to its definition... in a macro at the bottom of a big visitor class in the toir.cpp file. Yikes.
But this told me the message was formed from the kind() method. What has override char* kind() { return "class"; }? Time to grep the dmd code. ClassDeclaration came up, and I tried to add it to the visitor (even though that makes little sense), and it didn't work. But then the next hit looked promising: the ObjcClassReferenceExp class. This clicked - it was right there in the class name! (Pity it wasn't in the error message! But oh well, the two-step grep worked. That's a technique I use a lot for navigating an unfamiliar codebase - search for some user-visible string and follow the breadcrumbs back. Not always easy with virtual methods and variables and whatnot, but at least it is a starting point and it often does yield results.)
I added that to the visitor, recompiled (ldc is so slow to compile compared to dmd. seriously, sometimes people tell me to stop work on dmd, but I can't see that as a good idea - hacking on dmd has much much iteration cycles than ldc), and now saw a different error message. That's the right thing!
Next came getting that reference to the right Objective-C class. In dmd, the implementation used an object called Symbol, from its own backend. I needed to know the analog in ldc. The existing implementation for instance methods used a llvm::GlobalVariable, so I copied that here too. In fact, I was tempted to do typedef llvm::GlobalVariable Symbol to ease copy/paste, but the objects were too different for that to work anyway. However, in my mind, whenever I saw Symbol in dmd, I thought llvm::GlobalVariable in ldc.
Creating it was one thing - and speaking of slow compiles, edit a .h file in ldc and the recompiles that were already too slow for me, became SO MUCH SLOWER. I dreaded having to edit a .h file, this is why some of the functions in objgen.cpp are just randomly defined there with no prototype and take the class as an explicit argument - I just didn't want to edit the .h file and wait for the much slower recompile that'd cause.
Anyway, creating a llvm::GlobalVariable was one thing, I could copy paste that as a starting point, but now, what do I do with it? Just setting it to the result wouldn't compile. I had to look for some roughly analogous implementation elsewhere and copy it over. tbh, I'm still not sure I did this right, but I did eventually end up with something that works.
Eventually. First, I had to suffer a little.
The result = new DImValue(...) line from some other methods looked promising, so I copied that in. Recompiled - yay, it built! Used the dev compiler to build my test program - yay, it worked too! Was it really this easy? Ran the test program. Nope, it crashed.
It took me actually an embarrassingly long time to figure out why, especially because the disassembly comparison to dmd's working implementation should have screamed it out to me immediately, even using the horrible at&t syntax llvm-objdump was spitting out: one was doing essentially ptr and the other was doing *ptr, which makes sense - the global variable is itself not the class object, but rather is populated with the object, but it took me a while to realize this. And wanna know the worst part? I'd repeat this mistake later when doing the super call implementation at the end of this project. Started and ended with the very same mistake... I'm not as smart as I pretend to be lol.
Nevertheless, even after realizing this, the problem was not solved. Sure, I know how to write this in D, or even in x86 assembly, but how do I write this in ldc's llvm objects? Time to go hunting for another analogous example.
After much tribulation, I found the (tbh kinda obvious in retrospect) DtoLoad function. Compiled and... it worked! Yay.
But yes, it took several hours spread over two nights to end up with just this:
void visit(ObjcClassReferenceExp *e) override { auto loaded = DtoLoad(DtoType(e->type), gIR->objc.getClassReference(*e->classDeclaration)); result = new DImValue(e->type, loaded); }
I started to question if this was really worth it. In the time I originally thought I'd be almost done, I had instead only actually finished the most basic function. I could instead give it the arsd.jni treatment - some of my library magic that I already know so well to make the existing library implementation look more appealing. (Note that I had already done the dmd @section thing by this time too, which would make it possible to make a fairly comprehensive library solution.) But meh, I had my implementation in simpledisplay that depended on this language feature and I didn't want to redo that.
Besides, the language built-in IS cool. Really, the library solutions aren't bad, but the built in thing is actually nice to use. Especially for me: I'm not an Objective-C developer. I don't know (well, didn't know) what a "protocol" is, or what the heck "passing a message to the metaclass instance" means. But I do know what an "overridden method" and "static factory function" and such are. It made this foreign environment feel like something I could tackle, rather than waiting for another kennytm type to come in and contribute a wholesale implementation.
And I think there's real value in that. Part of why I started using D at all (many years ago!) is that it looked familiar enough to the C and Javascript I already knew. Programming languages need to meet users where they are. Sure, from there, you can branch out to new things, but doing everything new all at once just feels daunting. I like being able to relate things back to concepts I already know - instead of a whole new paradigm, it just feels like another new system API. I can handle that.
If it helped me branch out a bit, it might help others too. I've been talking about doing it for 5 years, and I had a guarantee this time that if I got it working, it'd actually work - I'm my own gatekeeper. And I made some progress. so let's finish this.
I thought super calls might be easy, but after doing some of this, I wasn't so sure. Instead of implementing them at this time, I just wanted to trace where they were happening and make the compiler error out on them.
Even this proved more difficult than it seemed. See, in dmd, there's a member called directcall used for this. It is named directcall because it called the function directly, instead of through the vtable. Makes enough sense, but at first, I didn't realize that was what that meant - I assumed it was objc specific and made the compiler error on any directcalls.
Then my other code started to randomly error. A method from a mixin template became a "super calls are not implemented in ldc objc yet" and I was like... what? But that's just because they're non-virtual.
super.x() (or its related ParentClass.x()) in D bypasses the virtual function dispatch and are thus directcall. final functions and some mixin template calls also are non-virtual, and thus directcall. Objective-C super calls are an extension of this concept, not a new thing.
So my implementation changed to pass down a new variable, objcdirectcall, down the chain to add the this implicit arg. This was only true if it was both directcall and objc. Kinda hacky, but it generated the error in the right place and didn't generate it elsewhere.
Good enough for now, I can come back to this, but first, let's get back to implementing stuff I actually needed to work.
Next, I wanted to declare and access an instance variable. simpledisplay uses one to tie the NSWindow object back to its own SimpleWindow object - it does similar on Windows and Linux to tie system event loops back to its own D handlers.
This was basically the rabbit hole that led to the rest of the implementation, as this meant not only its own implementation - instance vars in extern objc classes are not laid out like in normal D classes, but rather offset at runtime so that base class changes don't require recompilation of child classes (I actually really like all this stuff, it makes binding sooooo much easier, you just bind what you need and don't worry about what you don't), so it needed more of those llvm::GlobalVariables, more DtoLoad calls,
But then, my new class that uses those instance variables needs to be defined. And this means defining the metadata. How the heck do I do that?
The D impl, in objc_glue.d, uses this object from dmd's backend called DtBuilder. This works kinda like an output stream:
void toDt(ref DtBuilder dtb) { auto baseClassSymbol = classDeclaration.baseClass ? Symbols.getClassName(classDeclaration.baseClass, isMeta) : null; dtb.xoff(getMetaclass(), 0); // pointer to metaclass dtb.xoffOrNull(baseClassSymbol); // pointer to base class dtb.xoff(Symbols.getEmptyCache(), 0); dtb.xoff(Symbols.getEmptyVTable(), 0); dtb.xoff(getClassRo(), 0); }
My first thought was to do similar, I can just collect bytes and output an array, right? I called the function getGlobalWithBytes with this in mind - similar to getGlobal which fetches a Symbol (in dmd) / llvm::GlobalVariable (in ldc), but initializes it with some bytes (the llvm thing takes an initializer argument). I was even thinking maybe I could take the dt bytes from dmd and just pass that straight to ldc. That'd be nice, reuse the whole impl at once.
But... those xoffs are symbols, do they have a plain byte value? Isn't that up the linker to fix up and as such, they need to be marked appropriately, right? Well, tbh, I don't know for sure if that's true or not, but I assumed it was and went back to the drawing board. I needed something similar to DtBuilder but in ldc natively.
In the LLVM documentation, they have these class inheritance diagrams. I don't really see much value in those - a bunch of lines going this way and that way and curving around another thing - but this time, something struck me: there was a llvm::ConstantAggregate that had a child llvm::ConstantStruct... that had a static method getAnon.
An anonymous constant struct is a typed collection of bytes, right? And that getAnon method took an array ref of llvm::Constant*, of which llvm::GlobalVariable* is an acceptable item... I think I can work with that.
My C++ is rusty af, so I had to look up the difference between std::vector and std::array. I was pretty sure std::vector was the one I wanted, but had to double check! Nevertheless, std::vector<llvm:Constant*> became my new mental substitution for ref DtBuilder. I even made the xoffOrNull, size_tV, and dwordV helper functions to make the code look even more like the original D... and put it together. (These are the functions that i did kinda sloppily in the .cpp file to avoid touching the .h and triggering the slow slow full recompile, but meh, it worked.)
To my elation, this made what I needed to see in the output file. Adjusted the test program to use two separate ivars and assert that assignment did what it was supposed to do... and it did!
This was finally really starting to come together, it just took about 4x as long as I thought it would every step of the way. Time to test it... and probably fail, everything else was going overbudget, testing probably would too.
For the last several coding sessions, I was using my minimal "unit test" application, and now I wanted to see if the full goal worked - the gui program. So I got that simpledisplay program back out, compiled it... and it failed because super wasn't implemented yet. But meh, I happened to know the overridden method actually didn't do anything, so I just commented that out for now and compiled.
It worked! I copied the .o, linked on the mac, and ran it.... and... it worked! First time running the integration test at all for ages, and it was fully functional. The window came right up, the menu was populated, the timer went off on schedule, the keypresses registered. Everything.
Finally, something went right the first time!
Now, this test program didn't actually use everything. Its drawing was super minimal. I should try a more involved program - I haven't even done it as of writing this blog - to ensure the floating point things and structs all work right too.
Nevertheless, I was surprised at the program working correctly my first time trying to run it. This was a good sign. Time to move on to the next test.
In 2019, I found bugs in dmd's extern(Objective-C) implementation by testing it on an iphone simulator. This time, since I had ldc with its aarch64 codegen, I wanted to test on a hardware iphone. This was not easy for me - remember, I have almost zero experience with this at all (that 2019 thing being my only other attempt) - but after more pain with the ide and trusting myself and such, I got a sample program from xcode running on this device I bought from a friend a while ago and stuck in a box. More details on the xcode saga in a separate section of this blog post.
Then I tried my program... and turns out my dev build of ldc didn't support cross compiling to ARM. Ugh. It depends on what LLVM you build it against, I guess. I know the CI build did, but I didn't want to push up and wait for that until I was pretty sure the project was done.
But that's ok, the iphone simulator on xcode is x86, so I built for that and got the D running on it!
Well, I say "running", but more like "crashing".
I missed a spot. It complained it couldn't find the ViewController class I defined in D. Very similar to that 2019 experience, but I copied the working dmd code, which is still working today. What went wrong?
Another embarrassingly long time of a search, I found it: the class list metadata was incomplete. I did each individual class, but not the class list for the module as a whole. Had to add it to that finalize method... did that, and it worked!
Last thing was to uncomment that call to super() in the overridden method, then I'd be ready to push it up to CI and test it on that actual hardware with the downloaded ldc with arm support...
I wanted to uncomment that one line I skipped to make the integration test build, a call to super.
Remember how these work? objc_msgSendSuper, which is just like objc_msgSend, but instead of passing just the this pointer, you pass a struct that combines the this pointer with the class object to start the lookup from. Should be ez, right? I've already done all that elsewhere - anonymous structs, class objects, this pointers, just bring it all together and I'm all set.
Well, not exactly that simple. See, the function does not pass an anonymous struct. It passes a *pointer* to an anonymous struct. How the heck do you make one of those?
I knew about the DtoLoad function, but I actually need an address... time to go hunting. In dmd, Jacob's implementation in objc_glue.d pretended this struct to be a delegate literal and then took the addressElem of that. Maybe I could find a clue by looking for D delegates' implementation in ldc's code?
This did lead me to DtoAggrPair which made the right object... but still didn't get me a pointer to it. Went to the web and it was like "you need to store it", but store it where? Searches in the llvm docs for words like "llvm::LocalVariable" turned up little of use.
But then I searched ldc for the word "Store" and found DtoStore - ok, an obvious analog to DtoLoad (... at least obvious in retrospect, again). But where do I store it? Searched for some uses of it and found references to alloca.
OK, alloca makes space on the stack. This is usually used for things like an array, whose size is not known at compile time, but meh, let's do it. Searched for the word and found DtoRawAlloca that had the pieces I needed.
auto val = DtoAggrPair(
thisarg,
getVoidPtrType(), gIR->objc.getClassReference(*cls),
"super");
auto addr = DtoRawAlloca(val->getType(), 16, "super");
DtoStore(val, addr);
args.push_back(addr);Create the anonymous struct, make space to store it, store it there, then push the address of that space to the arguments list. Done!
...right? Well, I spoiled this earlier, but I made the same mistake I made when starting this project, failing to load from the pointer! And it took me a whole night of staring to figure it out. My poor head was aching so bad again.
And on this one, the disassembly difference between the working dmd code and the broken ldc code should have called it out to me: one used lea, the other used mov. mov and lea are both exceedingly flexible instructions in the x86 world and clever compilers can sometimes substitute one for the other (indeed, a sufficiently clever compiler can make just about anything out of mov instruction variants), so I kept brushing past it thinking it was just some dmd vs ldc optimization, but no, that was important and I should have paid attention.
The correct (well, as correct as I know, again, I suck at this and idk if im doing it even remotely right, just this worked so im shipping it lol) code, same as before, is to add a DtoLoad to the class reference:
auto val = DtoAggrPair(
thisarg,
DtoLoad(getVoidPtrType(), gIR->objc.getClassReference(*cls)),
"super");
auto addr = DtoRawAlloca(val->getType(), 16, "super");
DtoStore(val, addr);
args.push_back(addr);So, with that fixed with another load instruction object, and it worked in the simulator!
Pushed to CI, downloaded.... and the ldc failed with an assert error. Ugh, the void* thing that worked on my dev llvm failed on the other one. A version difference? I don't know and I didn't care anymore, I changed it to int8* in that second commit and pushed up again.
This time it worked! This project was done.... or was it?
While writing this blog post, I tried doing everything from scratch to ensure it worked as expected, and the compiler crashed. A bit perplexed because the resulting code was almost identical, I looked at the stack trace and recognized where it was coming from - calling that llvm::ConstantStruct::getAnon with an empty array segfaults. OK, yeah, I should add an if(!array.empty()) check, since it is supposed to be empty when only binding and that shouldn't crash, but this program isn't only binding... so why is it empty here?
One of the things my sample program had, from early on, was a call to TestClass.alloc(); in a side function. This was meant only to ensure the compiler wasn't crashing on a static method call - left there from way back and forgotten. When redoing it for blog publication, I did it clean, removing superfluous test code... which led to this crash.
See, for a class to be added to that array that caused the crash, it has to be referenced somewhere; a call to getClassReference was required to populate it. This was, at the time, only called when you tried to either inherit from it or call a static method on it! So, when I removed that static call, it also never called getClassReference, and was never added to the classes array for final output in the class list that Interface Builder (among other things) tries to find it in. So even if it didn't crash, the program would not work since it would not realize the class was defined.
The fix was simple enough: call getClassReference when a non-extern class is originally declared, without waiting for it to be used. But this bug almost eluded me all the way to publication!
So now, I'm finally done for real! ... I think. ... for now at least.
I did not finish porting the objc protocol / D interface bridge. simpledisplay didn't actually use this, and I was well overbudget on time as it was, so I left that for later. I didn't add anything new, so the objc "categories" (which are basically extensions for existing classes, idk why they called it that, maybe analogous to a D mixin template just at runtime, so a category of functionality? idk) are still not done, etc.
Adding the protocol / interface thing shouldn't be too hard after doing all the rest of this, so I'll surely come back to it eventually (unless someone else picks up this work and beats me to it!). Much of it is creating the same kind of metadata already done for classes, so that ought to be doable, and then adjusting semantic passes for optional methods - if required. This might not even need to be done in ldc since most the semantic passes are shared with dmd! (OpenD merging those two repos in one codebase has been nice.)
Lastly, the ldc impl errors on missing @selectors in extern objc methods, and the dmd impl does not, but it also doesn't work correctly with them. The dmd impl should also error on them, requiring you to mark non-objc methods with extern(D) consistently. I'll come back to add this later.
I kept getting annoyed with C++ having to know if things were refs or pointers or functions or members because you must either -> or ., and the parenthesis must be used a particular way. I just don't care, I simply wanted it to work! It was kinda annoying to have to keep looking it up or have the slow compiler correct me over these trivialities.
I like D lol.
This also brings me to the question of dmd vs ldc. A lot of people have advocated for dropping dmd entirely, but I think that would be a big mistake. Why? Those compile times! dmd is just so much faster to iterate on. ldc has its advantages, no doubt, but playing with language changes is so much faster and easier in dmd. OpenD has the right answer here - merge the codebases so you can build either one (or both) at any time with minimal fuss.
I also wanted to make linker sections work in dmd similarly to ldc. This is not strictly needed for this; it'd help with objc interfacing in a library more than in the compiler, but since I was there, I went ahead and did it too with this commit: 048c509.
pragma(linkerDirective, "-framework", "Cocoa"); // on Mac pragma(linkerDirective, "/subsystem:windows"); // on Windows pragma(linkerDirective, "\"/manifestdependency:...\""); // also Windows
pragma(linkerDirective) tells the linker to do something when linking the object file from this module. It can be all kinds of things - telling it to also link in a particular library, or add metadata to the executable, or various other random things.
In particular, in the Apple ecosystem, it was the suggested alternative to pragma(framework), which would analog to pragma(lib). A "framework" is an Apple term for a certain type of bundle of libraries that must be linked in differently than pragma(lib) supports. But, they're not that much different than any other library, so it is frustrating that pragma(lib) doesn't work on them and I have wanted some solution for years.
It was implemented on Windows in dmd released in November 2018 and in ldc some time later, for both Windows and Mac. (I believe it works on Linux too but has less purpose there, as the Linux user environment doesn't use exe metadata as much as Mac and Windows.)
It was not implemented on Mac in dmd upstream (well, it was, but the PRs were always rejected) and only recently in OpenD.
Our pragma(linkerDirective) on Mac for dmd is implemented with a bit of a hack - on ldc, it writes out to the object file, and my dmd impl just edits the command line. This is less than ideal, but functions with similar caveats to pragma(lib), which I find enormously useful, so I called it good enough and committed it here: d514f2e Might revisit it to actually emit the directive to the object file, which will work in more circumstances, but this current implementation is better than none.
The end result is using a module that requires a certain system library on Mac or ios, etc., now Just Works, the same way it just works on Windows and Linux.
If you read the whole story to this point, you saw that I promised to tell the xcode saga in detail later. Well, as my child likes to say, "now is later".
If you've done iphone stuff before, you'll perhaps laugh at me for some of this, but meh, there's a first time for everyone, so I'll go into more detail than you need. But other people might need it. I wish my own notes from 2019 had all this!
To do it:
It is possible everything I say here is different for new versions. I don't know, but hopefully you can still follow along.
At first, we're going to just use the stock hello world template, but later we'll modify it a little.
If you don't do this, you're liable to get a "failed to prepare device for development" message.
See: https://developer.apple.com/documentation/xcode/enabling-developer-mode-on-a-device
Try to deploy a stock hello world from the ide to it. It will probably fail but will unlock the next step.
See: https://stackoverflow.com/questions/61865231/invalid-code-signature-due-to-inadequate-entitlements
Add a Horizontal Stack View by dragging and dropping it under the View Controller label in the main window. Then add a Button and a Label, again from the View -> Library dialog, to this stack view.
Click on your Label, then go to View -> Inspectors -> Attributes. Here, you can change the color and size of the text and background to make it visible. You can change the font and colors of the button too. In the picture at the top of this post, I changed both to try to make it more visible, as the default was blue on black.
Now, click on the ViewController.m tab and press the + icon with tooltip "Add Editor on Right" to get a split view. Go back to the Main tab on the left.
Ctrl+Click and drag your label into the interface of the ViewController. This will let you name the variable and create a property. I called it "MyLabel".
Now, Ctrl+Click and drag your button to the ViewController interface too. For the button, you can select "Action" as the connection. Give it the name "clicked" and keep the rest of the options at the default and click "Connect".
(I'm sure there's far better ways to do this, but 99.99% noob here, so I'm just writing what I'm doing. This all took me longer to figure out than I care to admit.)
From here, you can try setting the label text on click:
// inside the @implementation ViewController thing
- (IBAction)clicked:(id)sender {
[[self MyLabel] setText: [[NSString alloc] initWithUtf8String: "clicked!"]];
}And run it. You should see the action now.
With the basic baseline set, now we're ready to insert some D.
1 module ios; 2 import core.attribute; 3 4 extern(Objective-C) { 5 6 // declare our own basic bindings 7 // these will be found in libraries soon, but 8 // had to get everything else working first... 9 10 // (actually, these are copy/paste from arsd.core, 11 // but those declarations are `package` protected 12 // until things are ready for final release. 13 // Nevertheless, if you want to sneak preview, search 14 // for NSObject in the arsd/core.d source file to see more.) 15 16 // make sure you mark all classes you just want 17 // to bind to as `extern` here, *in addition* to 18 // marking them `extern(Objective-C)`. 19 20 extern class NSObject { 21 // every extern(Objective-C) method must have 22 // a `@selector("")` annotation. The Apple 23 // documentation lists this as the method name 24 // in links 25 static NSObject alloc() @selector("alloc") { return null; } 26 NSObject init() @selector("init") { return null; } 27 } 28 29 30 alias NSUInteger = size_t; 31 32 extern class NSString : NSObject { 33 override static NSString alloc() @selector("alloc"); 34 override NSString init() @selector("init"); 35 36 NSUInteger length() @selector("length"); 37 const char* UTF8String() @selector("UTF8String"); 38 39 NSString initWithUTF8String(const scope char* str) @selector("initWithUTF8String:"); 40 } 41 42 // some bindings to the UIKit framework 43 44 extern class UIViewController : NSObject { 45 void viewDidLoad() @selector("viewDidLoad"); 46 } 47 48 extern class UIButton : NSObject { 49 50 } 51 // I don't actually know what an IBAction is 52 // this might be wrong, but binding to objective-c 53 // is fairly forgiving, you can often be incomplete 54 // and it still works. The runtime looking, stable 55 // base class ivars, and message passing paradigms 56 // let you get away with quite a bit of minimalism. 57 extern class IBAction : NSObject { 58 } 59 extern class UILabel : NSObject { 60 void setText(NSString txt) @selector("setText:"); 61 } 62 63 // and finally, bind to the base class we edited in XCode 64 65 extern class ViewControllerBase : UIViewController { 66 // declare the label property here 67 @property UILabel MyLabel() @selector("MyLabel"); 68 // the load method 69 override void viewDidLoad() @selector("viewDidLoad"); 70 // and the click method 71 // `id` in Obj-C is aka `any` and we should bind 72 // it properly, but really, it is a `void*` so we can 73 // use that here 74 IBAction clicked(void* sender) @selector("clicked:"); 75 } 76 77 // the selectors are the same as the name here most the time, 78 // but when you have arguments, there must be colons. They 79 // are generally the method name, colon, then argument names 80 // with colons. Again, the Apple documentation tends to list 81 // it as the links, but you can guess it from your own stuff 82 // a lot too. If you get it wrong, you'll get a runtime error 83 // when trying to call the method. 84 } 85 86 // now it is time to write our controller class 87 88 extern(Objective-C) 89 class ViewController : ViewControllerBase { 90 // selectors required on all extern(Objective-C) methods, 91 // including overrides where it is the same as the one 92 // you're inheriting. 93 override void viewDidLoad() @selector("viewDidLoad") { 94 super.viewDidLoad(); 95 // set an initial thing to show D overrode this 96 MyLabel.setText(NSString.alloc.initWithUTF8String("D ROX")); 97 } 98 99 override IBAction clicked(void* sender) @selector("clicked:") { 100 // now when the button is pressed, we'll change the label 101 // to prove interactivity worked 102 MyLabel.setText(NSString.alloc.initWithUTF8String("Clicked from D")); 103 } 104 } 105 106 // just fyi: 107 // of course, the extern(C) interface also works to Objective-C 108 // (this is not new, it has worked from both compilers for ages) 109 extern(C) const(char)* fromD() { 110 return "D is here"; 111 }
Please note that better library support is NOT available in this release, but is coming soon - I've already adapted arsd.core with ios support and expect simpledisplay to work (at least minimally) soon.
I'd also like to get the system framework apis bound into the release package, so they just work without defining your own things like we do here (or I do in the arsd libs), but that will probably be coming a little while later. I know Jacob Carlborg's dstep has some support for objective-c extraction, or I might do it differently. Also not that hard to just write them by hand as you need them - remember, extern(Objective-C) bindings do not need to be complete to be used. (Though their structs still generally do! Same rules as C for structs. It is classes that are forgiving.)
Anyway, let's build it.
You'll also want one or both of the OSX releases, as the iOS runtimes are included in those bundles. Get the osx-x86_64 download if you want to run the simulator on an x86 Mac, and get the osx-arm64 download if you want to run on physical iphone hardware or an arm Mac simulator. You might want both.
(It is my hope to make this easier in the future, perhaps by re-enabling the osx-universal download that bundles both together, or maybe by offering a cross compile setup wizard. But for now, you need to do the steps yourself)
You might just call it directly, e.g. when I say run opend, you run ~/Downloads/opend-latest-osx-x86_64/bin/opend, or you might add that bin directory to your PATH, or, what I do on my computer, is create a little script in my PATH
#!/bin/sh ~/Downloads/opend-latest-osx-x86_64/bin/opend "$@"
and call that. This way, updating is easy too, next time I download and unzip, the same command can call it. You can also have multiple versions live side by side, etc..
After updating, on my Mac, I open up the Apple Menu -> System Preferences -> Security and Privacy window. Have the General tab selected.
Then go to a terminal and try to run the opend ldmd2 command. It will pop up a window saying it cannot be opened because the develper cannot be verified. Hit cancel on that popup. Then look at the Security and Privacy dialog and you'll see a message saying it was blocked with a button "Allow Anyway". Click that.
Run opend ldmd2 again, and now you can click "Open" in this dialog. Keep going this until it actually works. (Might take up to three times - once for opend, then for ldmd2, then for ldc2. You can also do opend dmd if you want to authorize that too.)
You're finally ready to run it!
(If anyone knows if a reliable way to stop it from doing this and can integrate it into the CI build, please let me know, PR welcome to the opend github too.)
# if you want to target the simulator on an x86 computer $ opend ldc2 -mtriple x86_64-apple-ios -lib ios-in.d# OR if you want to target the simulator on an arm computer # or also use this command for the actual iphone hardware: $ opend ldc2 -mtriple aarch64-apple-ios -lib ios-in.d
Thanks to ldc's cross compile capabilities, you can run this on any computer. I actually ran it from my Linux box. But either way, you want to get the generated ios-in.a file over to your Mac for the next step.
# for a build on the x86 simulator $ libtool -static -o ios.a ios-in.a Downloads/opend-latest-osx-x86_64/lib-ios-x86_64/libdruntime-ldc.a# or similarly, for the real hardware/arm mac simulator $ libtool -static -o ios.a ios-in.a Downloads/opend-latest-osx-arm64/lib-ios-arm64/libdruntime-ldc.a
It may warn about some objects inside having no symbols, but this is no real problem (the library bundle just contains some unnecessary files).
You should now have an ios.a file, ready to be linked into the iphone app!
What I did was click the project navigator icon on the left (it is the leftmost icon that looks like a folder), then click on the project itself - it has the xcode icon and is the root of the tree, first item at the top of the list. This loads a big properties page on the right.
On the right, the left column has "PROJECT" and "TARGETS" headers. Select the first item under the TARGETS header. Now, on the right, is a tab sheet. Go to the General tab (which is the first one so probably preselected), then scroll down a little to "Frameworks, Libraries, and Embedded Content".
There's a grey "+" icon there. Click it to open a file dialog. I then did "Add Other", "Add Files...", and then used that dialog to get to my .a file. To get from the current directory back to your home directory, click that thing in the top middle and you can navigate to parent directories. (I told you, I am not a Mac user! This was not obvious to me the first time.)
Open your ios.a file. It should now be listed in the table above that + icon. If it doesn't show, just try it again... mine was picky about it, I don't know why, but it worked on the second try pretty consistently.
Hopefully, that works for you like it worked for me. I haven't gone much beyond this, but the groundwork ought to be laid now for enhanced library support and a better user experience with downloaded bundles and automatic scripts.
It should be possible to completely replace the generated code from the IDE template with D - we can provide our own classes (as shown here), our own extern(C) main, even our own D main that calls in. If you want to try it, start by commenting out the code in main.m in the ide and moving it to your ios-in.d file:
// added at the bottom of ios-in.d extern(C) int UIApplicationMain(int argc, const char** argv, void*, NSString appDelegateClassName); int main() { import core.runtime; return UIApplicationMain(Runtime.cArgs.tupleof, null, NSString.alloc.initWithUTF8String("AppDelegate")); }
Do the same recompile, libtool to merge druntime, then click the Play button in XCode (it will remember the filename you added before, so any updates can be retested by just pressing the button again). You should see the same results on the phone - the same code, just moved to D.
But it isn't exactly the same code - this is a D main rather than a C main, meaning druntime is now fully initialized and ready for you to use! You can now use array concatenation and other D features in your code with no trouble. (If you wanted that without moving main to D, you'd want to call Runtime.initialize() somewhere, maybe from the ViewController.viewDidLoad method or something. Or call rt_init from the main.m before UIApplicationMain, plenty of options, but having main be in D is the one I like most.)
(Please note: the photograph at the top of this blog post used the D main to enable that ~ operator. I also changed colors and font size of the label and button to make them easier to see in the picture.)
Similarly, moving the other classes to D should just work. You don't need to inherit from the ViewControllerBase, you could just come straight from UIViewController. Though, keeping the base there might be a good idea anyway to get the benefits of IDE integration, at least from the interface, then you can do the implementation in the D subclass, just like I did here. I don't have enough experience with ios programming to say what is best.
Similarly, you can create and add subviews - buttons, labels, etc. - from inside the code too. You don't need the IDE's interface builder for that, and all these function calls will work from D - you can create Objective-C objects and call their methods to put it together. Will take only a little binding effort, remember, you need only bind what you actually use with extern(Objective-C). This is something I'd like to do eventually, maybe make minigui work with the basic system widgets, but this blog post is long enough as it is, so I'll leave that as an exercise for the interested reader.
(Can you build it without xcode? I imagine xcode is calling some commands that you might be able to script too, but I don't know anything about that. If you haven't noticed by now, I'm not good at this platform lol.)
But I can say with some confidence that you should be able to make a pure D program running on the iphone with full features available. If we do hit a barrier, I'm sure we can fix it, either in the compiler or in the library.
I don't know. It depends on what they look for. I know some of the generated symbols in my implementation are not identical to what Apple's Objective-C compiler produces. This could probably be fixed with more time and attention to detail (and knowledge of llvm's api), but since it works for me, I'm happy enough with it now.
It works so it might pass, but I've heard stories of Apple failing things just for referencing deprecated apis somewhere in the binary, so I really just don't know how strict they are.
If you try it, let me know the results.
I have limited experience (the only reason I have a Mac at all is to test my libraries for other people to use, and I don't do much of anything on smartphones - I prefer laptop computers), but I feel that Objective-C isn't awful. I'm not a fan of C, but the objective parts it added on really aren't bad to me, now that I've gotten to know them. I've also heard plenty of good things about the Swift language.
Given the wide resources available from Apple and others about those languages, including the first-party IDE support in xcode and its storyboard interface builders and whatnot, it is hard to imagine somebody who is not already aboard the D train really caring about this. If you wanted to use generic D libraries, that was already an option; ldc could build for the aarch64-ios platform before this, so this new work is more about D classes just being a bit easier to be called from the existing system classes.
So, at the end of the day, realistically, I don't think this new functionality in D is a game changer.
But I still think it is kinda cool anyway, and I'm glad that I, at least, have the option for myself.
Go to https://opendlang.org/ to learn more about OpenD. We're working on improving quality-of-life issues by being open to some out-of-the-box ideas, like that things ought to just work out of the box! A lot of work to do, but we're getting there.
Next projects, as hinted to in previous text, is to continue bridging gaps in compilers and improving library support and cross compilation experiences for many common cases: iphones, Windows, webassembly, and more. No promises as to timetables though, I, and the other OpenD contributors, all tend to work in spurts in between other obligations in our lives, but some of this is already in progress.
I suggest you check back at this blog about once a month (in theory, I post once a week, but in practice I get busy and miss a lot of weeks) to see updates, or join the discord chat to see my liveblog spam if you want to watch in real time (just notice a lot of my liveblog spam doesn't actually go anywhere in real time! Let's be honest: I talk more than I work.)
OpenD doesn't do formal releases, so to get the new one, just check that CI Automatic Build link and see when the files were last updated. We try not to break things, most changes are additivie, but again, no promises at this time - in the open source tradition, THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
Next time I blog, it should be about porting druntime to webassembly, unless something cooler comes up first. So stay tuned...
In my previous post, I said I'd still like to fix some of the feature disparities between dmd and ldc in opend: in particular, when it comes to feature support on Apple computers.
Until now, there was no one compiler that worked well. dmd had a decent extern(Objective-C) implementation, but no pragma to link in frameworks and had minimal support for running on ARM hardware, like newer Macs, iPhones, etc. On the other hand, ldc had those, but its extern(Objective-C) implementation was minimal at best, giving incomplete library support.
OpenD has now solved this problem.
I'll walk you through the steps I took to get there, from figuring out the llvm glue in the ldc compiler to awkwardly exploring the xcode ide, this post should help you recreate this yourself if you want to.