Android project update, introduction to arsd.jni

Posted 2019-12-09

Android code works - big thanks to other people for doing the work in ldc as I'm building on that. Also the JNI stuff I think may be generally useful for D.

Core D Development Statistics

In the community

Community announcements

See more at the announce forum.

What Adam is working on

Android project update

I made some solid progress on the Android project last weekend, writing three new programs to help out:

  • android-dub-build wraps dub build in such a way to generate .so libraries you can put in an existing Android project to use on the various platforms Android supports.
  • android-ldc wraps the ldc2 compiler to build the library, using the native toolchain linker from Google. It is called by android-dub-build.
  • arsd.jni is my new module to make interfacing with Java easier. More on this in a bit. PS you might notice my new docs coming together from http://code.dlang.org/packages/arsd-official too, try the Documentation link.

The first two programs can be found here: https://github.com/adamdruppe/d_android

I started by creating a project (called "omg" lol) in Android Studio and selected the C++ icon for project type. It spat out something like this:

1 
2 package com.example.omg;
3 
4 import androidx.appcompat.app.AppCompatActivity;
5 
6 import android.os.Bundle;
7 import android.widget.TextView;
8 
9 public class MainActivity extends AppCompatActivity {
10 
11     // Used to load the 'native-lib' library on application startup.
12     static {
13         System.loadLibrary("dubtest");
14     }
15 
16     @Override
17     protected void onCreate(Bundle savedInstanceState) {
18         super.onCreate(savedInstanceState);
19         setContentView(R.layout.activity_main);
20 
21         // Example of a call to a native method
22         TextView tv = findViewById(R.id.sample_text);
23         tv.setText(stringFromJNI() + " D said");
24     }
25 
26     /**
27      * A native method that is implemented by the 'native-lib' native library,
28      * which is packaged with this application.
29      */
30     public native String stringFromJNI();
31 }

You can see there that the library is called dubtest to be compatible with the dub package in that repo.

Then I took my program:

1 import arsd.jni;
2 
3 /// our implementations of the native methods in Java
4 final class MainActivity : JavaClass!("com.example.omg", MainActivity) {
5         @Export wstring stringFromJNI() {
6                 return "Hello, this is D!"w;
7         }
8 }
9 
10 
11 // hack for tls
12 void main() {}
13 
14 // hack for linker on android (idk exactly, prolly a macro or something. maybe can fix in druntime later)
15 extern(C) void atexit() {}

And after compiling the android-ldc and android-dub-build helpers in the repo, and building the D runtimes (see the android-setup.d program's comments for how), I could run:

$ NDK=/home/me/Android/ndk/android-ndk-r20/ ../android-dub-build ../../omg/app/src/main/

then went back into the Android Studio and hit build after deleting the C++ part it auto-generated; don't need that anymore. And there we go, hello!

See the tips at the bottom of https://github.com/adamdruppe/d_android for some of the problems I saw in the process.

Still to come:

  • Finish my script for building the runtime libraries to make it easy for any user, perhaps just offering binary package downloads.
  • Bindings to the Android NDK and possibly select Java classes, with a process documented to regenerate these as needed. Should be basically as simple as running dstep over it.
  • Wrapping it all up for users! At first, I wanted to put the bindings in core.sys.android.*, but now I think I will just put the in a dub package since I'm using it to build anyway. Of course, I'll document the principles so you can use other systems just as well.

My JNI module

Part of the Android project is the Java Native Interface, which is accessible from D the same as C, of course, but I wanted to make it easier. The result is coming together as arsd.jni. If you click that link it will take you to the documentation which has a working example as of now.

There's also a working example in my d_android repo under the android-dub-test directory, which has stuff like above.

Here though, I want to talk a little bit about the implementation so far and the future directions.

It uses a pattern you have seen me talk about before if you are a long time reader: the curiously-recurring template pattern. You write a class that inherits from my base class, and pass your own class as its template argument. C++ sometimes uses this for devirtualization, but in D, I like to use it primarily for reflection.

So, the base class iterates over the methods in there and finds ones with the Import and Export UDAs (which are actually members of the base class, to keep it nicely scoped). When it sees Export, it generates a wrapper function with the appropriate extern(C) decoration for JNI to call.

This wrapper translates the Java data, using slicing whenever possible (hence all the warnings about lifetimes in my docs!), and passes it to an instance of the class that it manages. Then it takes the return data and translates it back for Java. All the while, it uses try/catch to get D exceptions and rethrow them on the Java side.

No need for extra registration steps since the CRTP reflection and see it there and inject necessary code.

I'm not quite done with it yet, but strings (and wstrings - more efficient) as well as value type primitives already work. Arrays coming soon.

The other side is Import. I have only implemented a proof-of-concept so far, but it should be pretty straightforward from here, I just need to find the time. This takes unimplemented functions and actually implements them separately by using pragma(mangle) hacks.

It is funny - people have asked me if such separate definition and declaration is possible in D, and I usually say "yes, but don't try it". And here I am doing it! In my defense, the only reason I'm actually doing it here is that I'm generating from reflection, and thus have some guarantee the signature isn't going to change without me noticing. (That's why I don't recommend it - you are responsible for keeping it in sync with zero help from the compiler. And it relies on ABI details that can get scary like where the this pointer is passed...)

Anyway, the body of that function then just translates arguments, calls the Java function it represents (this is the part I'm not finished with yet), and then translates the return value. Given unknown lifetimes and the impossibility of maintaining abstraction while making the user free the memory, I'll probably have to always copy things, no slices here... but it should work anyway.

The other open question I have is runtime initialization. Right now I have functions in the library to do it but I might change that later. Also gotta decide on making a Java VM for the case of D main calling Java libraries, but it should be doable.

Conclusion

It has taking me much more time than I had planned, but it is coming together pretty nicely. I have never used JNI before and it is nicer than I thought! Similarly, I have never used Android so that is all new too... but at least I finally have my code running, so now more fun to work on a nice api.

I'm hoping to get this good enough by the end of the year, and that looks possible now. D on Android is easy in termux and will soon be easy on desktop cross compiling too.