mutableRefInit, defaultStackPages, other small bug fixes, shipping experimental opUDAOn, arsd.shell discussion

OpenD recent changes

• druntime Fiber defaultStackPages changed from 4 or 8 up to 128. This makes stack overflow crashes less likely at almost zero cost since the stack is zero allocated and not actually committed until used.
• mutableRefInit is now deprecation instead of error to ease migration efforts. i want to make it less sensitive to false positives.
• a few small bug fix patches pulled from upstream, including a one-liner by dennis using the awful claude bullshit. I guess this makes opend ai slop now too but the were ordinary bug fix patches so im ok with the result for the ones i merged.
• Experimentally shipping Rikki Cattermole's opUDAOn implementation.
• opend command line program has some pieces to make it easier to test in the future with a dev build. works for me but not great for others yet.

D shell

I have written three applications in the last several months that I'd like to talk about. I'm way behind on writing but let's do one at a time here.

First, back in October, I started writing arsd.shell, and its user application, deesh. There's three main goals to this: 1) be embeddable in applications, such as build tools, 2) to work as a general unix shell ui with my stuff, and 3) to provide self-contained functionality with my ui on Windows as well.

I pretty quickly got the core working, though then it was a bit of a pain to get signal handling and job control right, and I still haven't implemented all the nuances.

User overview

First, let me post the code to the ui wrapper. You'll want the arsd git master available for best results, though the version bundled with opend will also work too.

import arsd.terminal;
import arsd.shell;
import arsd.file;

void main() {
        auto terminal = Terminal(ConsoleOutputType.linear);

        if(terminal.stdinIsTerminal) {
                enableInteractiveShell();
        }

        terminal.lineGetter = new FileLineGetter(&terminal, "deesh_history");
        auto shell = new Shell();
        string line;

        try {
                shell.executeScript(readTextFile("deeshrc"));
        } catch(Exception e) {
                import arsd.core; writeln("In deeshrc: ", e.msg);
        }

        again: try
        while((line = terminal.getline(shell.prompt)) !is null) {
                terminal.flush();
                auto jobToForeground = shell.executeInteractiveCommand(line);
                if(jobToForeground)
                        jobToForeground.call();
                if(shell.exitRequested)
                        break;
        }
        catch(UserInterruptionException e) {
                terminal.writeln("^C");
                goto again;
        }
}

You can see that the library tries to wrap most things up, and between it and arsd.terminal's pre-wrapped functions, it didn't take too much code to get this result operational, and it behaves reasonably well - Terminal.getline gives quite a bit of ui niceties out-of-the-box.

Now, let's take a look inside.

The function enableInteractiveShell is basically a copy/paste out of the bash manual. On Windows, it is pretty easy - juset set the ctrl+c handler so the key event comes to us normally. On Posix, it is more involved: it first waits until it is the foreground job (when the parent shell creates the process, there might be a race for them to assign terminal foreground to us, so it is a signal loop), then ignore a bunch of the control signals the kernel sends to the foreground group for things like job control. Since we are the shell, we want to exempt ourselves from the automatic behavior and get in our own group so we can manage our children.

The FileLineGetter is a built-in generic thing from arsd.terminal, which provides tab completion for filenames in the input line and saves to a history file - the deesh_history filename here. It isn't especially good for this job, but works for now. Ideally, I'd write a new LineGetter here that understands shell expansion, built in keywords, etc., but I haven't gotten around to it yet (though did try to write the library to make this relatively easy).

Now, it creates a Shell object. This is a high-level wrapper of the whole implementation, written such that you can replace various parts of the OS interface. It currently assumes, in the default constructor, you want a pretty functional OS shell, but I'll probably make that an argument at some point.

The high-level Shell object

What it does now is:

this() {
        setCommandExecutors([
                // providing for, set, export, cd, etc
                cast(Shell.CommandExecutor) new ShellControlExecutor(),
                // runs external programs
                cast(Shell.CommandExecutor) new ExternalCommandExecutor(),
                // runs built-in simplified versions of some common commands
                cast(Shell.CommandExecutor) new CoreutilFallbackExecutor()
        ]);

        context.getEnvironmentVariable = toDelegate(&getEnvironmentVariable);
        context.getUserHome = toDelegate(&getUserHome);

        context.scriptArgs = ["one 1", "two 2", "three 3"];
}

Which enables this, but notice the delegates for getting OS info and the list of command executors, which here includes external commands, but does not have to - you could use the same system with a restricted command set, if you wanted.

Of course, the script args there are still placeholders! There's several other methods available for overriding functionality like the prompt and glob algorithm too.

The other public, high-level method used here is executeInteractiveCommand, which does a lot of internal work and returns a Fiber subclass, which represents a command in progress, possibly suspended by shell job control. Hence, it returning a job we can .call to proceed.

So, let's look further inside. What happens in these methods?

Implementation overview

I started the blog with the top down, but I wrote the implementation from the bottom up. Starting with the thought that shell is a programming language, I first wrote a lexer, then parser, then semantic expansion and conversion to some kind of intermediate representation, and finally, an interpreter to execute that representation. Shell is a messy language, and it didn't quite come out clean, but still kinda works.

First, it calls lexShellCommandLine, which tries to understand some word separation and quoting rules, without actually applying them yet. It returns a struct with the content and quote rule, and you can expand variables in it using a runtime context object. Since expansions may make multiple arguments, actually applying this to get a result can get pretty involved. The unittests try go cover step by step to reduce the pain in understanding this.

Next, there's parseShellCommand. This uses a helper function called nextComponent which tries to turn those lexemes into actual argument groupings, and the parser turns these into ShellCommand objects. The parser needs a context and a globber to help expand those arguments, but aside from them, it is almost pure; it just reads its arguments and returns a description, it doesn't do anything yet.

The ShellCommand object describes what it needs to do, and has some private info to help track actually running it. It has the stdio fd info (which might be itself a command, to describe a pipe), the argument strings, set variables for this command, and how it terminated - stuff like & for if you want it run in the background. It is possible to print this command back out without running it and I provide the Shell.dumpCommand method to help do this, which helps for debugging. You might also create ShellCommand objects as the api instead of parsing shell command strings (though you'd perhaps be better just creating the processes yourself too).

Finally, once it is time to run, it does a call to startCommand(command) which kicks off a whole other process.

Mechanics of running commands

We saw earlier that there's a list of Shell.CommandExecutors. These are used to actually run those ShellCommand things: it goes down the list and calls executor.matches(arg0) and if it returns yes, it then immediately calls the executor's startCommand method. It can also return no, meaning move on to the next executor, or yesIfSearchSucceeds, in which case it calls executor.searchPathForCommand to make the decision. This design is not exactly elegant; only the external executor searches the path, but meh, it works for me. For now at least.

After deciding this executor is responsible for the given command, it actually runs it. This process is also pretty convoluted, and I did it three different ways:

• The ShellControlExecutor does built-in commands that modify the shell's own state, and some modify the process state too (like changing the current directory). I did this as an associative array of delegates.
• The ExternalCommandExecutor runs commands, like you expect from a shell. Most of this is delegate to arsd.core.ExternalProcess, which needed some extension to support the shell. (I originally wrote that with the assumption that it'd be only used for in-memory data, but it wasn't too hard to extend.) It sets up environment variables, pipes, signal handlers, and process groups so things work. This was kinda tricky, but also normal unix shell stuff, so nothing novel here. The Windows version is again much simpler, still pipes, but ExternalProcess does that already.
• Finally, the CoreutilFallbackExecutor is something I wrote to support the goal of self-contained similar functionality on Windows: it emulates basic functions you expect in a unix-style shell. I implemented this as a little reflection-based thing that provides some functions like writeln which writes to the shell output instead of the process's top-level stdout, and pulls names and args off function signatures. This uses arsd.cli, a fairly new module that converts argv string arrays to function parameters:

  void rm(bool R, string[] files) {
  }

  That understands rm -R a b to be rm(true, ["a", "b"]). Each command here is run in a helper thread so they can pipe to each other with a simple implementation.

Each executor has some in-process state to support job control. This is wrapped up in a fiber to make the top-level thing a bit easier to understand, which brings us back to the demo program at the start of this blog.

There's still a bunch of little details that need work, but I'm actually using this, on Windows, for myself, so it kinda works. It hasn't replaced bash for me yet though on linux, but perhaps someday.

I still want to write more documentation and stable interfaces, but I keep getting distracted with other things. I'm not in a huge rush right now.

At some point in the future, I'll also write about my sidescroller game and my audio player with NES/SNES/Sega/Playstation format support. I'm also supposed to release arsd 13 pretty soon.