Idea: user-extensible effect attributes

Posted 2022-08-15

What if we could define @nogc, pure, and even things like @vibe_fiber_safe in library code instead of compiler additions? Discussion to follow.

Core D Development Statistics

In the community

Community announcements

See more at the announce forum.

Effect system discussion

One of the (many) criticisms I have against @nogc is that it is a special case for one thing you might want to avoid without addressing other things that can be problems in similar situations. I've long thought that some kind of user-defined system would be nice, so it could potentially check multiple factors and maybe even be combined into a set, so you don't repeat a dozen attributes on each function. I remember at one point in the past, Andrei Alexandrescu was pondering adding computation complexity annotations (e.g. O(n) notes) to functions and interfaces, which might also be related.

D's built-in annotations tend to come in in the form disallowing operations from a list. For example, @nogc functions are not allowed to call anything that might call into a GC function, and pure functions are not allowed to read static variables. When it bans calling functions not similarly annotated, it is just a shortcut for performing this test down the call chain.

I can imagine a system where you mark what a function does, then can mark other functions with things they're not allowed to do. Language built-ins would have some core.attribute magic notes saying what they do, but you could just as well define your own too.

I described this to Paul Backus on the chatroom and he said it sounds like an effect system and linked this blog: https://www.stephendiehl.com/posts/exotic03.html. After reading it, I agree, this is an effect system! So you might read that or related literature to get a broader view.

Of course, if you can list what something isn't allowed to do, it also makes sense to list what it is allowed to do. There's two ways to treat this allowed list: either as something akin to D's @trusted, which allows it but hides the effect from the rest of the check system, or where you can ONLY call the things on the list, and anything else is an error. That kind of strict allow-only-this list would be quite difficult to use in practice though, since you'd likely prohibit too much, but this can sometimes be useful too, since strictly allowing only the properly-annotated effects of functions ensures something doesn't slip by just because its effect was never marked.

This leads me to four types of annotation: @effect!(...), @denied_effect!(...), @exclusively_allowed_effect!(...), and @allowed_hidden_effect!(...).

Most D's current annotations conflate the @effect with the @denied_effect; we don't say, for example, "this function accesses global data", we instead just say "not pure". This is practical for a lot of cases, but I think it is better in general to separate them. One example where the separation is useful would be something like @effect(writes_to_console). You might want to prohibit that in some functions, but just because a function writes to console doesn't mean it can ONLY call other functions which do too, like how pure can only call other pure functions.

Moreover, I think the @effect() would really need to be inferred up the call chain to be particularly useful. Or, equivalently, any check for it recursively scans the whole call chain looking for it. Whereas the allow and deny things are just proxies for what effects it actually has, and do thus not need to be propagated through the whole chain.

I would mark the effects with D symbols. This allows for namespace disambiguation, etc., through the normal module system. Let's take a look at some hypothetical code:

// this is a user-defined effect
enum writes_to_console;

// declaring a user-defined effect on a function
void writeln(string s) @effect!writes_to_console { ... }

// accessing an effect out of the compiler
import core.effects : gc_allocates;

// the built-in nogc annotation can thus become a library alias
alias nogc = denied_effect!gc_allocates;

import std.typecons : AliasSeq;

// you could also declare combined things!

alias no_combined = AliasSeq!(denied_effect!gc_allocates, denied_effect!writes_to_console);

void cant_write_to_console_nor_gc_allocate() @no_combined {
	/*
	compile error:
	writeln has effect writes_to_console, yet this effect is denied
	*/
	writeln("hi");
}

If a function was marked with directly contradictory things, that'd be a compile error:

enum thing;

// this is obviously impossible
void contradiction() @denied_effect!thing @exclusively_allowed_effect!thing {}

You can use the @allowed_hidden_effect to simulate something like built-in trusted:

enum thing;

void foo() @effect!thing {}

/*
	Since this calls foo(), it also has the same effects as foo().
*/
void inferred_effect() {
	foo();
}

/*
	This function calls foo, but since it is an allowed hidden
	effect, the `effect!thing` is hidden!
*/
void baz() @allowed_hidden_effect!thing {
	foo();
}

void error() @denied_effect!thing {
	/*
		error, thing is a denied effect, yet it happens
		in inferred_effect (from a call to foo one layer deep)
	*/
	inferred_effect();
}

void bar() @denied_effect!thing {
	/*
		But no error here, since baz allowed the hidden
		effect.
	*/
	baz();
}

And yes, I would want the error message to tell you what the source of the effect is, even a few layers deep. I've asked for this on existing @nogc and @safe, nothrow, etc.

Hiding effects like this would be generally discouraged - there's other ways you can write things like this - but it can be useful just like @trusted or debug escapes, just user defined.

Allowed and denied effects would have to work with inheritance. Subclasses can deny things the parent allowed, but can not allow things the parent denied, just like how you can add @nogc to an override but cannot remove it.

Inheritance of effects would be interesting because an overridden method can not add effects that weren't present on the parent. Doing this would break the interface - calling through there would allow things that are supposed to be denied. A subclass would essentially implicitly have @denied_effect!all_things_not_declared_as_an_effect_on_the_parent. It would be very tricky to do this while maintaining compatibility with existing code. I'd propose making an exception for classes that don't mention any explicit effect at all - let the code be written and compile, but a virtual function would have to assume all possible effects unless limitations are listed. Dynamic dispatch is just at odds with static checks by its very nature. But this is one example of where allowed_hidden_effect can provide a practically valuable escape hatch.

Finally, just like with the built in attributes, we'd also want some kind of dependent effects on arguments. This can be determined by the compiler's check outside the formal type system, or you could declare something like:

// forward the effect with relatively normal syntax, just the magic
// is it isn't statically determined at the declaration, but rather
// at the use.
void map(void delegate() dg) @effect!(__traits(getEffectsOfArgument, dg)) {
	dg();
}

I've spent most this time talking about replacing built-in things for ease of relating existing knowledge, but I think the real value would be things like alias fiber_compatible = @denied_effect!blocking for those kind of strict vibe.d style event handlers. But the main idea is if it was user-definable, you can do whatever you needed to do.