In old D, there's three levels of machine checking for memory safety: @safe which means do full conservative checking to the best the compiler can, @system, which means no checks, and @trusted, which means no checks but it is allowed to be used from a @safe function. These are applied on functions as a whole, but can also be applied to nested functions, so are frequently used as one-line escape hatches inside an otherwise @safe function:
@safe void foo() { // foo is opting into strict conservative checks // this next part wasn't part of the original design // but has been consistently used in practice anyway @trusted () { // put whatever you want in here }(); } @system void bar() { // bar has no checks, but safe functions like foo cannot // call it without a trusted wrapper } @trusted void tru() { // tru has no checks, but functions like foo are allowed to // call it without warning. }
In Old D, all functions are @system unless either you explicitly mark them otherwise, or they are templates or auto return functions, which the compiler infers automatically by looking at the function body. This means that for most ordinary functions, the safe checks are *opt-in*, so if you don't think to ask for it, you get no benefit from it (and if a function you call didn't put the annotation in place, which feels especially silly when it is something like a trivial getter, it will make your *whole* function (unless you use the unofficial trusted lamdba trick) opt out of checks just to call it!
I've looked at a few options over the years. Two big ones that come up are:
Inferring all functions might work for other attributes, but it isn't really a substitute for safe by default, since if a function fails the safe test, it just silently changes the inference up to the next explicit point - and if you have no explicit point, the error just disappears, same as it does now. So it is an improvement over the status quo, since you can get some improvement with just one annotation, on main, but it still isn't really what we were promised with "SafeD" all those years ago.
Swapping the default, in some way, is necessary, even if we go forward with inferring for other attributes. But, how, exactly, should we do this?
I've written in the past about opting in on the module level using @safe:, redefining it to not apply to cases where inference or inheritance would be used. This is still a decent idea, I think (though I also kinda prefer using @safe module foo.bar; to change the default... sort of. I can go either way). But this still doesn't actually change the default; you still need to opt in, and that seems antithetical to the "correct by default" idea we're supposed to do.
Well, what if we did some combination of these? Change the default to @safe, let some modules change their own defaults, infer selectively or infer with warnings or similar?
We could change the default for main to safe, so there's an automatic anchor point for inference failures, and infer the rest. But I don't really like this; the errors can be far away and it doesn't help libraries that much. I think the default should be safe for at least most things; each exception should be individually justified. (Two exceptions that might be notable are extern functions or virtual functions, but let's come back to this later.)
Let's look at more options for inferring. Perhaps we could allow it to go one way, but not the other: system could infer to safe, but safe will never disappear. Coupled with changing the default, this means you get errors by default, but if you set a function to system, it could be used from safe if it followed the rules anyway. On the other hand, what if it was marked system because it didn't strictly speaking break the rules in a way the compiler detected, but you're warning it is dangerous anyway? (Though perhaps this is abusing the language where there should be a generic, user-defined effect system.) I don't think this delivers real value.
Perhaps @trusted functions use inference for something? But the whole point of trusted functions right now is you know you're breaking the rules and want to silence the error. So this seems unlikely to deliver real value either.
For the record though, I talked about making @trusted still warn when an unsafe thing happened unless you silenced it in that specific line with a local block. This could use inference to help avoid spurious warnings about calling a function marked system, but follows the safe rules anyway.
But it didn't work well because the @trusted lamdba escape pattern - while technically non-conforming to the rules - is very common in existing code and it just spewed warnings. While we could go through and fix those, I am just not convinced it is worth the time, at least not until other things are in place.
What about the escapes? Do we just keep the @trusted system? Or do we transition to some kind of __unsafe {} blocks, given the presence of @trusted lambda escapes in the wild?
On @trusted escapes: you're not supposed to do this - the trusted annotation is supposed to indicate that the *entire* domain of the trusted block is @safe, just the compiler can't prove it, so it trusts you to.
What is the domain of a function? You might remember from high school math that given a function f(x), it is all values of x where f is defined. In computer programming, the definition is similar: it still talks about what arguments it can take, but a bit different because we need to remember that there might be some hidden "arguments", like this and global variables. Or, for nested functions, the local variables are in scope too, so they are part of the nested function's domain.
With the high school math function, the domain of f(x) is often relatively simple to determine. But with the nested lambda, it can be very complicated since you need to consider multiple layers of scopes worth of variables! Top-level trusted functions are somewhere in the middle - not as easy as high school math (which I should note is not exactly "easy", some of those are quite tricky!) but not as complicated as the nested function, because there's at least no local variable state to consider.
Anyway, that's why you're not supposed to do it. But in practice, it is done anyway. Searching the OpenD bundled libraries, @trusted appears 3,623 times. At least 169 of them are the trusted lambda pattern. @safe appears 10,154 times (3,916 of which are unittests, interestingly enough).
On the one hand, that's like 5% of the usages, so maybe the system is kinda working. But on the other hand, it is at least 169 times where people thought it was a good idea. We also see some idea of unsafe blocks in many other programming languages. Indeed, I'd argue the same logic against @trusted lambdas applies to all @trusted functions - are you sure you checked the whole domain including global variables and this object states? If you can be sure about that, yes, it is multiplied by the number of local var states too, but I think both cases are easier said than done anyway. And the @trusted lambda (or unsafe block) still has the compiler checking certain things in other parts of the function.
So... I'm not completely convinced it must change, but I certainly lean in that direction. Nevertheless, at this point, I haven't yet tried to change any implementation on this topic.
But if we do decide to embrace this pattern, it'd probably look something like:
@safe void foo() { __trusted bar(); // or __trusted { bar(); } }
The syntax would be like other D blocks like if and debug, for one-liners, it is minimal punctuation, but you can also use {} for multiple statements under it. It would create a new scope for variables under it, just like if, etc. The name of the keyword is undetermined.
This blog post is behind schedule and rewritten a couple times because every time I think of an idea, I actually try it. I have had some success enabling inference. Thanks to some help from others, I also had some success flipping the default. The implementation in the compiler seems not only doable, but reasonably small.
The hard part is updating existing library code to be compliant with the new rules. In my current draft implementation, which you can preview here: https://github.com/opendlang/opend/tree/safe , I put in some hacks to exclude druntime from the safe-by-default rule. Realistically, druntime is probably an exception anyway; a runtime library is often system or trusted code by its very nature. But we probably want something other than weird hacks; what's good for the goose ought to be good for the gander.
This implementation also has no special case for extern or virtual functions. Virtual functions can probably be safe by default like others (though this is not without cost! A @safe function can always override a @system function (though then it can't call super without being @trusted, since the normal rule of safe functions cannot call system functions does apply), but not vice versa. Meaning making interfaces/base classes safe by default limits what the child classes can do. This is probably acceptable, just worth noting), but extern functions should almost certainly be assumed to be @system. The implementation sidesteps this right now because most extern function definitions here are in druntime, and thus get the special case of system default from being in that package. But it should still be done.
Nevertheless, this basic implementation has given me some real world experience in using it already... and it hasn't been fun yet. A lot of library fixup work is going to be necessary, especially in my arsd libs, much of which interfaces with ported C code or operating system functions.
But... I hope it is worth it. I'm still not completely convinced. To be honest, my decision might just be to say the original vision of SafeD isn't worth it at all, and the solution will be to promote certain individual checks to default while leaving the others are opt-in.
For example, my main driver I have for safe by default is not banning pointer arithmetic; that's enough of a hassle, I already take some care with it. Safe by default is not necessary for bounds checks, since they're there in all cases anyway until you opt out. What I want is the "reference to local variable escapes" check to be enabled by default. I'm not even sure there are other specific checks I want right now. So maybe that escape check should be enabled by default and leave the rest of @safe how it is.
Of course, you still need a way to tell the compiler "yes, I know this looks wrong, but trust me, it is fine" even in these cases. So some kind of @trusted or __unsafe thing might make sense there anyway. But if we can get 80% of the benefit for 20% of the cost... maybe worth it.
Nothing is set in stone yet. We have a few promising directions, and we're actively experimenting with them. If you want to grab that safe branch and experiment with it too, please do! We want to do the right thing in theory, but also need to make it realistic in practice.
Of course, we also don't want to do the easy, lazy thing in practice and completely ignore the other benefits! So it is a balancing act, and I don't have all the answers here.
So we'll see what happens. But we're definitely doing something.
FYI: Language docs are also in the works. I'll hopefully have a bigger update on that and the website in general next week.
@safe is a bizarre spot in the D language. We've been saying for almost a decade that if it is going to be there, it should be the default, but this has never actually happened. Time for OpenD to take action.