February 2025 update - outdent heredoc strings, preview no more, arsd.rtf and more

OpenD features

OpenD's feature changes tend to be pretty conservative, and while we've been working on some bigger things in the background (in particular, error message fixes and if(auto x; tryGet(x)) hopefully coming soon), since the last post, we've focused on making executive decisions on preview switches, and also a use case came up for improving heredoc strings in a way that is a grammar change, but otherwise compatible with old versions.

Previews enabled by default

D has had a problem with things staying as "previews" for a great many years, and OpenD aims to solve this by simply making executive decisions on each one: enable by default or drop support. (Note that dropping support doesn't necessarily mean removal, but it does mean you shouldn't rely on it at all since we won't make effort to keep it either.)

We might still change our minds if it fails more in practice, but OpenD has now enabled:

• preview=fixAliasThis which defines an obscure edge case in alias this which I think was simply buggy before anyway
• preview=rvaluerefparam I don't think this is especially valuable, but since the minor breakage it caused was fairly easy to fix (arsd.database.DatabaseDatum.to!int became ambiguous matching both ref string and opCast, but I could version out the opCast since that was only there for compatibility with std.conv.to in the first place) and people have requested it, we decided to proceed with enabling it.
• preview=fixImmutableConv seemed like a straightforward bug fix with no breakage in anything we've tried.
• preview=systemVariables has been enabled in OpenD already for some time anyway
• preview=fieldwise again seems like a straightforward bug fix that didn't break anything we've tried so far.
• preview=inclusiveincontracts since it seems to align with people's expectations and caused no breakage in tests, even for people who followed an old tip of mine.

We are likely to abandon (but reserve the right to change our minds for a little bit longer):

• preview=nosharedaccess since I don't see where this is going.
• preview=in since it seems likely to bring breakage, despite not being a bad idea.
• preview=dip1000 since it brings risk of introducing new memory bugs while being too onerous to bring existing code into compliance adequately to show its errors by default anyway.
• preview=dip1008 since it is liable to cause silent memory corruption.
• preview=bitfields since it is a poor design.

Outdenting heredoc strings

D has had a feature for as long as I can remember called "heredoc strings". They look something like this:

string s = q"some_user_defined_thing
	content here
	of the string
	that can span multiple
	lines
some_user_defined_thing";

Old D required that the closing identifier be at the beginning of a line. This was illegal:

string s = q"ident
	stuff
	ident";

And now that's allowed, with "outdenting" rules similar to C#'s triple quoted string https://learn.microsoft.com/en-us/dotnet/csharp/language-reference/tokens/raw-string .

That is, the whitespace before the closing ident is stripped from each line in the string itself, and the final newline is removed the same as the first. Meaning, there, s == "stuff". You can format a block of embedded string in your code in such a way that it looks properly embedded, but works basically like you expect if it wasn't.

(so to be honest, I kinda wish the ending thing was one indentation level removed from the others, but that'd mean the compiler would have to guess which indentation method was used. Obviously, the only sane answer is one tab, but there's people who resist this clear, unambiguous truth, so we decided to avoid the holy war in the compiler and just copy the *exact* sequence at the terminator, same as C#.)

For existing D code, since indenting the terminator is illegal, they will always be at start of line, and thus have no outdent to remove. A consistent, simple rule that is also backward compatible! (Though those will have the final \n character on the string too - that is stripped only if the terminator is also indented. So not perfectly simple but close enough.)

Moreover, the ident string can be used to tag the embedded language for tools like syntax highlighters. The D compiler doesn't care what that identifier is and ignores it - all it cares about is that the first and last one match - but you might configure your edits to recognize things like

string s = q"html

	html";

and similar to syntax highlight the embedded language as well, similar to the language tag in markdown code blocks.

This change will not be recognized by third party lexers, but it is fairly easy to fix - simply remove the requirement that the ident be at the front of a line.

In adrdox, the diff is:

++ b/Dscanner/libdparse/src/dparse/lexer.d
@@ -1557,7 +1557,7 @@ private pure nothrow @safe:
             error("Newline expected");
         while (!(range.index >= range.bytes.length))
         {
-            if (isNewline())
           if (true || isNewline()) // OpenD allows these to end anywhere
             {

that's inside lexHeredocString)

You can fix vim syntax by deleting the ^ character from this line in d.vim: syn region dHereString start=+q"\z(\I\i*\)\n+ end=+^\z1"+ contains=@Spell.

I haven't looked at others, but I imagine they're similar.

This is also a breaking change for user code, since old code may have used the identifier followed by the close quote in the middle of a string, but I find it fairly unlikely. And if you do, it isn't a hard fix. So I'm not concerned about it. (tbh the requirement to close the thing at the start of a line made me simply never use this feature since it defiled the beauty of the source code's indentation lines.)

Please note that OpenD currently still does not allow iq"thing interpolated heredoc expressions. We might enable that later, as I expect these heredoc strings will be more used now than before.

Error message pulled from upstream

We also pulled an error message enhancement from upstream for opDispatch.

struct S {
        void opDispatch(string name)() if(name == "foo") {
                x = 5;
        }
}

void main() {
        S s;
        s.foo;
        s.bar;
}

Old:

opd.d(9): Error: no property foo for s of type opd.S
opd.d(9):        potentially malformed opDispatch. Use an explicit instantiation to get a better error message
opd.d(1):        struct S defined here
opd.d(10): Error: no property bar for s of type opd.S
opd.d(10):        potentially malformed opDispatch. Use an explicit instantiation to get a better error message
opd.d(1):        struct S defined here

New:

opd.d(9): Error: no property foo for s of type opd.S
opd.d(3): Error: undefined identifier x
opd.d(9): Error: template instance opd.S.opDispatch!"foo" error instantiating opd.d(10): Error: no property bar for s of type opd.S
opd.d(10): Error: template instance opd.S.opDispatch!"bar" does not match template declaration opDispatch(string name)()
  with name = "bar"
  must satisfy the following constraint:
       name == "foo"

arsd library changes

New modules

We added six new modules for common file formats over the last couple months.

First, 0xEAB contributed arsd.ini, for reading the common .ini configuration file format. The module tries to be configurable for reading several common variants of the file format.

Second, I wanted enough support for rich text format (.rtf) files to load a file I wrote up myself, so I wrote arsd.rtf.

Then, I got frustrated at the barely usable spreadsheet program on my Linux box and set out add support for loading some files emailed to me, so I also made support for Word, Excel, and PowerPoint files - again, at least the minimal support to get a good look at my email attachments, while fitting inside my limited time budget.

This led to four more modules: arsd.zip, which is a very thin wrapper on Phobos's std.zip for now but I want flexibility to swap out both interface and implementation to better mesh with the rest of my libs over time, then arsd.docx, arsd.xlsx, and arsd.pptx using it.

You'll find all those modules are extremely minimal and likely exceedingly buggy in general, but they're already useful to me, and provide a base for me to expand later.

Updated modules

arsd.pixmappaint got a ~1000 line update from its author, 0xEAB, including an upscaler/downscaler, fixed point math, more documentation, and several other fixes.

arsd.archive got several bug fixes.

arsd.cgi now lets you listen on scgi and http, even in the default build. When using one of the provided GenericMain, DispatcherMain, etc., mixins, the command line processor will check for --listen=protocol://address:port. http and scgi are compiled in the standard build and can be used at-will to match your deployment.

The standard build will now also print out a message to stdout explaining this when used outside a cgi context and no command line arguments.

(please note, if you are a dub user, you are using a non-standard, build configuration, so this may not apply to you. a standard build of any arsd library is done with opend yourprogram_that_imports_it.)

arsd.color now has a method Point.fromLinearOffset.

arsd.core is starting to compile on Emscripten, but this isn't complete yet, and gained some more internal helper functions like reinterpretCast, isSliceOf, getCurrentWorkingDirectory, and types like NonOverflowingInt and enhancements to FilePath to work better with relative paths and such.

Its logging framework is also nearly usable, and its writeln handles more types than before.

Remember, you can use arsd.core directly if you want, but it has no public api compatibility guarantees. It is there to support upstreamed arsd libs, not your stuff. Nevertheless, I try to keep it compatible, but anything here is really more of a preview to future modules rather than something you should use yourself.

arsd.discord now tries to reconnect in a loop to increase unattended uptime. It still isn't quite where I want it, but better than it was.

arsd.dom got improved support for processing documents as streaming dom fragments instead of as a whole tree processed up front. You can pass chunks of a file to it at a time and process data as it happens, reducing total memory usage (and with it, GC scan time) in some cases.

arsd.email got a new convenience function addFileAsAttachment to load a file for you and arsd.http2 got better error messages for connection problems.

arsd.minigui got a bunch of changes. One is a small breaking change (the next tag might be 12.0 due to a few of these): all the defaultEventHandler_x methods now use the typed event instead of generic Event. When you override those, you'll need to update your code to use the new the more specific types.

It also now has some new context menu things like createContextMenuFromAnnotatedCode and several other convenience accessors to existing functionality.

minigui's combox box and menu were overhauled in its custom widgets implementation. They now work just much better. Additionally, the ListWidget on custom widgets (so the linux version basically) is written in terms of the GenericListViewWidget, giving it more potential for improvement in the future and finally clearing the way for me to deprecate the old, broken ScrollableWidget.

Mouse capturing in minigui can now be done by a child of the currently capturing widget, allowing more encapsulation. I still plan to overhaul keyboard focus similarly as well, but haven't gotten around to it yet.

The helper template, callAsIfClickedFromMenu, lets you trigger an auto-wrapped menu command as if the user called it, including triggering an automatic dialog. This lets you do things like call menu.open(args[1]); when invoked from the command line` with little fuss.

minigui toolbars can now belong to sections. You give it a section name when annotating the function and the sections are grouped together on screen, with a slight separation between each one. The details may change later.

minigui status bars can be proportioned up easier with a single function call: toolbar.setSizes(...).

minigui's Button class now allows for more customizations on Windows, adding support for owner-drawing. This means background colors, font selections, and some more details can be swapped out while keeping the standard behavior.

minigui's CustomTexEdit classes got improvements to shift+page up and down, preliminary support for image embedding.

minigui's dialogs now explicitly associate with their parent widgets, allowing for more intelligent placement by the window manager.

minigui's file picker dialog got a log of changes. It now shows the active filter and current directory as gui elements, lists files in "natural sort" order, has shortcuts to common directories, and retains the previous tab complete features as a gateway into those elements.

It also has a FileDialogDelegate interface to allow for further customization, such as file preview panes. Additionally, the file filters are more structured internally, though the api retains the Windows list of nul-separated strings for now. That will likely become an overload option soon.

Please note that on Windows, it still uses the standard system dialog.

arsd.terminal got a few small regression fixes in its cursor tracking for minimal processing objects (something you probably never used).

arsd.libssh2 and arsd.terminalemulator got small bug fixes related to an update in the upstream libssh2 library. I now use 64 bit builds for it on Windows.

arsd.textlayouter got initial, experimental support for inline embedded objects and fixed a bug in its application of tabstops. I plan to add support for elastic tabstops, easier line counting, and more paragraph formatting soon. I might also add some kind of inline table display support so it can load and edit more rtf files.

Getting OpenD

The arsd libraries and opend compiler are all bundled together, go to https://opendlang.org/ and hit get started now to get a link to the latest build to download.