dpldocs.info cross-package search finally released! and more terminal getline enhancements

Posted 2020-11-30

I've been talking about a cross-package documentation search for a couple years now and my first attempt didn't work very well, so it went on hold again. Last week, I wrote a new postgresql based index built into adrdox, and this seems to be doing a good job.

Core D Development Statistics

In the community

Community announcements

See more at the announce forum.

dpldocs cross package search live!

If you go to dpldocs.info/some_search_term now, you'll find it redirects you to http://search.dpldocs.info/ and displays search results from a variety of packages. For example, searching for discriminated union will give results from Phobos, mir, and sumtype at the time of this writing.

The search in the upper right box still searches just the package you are looking at. If you want cross package results, you need to go back to the top-level domain. I might change this later as I enhance features and build the database.

The way it works is when you rebuild the docs for a package, it populates a shared search index as it builds the HTML. (Getting this to work reasonably quickly was fun, perhaps a story I'll tell another time.) This is stored in a postgres database so I get those benefits without a lot of work, including being able to update indexes incrementally. My previous two attempts failed at this, so the search would get stale and/or run out of memory too much to be reliable.

As you can see from the results, there's still a lot of things that can be improved about it. The index got cut down a bit from my original intent in an attempt to improve performance. The scoring algorithm doesn't take "undocumented" into account - it just looks for hits, especially a hit in the name. My old one would also spell check, but the new one doesn't, again, because I was getting poor performance this time. I might be able to fix that later.

But at the same time, it works reasonably well already. And the most useful thing is you can look for names without knowing the package or even the module. Search for "GC.free" and it tells you it is in the druntime project, in the core.memory module. (Though it could show the distinction between module and nested scopes more clearly in the UI, I think I will come back to that too.)

As people rebuild their docs, it will grow the search database, and I figure I might scrape all of dub at some point too and add package names, descriptions, and maybe even readmes to the index.

Lots of potential here! But it took me a couple years to get to this point, so best be advised not to hold your breath, I have a lot going on. But even without attention from me the index will grow and hopefully the results get better and better as time goes on.

What Adam is working on

A user contributed emacs/gnu readline style keybindings to terminal.d's getline function. I don't often use those since they conflict with other things I'm used to (like ctrl+a which I use for screen switching) and/or have alternatives I prefer (like arrow keys), so I didn't think much of them. Having an actual user contribute is thus appreciated.

I also got a request for ctrl+r "reverse-i-search" support. That's one of the getline features I would avoid because I found its interface bizarre. I tend to hit arrows in an attempt to move through results, but that cancels it. Pressing enter executes it! And the cursor jumps around without representing where you are typing.

With this request though, I took more time to understand how it works and now my getline does something similar, but with an interface I hate less. First, my version calls it "history-search" on the prompt. Second, I put the cursor where you are editing. Third, the arrows search results. Fourth, to keep the indication of the match and I'm already using the cursor, I highlighted it in color. (Highlighting with color can give terrible results on certain colors, but since it is the search result you typed, I don't think it is as bad if it is hard to read, since the stuff surrounding it is the novel information.)

Lastly, pressing enter just populates the line. You must press enter a second time to execute it. This gives you a chance to edit or cancel the operation before actually doing it.

I think I'm pretty happy with it, and keeping the rest of the keybindings there for the readline users means it shouldn't break their muscle memory too badly. I hope the users will like it, but time will tell once I make the official release.

I think I will apply a few more shortcuts from readline too. But the big thing I want to fix is highlighting and copy and pasting stuff from the line. All that clipboard stuff may only work on my terminal emulator, but it would still be nice where it does.

The only worry is conflicts again. gnu readline uses ctrl+f to mean "move forward one char". (I'd rather use the arrow keys. But meh.) I'd like it to be a "find" shortcut like is common on Windows... but idk. Same with ctrl+c, commonly meaning cancel/interrupt on terminals, but copy in other contexts. Maybe I'll use ctrl+shift+thing - which will only work on my terminal emulator too, but be relatively familiar without breaking in conflicts. I'll let you know.