A few more debugging tips - run debugger automatically on linux

Posted 2023-03-13

Some more things that might be helpful about using gdb on linux, especially for segfaults.

Core D Development Statistics

In the community

Community announcements

See more at the announce forum.

Tip of the Week

I previously wrote a few gdb tips and want to add a few more here and update some of that. I suggest you skim the old tips there, it isn't long.

druntime has changed its GC signals in recent versions, so my .gdbinit looks like this now:

# old druntime gc signals
handle SIGUSR1 noprint
handle SIGUSR2 noprint

# new druntime gc signals
handle SIG33 noprint
handle SIG34 noprint

So routine GC behavior won't trigger the debugger.

I also like to use a helper script which I call gdbr:

#!/bin/bash

gdb -q -ex run -ex quit --args $* --DRT-trapExceptions=0

This runs the given program with the given arguments, but:

  • Automatically runs it without waiting for input
  • Automatically exits the debugger when the program completes normally
  • Tells druntime not to print uncaught exceptions, meaning the debugger can see them at the throw site

This means running gdbr foo is just like running ./foo except if it crashes, the debugger is right there, ready and waiting to be used at the crash site.

Another thing you might consider doing is enabling core dumps and even setting a PROMPT_COMMAND - which is run before each time the bash shells hows its prompt - to check for a segfault return and fire up the debugger on the core dump if it happened. This varies a bit by linux distro, but it could be like:

A screenshot of a program crashing and the debugger showing a backtrace before the prompt automatically with no explicit interaction

The PROMPT_COMMAND is this:

ulimit -c unlimited # enable core dumps in this session
# and now enable the automatic command
export PROMPT_COMMAND=' if [[ $? -eq 139 ]]; then gdb -q -ex where -ex quit $PROGRAM_NAME core; fi'

You'll gave to set the export PROGRAM_NAME=./whatever as part of the setup too. Then run your program in this shell. If it crashes, this command will see it - a return value of 139, aka -11, indicates a SIGSEGV exit - and fire up the debugger. It uses the where then quit commands automatically, meaning it will dump the backtrace and return you immediately to your command prompt.

Side effects and caveats:

  • If you press enter again without running another command, it will show the backtrace again, since the $? variable will be unchanged.
  • If you are on a distro that sets up crash reports, your core file might not be called core. Perhaps try coredumpctl debug instead of the gdb command in PROMPT_COMMAND if you have systemd on your distro. Generally speaking, this is configurable by the kernel. More on this soon.
  • The core dump will still be there for future inspection. You might want to delete it if you're done with it.

The core filename is configurable. The default is often ./core but distros with crash reports often change this. You might check cat /proc/sys/kernel/core_pattern to see what it is configured on your system. And what's very interesting about this is it need not be a filename.... it can be a pipe. This is what systemd and the coredumpctl depends on.

If you set it to a pipe, you can direct the file wherever you want and trigger something on demand. This program is triggered by the kernel and is run in a global context as root, without the same controlling terminal as the crashed program. So doing the inline backtrace info there in the terminal is another exercise - you'd probably want to have your active terminal configured to opt into the info - but it'd be possible to set it up to run your debugger on demand too.

Details here: https://man7.org/linux/man-pages/man5/core.5.html

Fun fact: gdb comes with a program called gcore that can make a core dump from a running program without stopping it. This lets you take a snapshot of its state that you save to a file for later inspection.

See https://en.wikibooks.org/wiki/Linux_Applications_Debugging_Techniques/Core_files for some more information.

Conclusion

Personally, I prefer my gdbr script. It is a local solution to a local problem and the script makes running it very easy without needing a core dump file. But the automatic shell command or even the system-wide jit debugger the kernel enables might be fun for you to explore. A CI runner or running server capturing coredumps on demand is probably better than just always having the interactive debugger standing by, but when actively working on something, the "build, debug+run" pattern is nice to have and easy to do from a shell as well as IDEs.

If you're trying to set segfault signal handlers, I encourage you to consider these options instead.