Delve crashing is just related to Delve being buggy. Single-stepping is done at the instruction level, and the task of a Go debugger mostly relates to mapping instructions to source code, and understanding the runtime.
You can debug Go with gdb, but without Go runtime comprehension, all you will see is a bunch of worker threads executing Go runtime code, rather than the individual goroutines of your process.
emacs + gud mode is a great experience for go debugging. I've even used it to debug low-level issues like plugin loading and verification.
Does anyone know whether dwarf simply doesn't support the necessary debug information or whether compilers just don't retain it through the optimization process?
DWARF appears to support mapping symbol locations to registers, but I’ve never seen a language environment/debugger that takes advantage of it for optimized binaries.
Are you able to step inside code running in a goroutine? with GDB? Because it's where GDB starts to become useless for debugging.