Your binary usually won't get loaded at the same address in memory. The addresses would be useless without the memory map.

That's solvable though. The bigger problem is how you unwind the stack. the stack is not generally unwindable, unless you're the compiler. Debug symbols include information from the compiler about the stack sizes and shapes to help backtrace with unwinding the stack. It's quite possible to include such symbols in the final binary without adding debug symbols, a lot of compilers just don't have a specification for that.