And you are exactly right.
The problem is: People are so used to the "exceptions" paradigm from other languages, when they see "panic-recover" many immediately think "That's the same thing!!"
It isn't, because the only VALID usecase for panics is exactly what you describe: unrecoverable error conditions where terminating the program is the best course of action.
`panic/recover` used like exceptions is an antipattern, and one of the worst code smells in a Go codebase.