We call it panicking because there's no assumption that you're always able to catch panics. Panicking is an AAAH EVERYTHING WENT WRONG kind of thing, which might trigger an immediate abort if the program is compiled in super-ultra-speedy-make-debugging-really-hard mode; if your code panics, it's always the programmer's fault.
The language provides a way of kinda-sorta recovering from an everything-goes-wrong situation only to be nice to you, and not because that's a thing that's generally possible. You can't always recover from a segfault, or a buffer overrun, or the rest of the kind of things that cause panicking, even if you know that they're theoretically possible in advance, because if you could you would've fixed them already. The best you can do is make your web server return a 500, send logs, ditch its cache, restart in a sane state and hope for the best. You're not recovering that half-finished task, because the programmer can't reason about the program state in a situation where their code has been shown to be wrong.
Panics are only for exceptional failure cases. If this confuses people, it's (probably) because they only thought they understood; they needed confusing.