For example, with ASLR, it's easy to retrieve the current layout of the processus and therefore to debug it (if you have the appropriate symbols) despite the randomization. But an exploit has to be built for the specific instance of the application running. If you share the same privileges as the process, you can get the current mapping but an exploit is useless. If you don't, you don't have access to the mapping either and it's difficult to get it while you are executing code inside the process as you can't easily access the functions you would need for that.
e.g. if your stack is hosed, etc, things could be affected
Also, much of kernel debugging involves poking at core images, which wouldn't be coherent with the kernel binary.
That said, I'm fairly certain I recall reading that the logic saves your previous kernel for such a case, and otherwise someone could make scripts to save more copies if needed.
Others have pointed out reconstructing kernels for debug purposes.