Those people might be right. Defense in depth is a legitimate tactic, but that's all it is, and it's often an excuse for people to waste time layering stupid stuff on top of real security controls.

ASLR, NX, and CFI would be an example of a defense in depth stack that is meaningful.

SSH, Fail2Ban, and SPA would be an example of a defense in depth stack that basically just wastes time.

I would be more comfortable with a system where I knew I had to burn the box if I lost RCE on it than I would be with a system that somehow depended on RCE not coughing up kernel, and persistence, to an attacker.

The other thing defense in depth can provide is increased attacker cost. That's why there are economically valuable DRM systems (BluRay's BD+ is an example here). All you have to do is push attacker cost across a threshold (for instance with BD+, that's keeping titles secure past the new release window) to make a defense in depth control valuable.

But if someone has a kernel exploit, probably nothing you've done for defense in depth is going to meaningfully increase costs.