Usually (not by design, but by circumstance), if someone gains RCE on your systems, they can also find a way to bring the tools they need to do whatever they originally set out to do. It's the old "I don't want to have a compiler installed on my system, that's dangerous, unnecessary software!"-trope driven to a new extreme. Unless the executables installed are a means to somehow escalate privileges (via setuid, file-based capabilities, a too-open sudo policy, ...), having them installed might be a convenience for a successful attacker - but very rarely the singular inflection point at which their attempted attack became a successful one.

The times I've been locked in an ill-equipped container image that was stripped bare by some "security" crapware and/or guidelines and that made debugging a problem MUCH harder than it should have been vastly outnumber the times where I've had to deal with a security incident because someone had coreutils (or w/e) "unnecessarily" installed. (The latter tally is at zero, for the record.)