Exactly. This is what I employ as one of the safeguards as well. All important resources accessible only to via a jumpbox, and jumbox itself is only accessible in a VPN.
I typically have a public bastion and just shut it down. In AWS you aren't paying for it if it's turned off and you have an "escape hatch" if you need it.
If someone manages to launch another instance or start the stopped one then they would have access, but that's another can of worms.