Doesn’t that just move the problem?
Besides, for many use-cases, SSH is not the only service you need to access remotely.
Google, for example, proposes a different school of thought – zero trust network, and strong contextual authentication of each individual request.
Precisely because you need to expose more services to more users, you need to be extremely conscious about treating singular network ingress point as a primary security gateway.
Check out https://beyondcorp.com, it’s a very interesting concept.