You don't need to dream about it. You can absolutely do this today with OpenZiti. You just need to be able to set it up which is - imo (I am a dev on the project and wrote the quickstarts) just as easy to get up and running as anything. I do it in "under a minute" but I work on the project so my timing is not fair... :)

You can find information about it over at https://openziti.github.io/ you don't even need to trust the software itself. You can add a 3rd party certificate to the server and mint your own private keys/certs and deliver them to your friends and have 100% control over where and how and whom you trust. You control access down to individual services, not CIDR blocks, not IP addresses. You can embed the sdks into any of your own apps if you're into that sort of thing. :) you could setup a relay server in some cloud provider for the 'untrusted' traffic (hmmmm you make me wonder if we could integrate with tor somehow now too...)

Seems like it'd do most/much of the things you want it to. I'd be happy to help you out. We have a discourse you can post questions to.