1. Use IPv6 which works and goes directly to the virtual machine because each virtual machine grabs its own address from one of my 18446744073709551616 addresses.
2. Use IPv4 and either have to do a jumphost or do port forwarding, giving each virtual machine its own port which forwards to port 22 on the virtual machine.
3. Use a VPN.
I have all 3 working, but #1 was significantly less setup and works the best.Also being able to generate unique ULA subnets is super nice.