You can set it up that way, but then you wouldn't be able to deploy from it ... unless you're developing something that is being used internally?
Using a VPN? Or are you hosting your own stuff?
If your public facing server gets breached and it is sitting on your actual internal network or is connected through a VPN ... well, that means the attacker just got a free pass right through your firewall.
I used to put my production boxes on a VPN, but now I don't have to because I can deploy from GitHub. That was the main reason I signed up with GitHub.