Set up a network, don't make it public. Block everything incoming except your VPN tunnel should you need remote access. That's how a private network works.
With the slight caveat that you should have at least a second out-of-band access method for when you bork your VPN config :)