I would worry about being DoS'ed, and having my home network compromised because of a vulnerability in raspberry pi software stack. Other than that, I think it's a good idea. I do the same for a personal use server, but don't expose the port on the public internet.