As far as configuring your VPC having the bastion (Proxy) as the only server with a public address is reasonable. One of the nice things about Teleport is that the Teleport Proxy itself doesn't have access to much, so exposing it to the Internet is fine. The Auth Server is the one that holds sensitive information and we recommend you create a security group for it and only allow it to be accessed from Teleport Proxies or Teleport Nodes.
With respect to keys, they are stored and accessed via the Auth Server in Teleport. We recommend you have strong access controls on the Auth Server. If you are using the default backend (BoltDB) or directory based backend that's all you need to do. If you are using etcd we recommend you have strong access controls on the server that runs etcd as well as etcd itself, we have an example in our Teleport repo for etcd configuration if you're interested[1]. If you are using DynamoDB, we recommend having a strong IAM policy. We are not using Vault at the moment.
[1] https://github.com/gravitational/teleport/tree/master/exampl...