So does this seem like a typical use in an AWS VPC:
- SSH bastion server with public IP with security groups for each user
- Every other server only has private IPs, only allowing access between each other and then the bastion server
How do you generate and store keys? Are you using something like Hashicorp's Vault?
As far as configuring your VPC having the bastion (Proxy) as the only server with a public address is reasonable. One of the nice things about Teleport is that the Teleport Proxy itself doesn't have access to much, so exposing it to the Internet is fine. The Auth Server is the one that holds sensitive information and we recommend you create a security group for it and only allow it to be accessed from Teleport Proxies or Teleport Nodes.
With respect to keys, they are stored and accessed via the Auth Server in Teleport. We recommend you have strong access controls on the Auth Server. If you are using the default backend (BoltDB) or directory based backend that's all you need to do. If you are using etcd we recommend you have strong access controls on the server that runs etcd as well as etcd itself, we have an example in our Teleport repo for etcd configuration if you're interested[1]. If you are using DynamoDB, we recommend having a strong IAM policy. We are not using Vault at the moment.
[1] https://github.com/gravitational/teleport/tree/master/exampl...
https://github.com/gravitational/teleport/tree/master/lib/ba...
yeah this is a bad idea in general. If you have critical stuff you need to SSH into from the public internet, keep it all in private IP space and have an openvpn gateway (or IPSEC VPN) with a public interface, and a private interface facing inwards towards the hosts.
you should not even be able to route to the IP of the thing you want to SSH to unless you've authenticated to the VPN and your client device has been handed out an IP in your RFC1918 IP space.
a machine like an openvpn gateway can also serve the purpose of getting you access into an OOB network (example: a public facing IP on a 100Mbps DIA circuit you've bought from a totally diverse ISP in the same colo, with a static /30), which has access into internal IP space devices such as serial console servers and ssh bastion hosts.
authenticate the clients by a unique public/private key pair per client device. Easy to revoke a specific device's key from the server side if needed.
This is not a bad idea in general. In fact, teleport proxy implements this exact model you have just described, where only proxy is available and acts as a jump host to the set of machines available only on the private net. The only difference is that instead of open VPN gateway it uses SSH jumphost model.
Teleport proxy uses OpenSSH cert auth, in addition to that teleport node also does cert auth.
Not everyone needs to always set up VPN, sometimes jumphosts + cert auth are perfectly fine.
* first when connecting to the jumphost public ip and requesting to execute a subsystem to allow proxying to some internal IP/host
Jumphost does not terminate the SSH and in fact it is MITM capabilities are very limited.
* Second time authentication and authorization happens when ssh client connects to the target SSH node.
This pattern is in fact quite modern and is being expanded in the beyond corp architecture.
https://research.google.com/pubs/pub43231.html
It deprecates perimeter security model that you mention via VPN gateways and replaces it with on-demand end to end access via controlling gateway.
With the VPN as your boundary, you can configure it so the VPN doesn't even respond to packets unless they are TLS authenticated. A huge security win, you're not exposed to random attempts to do remote code execution against your open sockets!
Plus, this commercial thing doesn't do anything pam ldap and config mgmt can't do. Just reinventing the wheel yet again to charge people money for proprietary "solutions."
That's a ton of complexity when you could just run knockd on public facing sshds and make them disappear that way.
It's extremely tight, simple code - consisting of a single binary - and it never crashes or hangs.
No, I am not suggesting that you get rid of all of your keys and passwords and rely only on the knock for your security. (I have to write that because response-comment-numero-uno will strawman that to death). Keep your keys and passphrases in place and add the knock.
Port knocking is just the best thing.
The idea is, in addition to the normal security measures you use with sshd you also hide the service with port knocking.
Nobody anywhere, at any time, has ever suggested using port knocking as the sole means of securing your sshd.
State actors can afford millions to spend on build/buying sploits for [insert technology]. For example, use different standard for OS at edges where possible to reduce attack surface. Preferably scrub network traffic at edges (not just web traffic) and really lock down traffic to remote access boxes.
If exposing Teleport (or OpenSSH or any piece of software to be honest) to the internet is a significant concern, the solution I recommend is to put it behind spiped. Spiped has a very small and well written code base and well suited for this problem.
Because Teleport is fully interoperable with OpenSSH you just follow the instructions in the README to get it working: https://www.tarsnap.com/spiped.html
Plus, there's already plenty of ways to AAA OpenSSH using puppet/chef, PAM, RFC 4255, google authenticator (via pam plugin). It's really easy to set up if you've done it before.
a) laptop
b) home office desktop
c) android or ios device (phone/tablet)
then, of course, once connected to the vpn, to authenticate to ssh the person will probably be using their per person ssh public/private key pair from their workstation.
So we have the ability to revoke an individual client device's vpn keys separately. In event of total compromise we can revoke both vpn keys for device(s) and the person's ssh key.