Hacker News Reader

Comment by monocasa | Hacker News Reader

Oh thank god. I had a customer once that erased their SSH keys, and had a running database cluster on EC2 that they couldn't get access too anymore. That was... fun.

This is a long time coming.

View on HN

I've used SSM Agent to get out similar hot spots in the past, but this will be a nice option for those instances that are somehow both broken to the point where you can't connect to them and still important enough that they need to be up _now_.

Sure there’s down time involved, but you can always stop and mount the volumes elsewhere and then update the SSH keys.

They could not accept downtime. They were also using a database that had it's consistency guarantees scaled back for "web scale" reasons and were terrified of what a hard reset would do. Yes, the VM could just disappear randomly anyway, which made doing that a very poor choice.

There were a whole lot of questionable prior decisions that had been made that did not help.

You learn a lot in situations like this about being helpful with the customer without being judgey. Get them back on their feet with a smile when they thought they were screwed keeps a lot of contracts around.

Probably shouldn't have erased their SSH keys then.

I mean, giving customers the finger is one strategy. Bending over backwards for the ones that pay out the ass for a good support contract is another.

"Your business might not exist if our engineers didn't dedicate themselves to your problem even though we didn't need to according to our SLA" goes a real long way at reup negotiation time with the right sized business (not too big, not too small).

To use something like this you’d still need an SSH root password setup in advance. So not much different than not losing your SSH keys or creating fallback ones.

One other option is to exploit a bug in managed software to escape to a shell. One man’s CVE or backdoor, is another support engineers magic sword to save the day.

Is that strictly true?

Other implementations I've seen drop you right into a root shell, relying on equivalents to IAM to govern access to the other side of the virtual serial port rather than machine local permissions.

Depends on how getty (or other local TTY manager) is setup. Most (if not all) Linux and BSD distributions attach login(1) to console and serial TTYs by default, and login(1) will normally require password authentication. Though, whether login prompts for a password is (I think) also a function of flags in /etc/passwd (or /etc/shadow or equivalent), so you may be able to login as `root` or `$USER` without entering a password if the account was setup that way.

if you have serial console access, can't you key-combo it to boot into single user mode?

Not by default as far as I'm aware. Though, it's been a long time since I've used a desktop Linux distribution. My experience is mostly with serial access for servers--Linux, BSD, and Solaris. I much prefer serial ports for backup administrative access as there's little chance of a misconfiguration. Whereas w/ port failover and other IPMI, BMC, etc non-sense if you disconnect the ethernet cable from the dedicated network management port you may still have admin/admin access live on the network.

once upon a time in another life it seems like, we had a modem hooked to a serial multiplexer in the "core room" in the library, and I swear once I used that to reboot a solaris machine to single user mode, but I probably telnet'ed in (this was before ssh) and then did the init 1 or whatever and then dialed in to the serial multiplexer.

I once had a second-hand Sun Sparc system, and several Sun amd64 systems (discounted for startups), and I remember Sun having a fairly sophisticated serial-attached firmware console. Though a fail-open default configuration that permitted trivially bypassing authentication seems more like something one would see w/ an x86 BIOS or bootloader. I wouldn't claim the serial-attached bootloaders on my x86 lab machines don't somehow permit bypassing login(1) authentication. But at least w/ EC2 and other hypervisors that element should be out of the equation.

That would still entail downtime, so if the goal is to gain access without downtime, single-user mode is probably a non-option.

(Though if some downtime is allowed, then it's probably possible to get into single-user mode, then manually start the relevant daemon and its dependencies, basically doing whatever init would normally do; I've done this in my homelab on Slackware, but it ain't something I'd be excited to do in production, and systemd probably complicates things further)

Also, not sure about other distros, but I recall that Ubuntu normally requires a root password even for single user mode. You might be better off using a boot disk and chrooting your way in.

Not without rebooting, no. None of the sysrq keys provide access to the system. Even if they did, switching into single-user mode would result in downtime for the running service.

The article says "the only requirement is that the root account has been assigned a password, as this is the one you will use to log in", but obviously that will only apply to the default Amazon Linux AMI configuration

This is what I do at home (kvm serial console on sulogin with root's password locked), but I would never consider that for a multi-tenant cloud provider.

A simple "rely on IAM for access control" doesn't match with my idea of defense in depth.

My understanding from the post was that it requires a system root account, not ssh. The former implies the latter, but you could (and usually should) disable root ssh access.

I like this paragraph. It says a lot for what divides the long term contractors with a full pipeline and those who don't. It would be a neat topic to blog on if you ever have the time.

That's what I was thinking... I mean, this is cool and all, but the only reason I'd ever muck around at this level is if I was actually responsible for the physical hardware and there was a cost associated with replacing it. The whole point of moving to the cloud is to pay somebody else to have to worry about anything I can comprehend needing this for.

A lot of medium sized companies moved to the cloud without understanding the tradeoffs. And they have big pockets when it matters.

I suspect the percentage of “cloud servers” that are just standard Linux boxes running relatively standard applications is quite high.

The FreeBSD kernel debugger is available via serial console and is really useful when it's useful. If you manage to have a system that breaks your kernel, chances are switching instances won't help, and figuring it out will help. If it crashes, you can get a core dump, but if it loops in a bad place and that only breaks some things, the debugger can show things quickly.

Also serial console is pretty handy when you push bad firewall rules and don't want to throw away your instance.

So, a lot of database systems aren't using EBS for many reasons: that's why the local I/O instances exist. And so, yes: you don't build out like that unless but you are prepared to just scrap the instance and rebuild it from WAL logs or have replication or whatever, but knowing "this is a 30 second fix with a serial console" really makes taking all of that cost in stride painful.

Reply on news.ycombinator.com