Hacker News Reader

Comment by monocasa | Hacker News Reader

They could not accept downtime. They were also using a database that had it's consistency guarantees scaled back for "web scale" reasons and were terrified of what a hard reset would do. Yes, the VM could just disappear randomly anyway, which made doing that a very poor choice.

There were a whole lot of questionable prior decisions that had been made that did not help.

You learn a lot in situations like this about being helpful with the customer without being judgey. Get them back on their feet with a smile when they thought they were screwed keeps a lot of contracts around.

View on HN

I like this paragraph. It says a lot for what divides the long term contractors with a full pipeline and those who don't. It would be a neat topic to blog on if you ever have the time.

To use something like this you’d still need an SSH root password setup in advance. So not much different than not losing your SSH keys or creating fallback ones.

One other option is to exploit a bug in managed software to escape to a shell. One man’s CVE or backdoor, is another support engineers magic sword to save the day.

My understanding from the post was that it requires a system root account, not ssh. The former implies the latter, but you could (and usually should) disable root ssh access.

Is that strictly true?

Other implementations I've seen drop you right into a root shell, relying on equivalents to IAM to govern access to the other side of the virtual serial port rather than machine local permissions.

Depends on how getty (or other local TTY manager) is setup. Most (if not all) Linux and BSD distributions attach login(1) to console and serial TTYs by default, and login(1) will normally require password authentication. Though, whether login prompts for a password is (I think) also a function of flags in /etc/passwd (or /etc/shadow or equivalent), so you may be able to login as `root` or `$USER` without entering a password if the account was setup that way.

if you have serial console access, can't you key-combo it to boot into single user mode?

That would still entail downtime, so if the goal is to gain access without downtime, single-user mode is probably a non-option.

(Though if some downtime is allowed, then it's probably possible to get into single-user mode, then manually start the relevant daemon and its dependencies, basically doing whatever init would normally do; I've done this in my homelab on Slackware, but it ain't something I'd be excited to do in production, and systemd probably complicates things further)

Also, not sure about other distros, but I recall that Ubuntu normally requires a root password even for single user mode. You might be better off using a boot disk and chrooting your way in.

Not without rebooting, no. None of the sysrq keys provide access to the system. Even if they did, switching into single-user mode would result in downtime for the running service.

Not by default as far as I'm aware. Though, it's been a long time since I've used a desktop Linux distribution. My experience is mostly with serial access for servers--Linux, BSD, and Solaris. I much prefer serial ports for backup administrative access as there's little chance of a misconfiguration. Whereas w/ port failover and other IPMI, BMC, etc non-sense if you disconnect the ethernet cable from the dedicated network management port you may still have admin/admin access live on the network.

once upon a time in another life it seems like, we had a modem hooked to a serial multiplexer in the "core room" in the library, and I swear once I used that to reboot a solaris machine to single user mode, but I probably telnet'ed in (this was before ssh) and then did the init 1 or whatever and then dialed in to the serial multiplexer.

I once had a second-hand Sun Sparc system, and several Sun amd64 systems (discounted for startups), and I remember Sun having a fairly sophisticated serial-attached firmware console. Though a fail-open default configuration that permitted trivially bypassing authentication seems more like something one would see w/ an x86 BIOS or bootloader. I wouldn't claim the serial-attached bootloaders on my x86 lab machines don't somehow permit bypassing login(1) authentication. But at least w/ EC2 and other hypervisors that element should be out of the equation.

This is what I do at home (kvm serial console on sulogin with root's password locked), but I would never consider that for a multi-tenant cloud provider.

A simple "rely on IAM for access control" doesn't match with my idea of defense in depth.

The article says "the only requirement is that the root account has been assigned a password, as this is the one you will use to log in", but obviously that will only apply to the default Amazon Linux AMI configuration

Probably shouldn't have erased their SSH keys then.

I mean, giving customers the finger is one strategy. Bending over backwards for the ones that pay out the ass for a good support contract is another.

"Your business might not exist if our engineers didn't dedicate themselves to your problem even though we didn't need to according to our SLA" goes a real long way at reup negotiation time with the right sized business (not too big, not too small).

Reply on news.ycombinator.com