Hacker News Reader

Comment by loafoe | Hacker News Reader

"ifconfig eth0 down" on the production bastion host, instead of on my localhost terminal -- and no hands on in the datacenter which was 160km away. Of course the bastion host was the only one not hooked up to remote power reset services.. and only 2 hours left in the service window.. sinking feeling

View on HN

For tmux users: put something like this in your .tmux.conf on production servers:

    set -g window-style 'fg=red,bg=black'

It will color the text red, hopefully reminding you to be extra careful. Adjust according to your preferences.

One other "defensive scripting" trick I frequently use is starting any `rm` command with `ls`, double checking its output (or triple checking if it's a recursive one), and then replacing `ls` with `rm`. It barely takes any extra time if you're proficient with emacs-style readline hotkeys:

    C-a M-d rm C-m

That's nice — I usually start my commands with `# ` but then there's no tab completion. I'll try `ls`

I usually use echo for the same purpose.

In this vein, I set my PS1 to bold red capital letters on bastion hosts and alias sudo="echo 'You're on a jump box moron :p'"

I do the tmux color trick too-- color coded by environment for each bastion.

We do this for our database connections in Azure Data Studio / the Windows version of it. Really great idea.

Something like this to obviously distinguish environments is good practice - at one company we implemented scripts for terminal color-coding like this after some downtime caused by a destructive backup restore accidentally being run in the production environment instead of an acceptance testing system, which was in all aspects identical to the production system.

And I've seen a solution for more secure environments where physical separation was used with the operator having separate monitors/keyboards, and the "important" system having a different color keyboard and monitor frame.

Oh, yes, that feeling when you mix up “init 5” and “init 6” on a machine 300 miles away. “Huh, it’s taking longer than usual to reboot...”. 20 years later and I still remember the name of the guy I had to call at two in the morning to go power it back on.

I've done that a couple of times, although in better circumstances; also fun variants like "the new kernel didn't have the right ethernet driver in it".

I think the first one in my career was discovering that "killall" does something very different on Solaris from Linux.

Oh nooooooo

Back in the 1990s, something like that lead to a major local data center having to admit that the “24 hour support” meant 12 hours a day with someone who could answer the phone but didn’t have access to the server room.

Or rebooting a container. But you’re not in the container anymore but on the host...

Reply on news.ycombinator.com