Hacker News Reader

Comment by jasongill | Hacker News Reader

In a past life, this is an interview question that I would ask people: "We have thousands of servers, and you need to run the same command on 10 of them, what's one way you would do it?" and follow up with "What if you wanted to run the command on hundreds or thousands - what problems would you expect with this approach, and what might you do differently?"

I didn't really expect them to write code on the spot (hate that in interviews), but just to describe a possible solution; there were no wrong answers. Seeing how people came up with a quick hack for 10 and then being able to evolve their thinking for thousands was the point of the question and it could be enlightening to see.

I had a lot of people come up with things that I never even thought of, such as SSH'ing in to all 10 machines and then using a terminal feature to type into all of the windows at once.

View on HN

That's a great question you used in interviews! I'm curious – how would you personally approach this situation? What would your solution be for running a command on 10 servers, and how would you scale it to thousands?

For a handful of servers I would just do a for loop like `for i in {01..10};do echo "command" | ssh -T server$i.example.com;done` because it was quick and dirty and worked fine for something quick, but obviously it doesn't really ensure a common state or handle errors at all (but I still used a for loop like that many times a day for quick stuff like "I wonder what size a specific file is on each server" or "let me quickly grep a config file across a range of servers because I forget which one this thing is running on").

For more than that, I used Puppet at the time (this was a decade and a half ago); I was a contributor to Puppet and standardized on it in my company. Eventually we moved to Ansible and I sold that business but last I heard, they are still using Ansible and likely using playbooks that were ported over from my Puppet stuff

thanks

Are you still asking this question in an interview? I know a thing or two about sshing into servers, I think I'd be a good enough candidate

Unfortunately (or, fortunately) no - this was a lifetime ago when I owned a large hosting company

Is there a standard way of doing this that comes shipped with GNU/Linux?

I'd just use Ansible for that.

Just found out about pssh through the comments

I dunno about standard, but it's been done a bunch.

* As sibling notes, there's ansible (or chef/puppet/salt/...)

* The traditional solution was https://github.com/duncs/clusterssh which opens an xterm to each target plus a window that you type into to mirror input to all of them

* I do the same-ish in tmux with

  bind-key a set-window-option synchronize-panes

and I expect screen and such have equivalent features

* Likewise, there are terminal emulators that let you do splits and then select a menu option to mirror input to all of them

Reply on news.ycombinator.com