mattjaynes

Docker is an amazing tool, but there are quite a few misconceptions about it.

Nearly all the articles and examples seem super easy, but it's quite a challenge to use in an actual multi-host production environment supporting mission-critical systems.

This can send the wrong message to beginners and send newbies down the wrong path, which ends up as a disservice to Docker when they hit the complexities and become disillusioned with it.

To address those misconceptions, I've just posted the article "Docker Misconceptions": https://devopsu.com/blog/docker-misconceptions/

I spent about a month working on setting up Docker for production scenarios and reading everything I could on it (all the docs, courses, and well over 100 articles - see Docker Weekly for a wealth of resources). My post covers the main misconceptions I see popping up and also some advice for simplifying Docker use if you do decide to use it for multi-host production.

I'm sure I've probably missed some things, so please chime in if I've missed something important.

This is the talk from Twitter's Steve Salevan at this week's AnsibleFest in NYC.

The ansible-based deploy system Twitter uses is here: https://pantsbuild.github.io/

For more from that event, see this week's issue of Ansible Weekly: https://devopsu.com/newsletters/ansible-weekly/33.html

Thanks! That's great to know :) I've been told Berlin is pretty spread-out, so it'll be good to have access to a vehicle for longer distances.

Was living in NYC and absolutely loved being able to walk everywhere and not own a car. The last few months I've been in US suburbia though (Texas/Utah) and it seems actively hostile towards walkers in many places.

Sometimes there's sidewalks, but often they're not continuous and just end for no apparent reason. You then have to walk in the road next to the cars for a ways and hope the sidewalk starts up again.

Even if there are sidewalks, there seems to be a unspoken "don't walk at night" rule even in safe neighborhoods. If I do walk at night, I get suspicious looks. Am I frightening the neighborhood by going on a walk after dark? It sure feels like it.

In suburbia, I'll even get suspicious looks walking during the day - as if I'm scoping out houses to rob or something. And I'm a pretty boring safe looking type of guy (so I'm told, haha).

The exception to this I've found is if I'm walking a dog or walking with another person (preferably female) to legitimize me as a walker.

The longer I'm in these parts of suburbia, the more I realize that these attitudes are somewhat justified. Sadly, I don't think I've ever seen a man walking around the neighborhoods by himself unless he was a vagrant.

Apparently the vast majority of men in these neighborhoods just don't ever go on walks unaccompanied. Otherwise it would turn into a 'normal' activity and the suspiciousness of it would subside.

I realize there are probably great neighborhoods in the US for walking, but from my limited experience (living in ~10 states), they seem pretty rare outside of urban centers.

I spent a lot of time in Germany (US military communities) growing up and it was a totally different experience. Much of Europe that I've seen is likewise a vastly better experience for walking.

In 6 days I'm moving to Berlin. Why? Well, I'm bootstrapping a small systems-training business and want to live somewhere low cost and without a car. While cheap walkable places surely must exist in the US, they're rare and I haven't been able to find many that are also safe and not run-down. If I'm missing some, please let me know!

After being in US suburbia for the last few months, I'm really looking forward to being a first-class citizen as a walker again.

This reminds me a lot of 'Volksmarching' in Germany ( https://en.wikipedia.org/wiki/Volksmarching ). It's still a popular past-time there from what I can tell.

There's even an American organization for it, though I'd hadn't heard of it before today: http://www2.ava.org/

I forgot to mention, the Phusion guys (who Mitchell mentions in the post and who create the excellent Passenger web server) have created some great assets for Vagrant and Docker:

https://github.com/phusion/baseimage-docker

https://github.com/phusion/passenger-docker (built on the baseimage above)

http://blog.phusion.nl/2013/11/08/docker-friendly-vagrant-bo...

And regarding role-based containers, Phusion's Hong Lai says:

"Wait, I thought Docker is about running a single process in a container?

Absolutely not true. Docker runs fine with multiple processes in a container. In fact, there is no technical reason why you should limit yourself to one process - it only makes things harder for you and breaks all kinds of essential system functionality, e.g. syslog.

Baseimage-docker encourages multiple processes through the use of runit."

This is a really great step forward - thanks Mitchell!

I've recently spent a couple of weeks doing a deep dive into Docker, so I'll share some insights from what I've learned.

First, it's important to understand that Docker is an advanced optimization. Yes, it's extremely cool, but it is not a replacement for learning basic systems first. That might change someday, but currently, in order to use Docker in a production environment, you need to be a pro system administrator.

A common misconception I see is this: "I can learn Docker and then I can run my own systems with out having to learn the other stuff!" Again, that may be the case sometime in the future, but it will be months or years until that's a reality.

So what do you need to know before using Docker in production? Well, basic systems stuff. How to manage linux. How to manage networking, logs, monitoring, deployment, backups, security, etc.

If you truly want to bypass learning the basics, then use Heroku or another similar service that handles much of that for you. Docker is not the answer.

If you already have a good grasp on systems administration, then your current systems should have:

    - secured least-privilege access (key based logins, firewalls, fail2ban, etc)
    - restorable secure off-site database backups
    - automated system setup (using Ansible, Puppet, etc)
    - automated deploys
    - automated provisioning
    - monitoring of all critical services
    - and more (I'm writing this on the fly...)

If you have critical holes in your infrastructure, you have no business looking at Docker (or any other new hot cool tools). It'd be like parking a Ferrari on the edge of an unstable cliff.

Docker is amazing - but it needs a firm foundation to be on.

Whenever I make this point, there are always a few engineers that are very very sad and their lips quiver and their eyes fill with tears because I'm talking about taking away their toys. This advice isn't for them, if you're an engineer that just wants to play with things, then please go ahead.

However, if you are running a business with mission-critical systems, then please please please get your own systems in order before you start trying to park Ferraris on them.

So, if you have your systems in order, then how should you approach Docker? Well, first decide if the added complexity is worth the benefits of Docker. You are adding another layer to your systems and that adds complexity. Sure, Docker takes care of some of the complexity by packaging some of it beautifully away, but you still have to manage it and there's a cost to that.

You can accomplish many of the benefits of Docker without the added complexity by using standardized systems, ansible, version pinning, packaged deploys, etc. Those can be simpler and might be a better option for your business.

If the benefits of Docker outrank the costs and make more sense than the simpler cheaper alternatives, then embrace it! (remember, I'm talking about Docker in production - for development environments, it's a simpler scenario)

So, now that you've chosen Docker, what's the simplest way to use it in production?

Well, first, it's important to understand that it is far simpler to manage Docker if you view it as role-based virtual machine rather than as deployable single-purpose processes. For example, build an 'app' container that is very similar to an 'app' VM you would create along with the init, cron, ssh, etc processes within it. Don't try to capture every process in its own container with a separate container for ssh, cron, app, web server, etc.

There are great theoretical arguments for having a process per container, but in practice, it's a bit of a nightmare to actually manage. Perhaps at extremely large scales that approach makes more sense, but for most systems, you'll want role-based containers (app, db, redis, etc).

If you're still not convinced on that point, read this on microservices which points out many of the management problems: http://highscalability.com/blog/2014/4/8/microservices-not-a...

You probably already have your servers set up by role, so this should be a pretty straight-forward transition. Particularly since you already have each system scripted in Ansible (or similar) right?

To run Docker in a safe robust way for a typical multi-host production environment requires very careful management of many variables:

    - secured private image repo (index)
    - orchestrating container deploys with zero downtime
    - orchestrating container deploy roll-backs
    - networking between containers on multiple hosts
    - managing container logs
    - managing container data (db, etc)
    - creating images that properly handle init, logs, etc
    - much much more...

This is not impossible and can all be done and several large companies are already using Docker in production, but it's definitely non-trivial. This will change as the ecosystem around Docker matures (Flynn, Docker container hosting, etc), but currently if you're going to attempt using Docker seriously in production, you need to be pretty skilled at systems management and orchestration.

There's a misconception that using Docker in production is nearly as simple as the trivial examples shown for sample development environments. In real-life, it's pretty complex to get it right. For a sense of what I mean, see these articles that get the closest to production reality that I've found so far, but still miss many critical elements you'd need:

http://code.tutsplus.com/tutorials/easily-deploy-redis-backe...

http://www.powpark.com/blog/programming/2014/01/29/integrati...

http://blog.buddycloud.com/post/80771409167/using-docker-wit...

https://journal.paul.querna.org/articles/2013/10/15/docker-u...

(If you know of better ones, please share!)

To recap, if you want to use Docker in production:

    1. Learn systems administration
    2. Ensure your current production systems are solid
    3. Determine whether Docker's benefits justifies the cost
    4. Use role-based containers

Shameless plug: I'll be covering how to build and audit your own systems in more depth over the next couple months (as well as more Docker stuff in the future) on my blog. If you'd like to be notified of updates, sign up on my mailing list: https://devopsu.com/newsletters/devopsu.html

This is a really great point.

If you set up your systems right, they pretty much run themselves.

I've been doing systems contracting for a few years and contracting often turns out to be much better suited for this work than full-time employment (unless you're a company at significant scale).

My typical contract goes like this:

    - gain an understanding of their current systems
    - document *everything* that matters in their systems
    - simplify *everything* that matters in their systems
    - organize *everything* that matters in their systems
    - secure *everything* that matters in their systems
    - automate *everything* that matters in their systems
    - monitor *everything* that matters in their systems
    - train their engineering team on how to run the systems

That's it. If I've done my job right, they'll never need me again unless something exceptional happens.

The best sysadmins make themselves redundant as quickly as possible.

The best sysadmins put the power of the systems in the hands of the other engineers.

The best sysadmins are well rested because the systems they set up are so damn stable and boring.

How do you know you have a bad sysadmin?

    - they are a 'hero' that often rescues the company
    - no one understands the systems but them
    - they are a bottleneck for deploys
    - little is documented
    - little is automated
    - they work long hours
    - they're often frazzled and stressed

A big part of DevOps is saving developers from bad sysadmins. Fortunately, the last few years have been a renaissance for usability in systems tools (like Ansible for instance). This newfound usability for managing and automating systems has finally put the power of systems management within the reach of many more developers.

For more, see my posts on these topics:

Boring Systems Build Badass Businesses https://devopsu.com/blog/boring-systems-build-badass-busines...

Metrics that Matter for your Systems https://devopsu.com/blog/metrics-that-matter-for-systems/

Yep: http://www.pixelmator.com/tutorials/retouching/smudge-tool/

Apparently my comment is being misinterpreted (evident by the downvotes) :/

To clarify:

It's simply amazing the scale at which you have to get to for spiped to need a new key. It's a tribute to its design and its author.

If I understand it right, in order for this to be a real issue, an attacker would have to spend decades or centuries diligently recording either 18 quintillion connections or 18 billion gigabytes of your encrypted traffic in order to even begin to try cracking the key in any serious way.

The solution to this "vulnerability"? Switch out your key (takes a few seconds tops).

I laugh because of Colin's dry humor matter-of-factness about this mind-blowing level of scale. If you missed his own poking fun at it, read his other comments in this thread :)

omg, I love you. Yes folks, also, remember not to create 18 QUINTILLION connections (18 billion billion) with the same key. :)

Amazing project. And I love the understated tone of this in the README:

"Security Requirements: The user is responsible for ensuring that any individual connection does not transmit more than 2^64 bytes."

If you don't happen to know how big 2^64 bytes is, it's over 18 BILLION GIGABYTES! We can probably safely assume most of us will never hit that limit on a single connection, haha.

Ref: https://code.google.com/p/spiped/source/browse/trunk/README

Just bought it - thanks for the recommendation :)

For US based folks, here's the US link: https://itunes.apple.com/us/app/tyme/id670592452

Thanks Gary :)

I got most of my inspiration from Kathy Sierra and Why the Lucky Stiff. They're waaaay better than me in this regard, but they taught me that most brains love a little whimsy. I try to add a little since systems engineering can slip into dryness pretty quickly if you're not careful.

Little jokes like the borg cow can lighten things up: https://devopsu.com/newsletters/ansible-weekly-newsletter.ht...

An added benefit is that whimsy helps filter out trolls. Adorable puppies, kittens, squirrels, cows, etc have a way of turning trolls away or at least softening them up a little :)

That's a great point. It's one reason Docker is so exciting - to be able to replace a running server in a sane and organized way is a killer feature.

For many cloud environments, it'd be costly (in time and energy, not necessarily $) to replace all 1000 virtual servers for an update. With Docker, you can essentially do that in a trivial manner. I'm still learning Docker and my understanding is still a bit weak, but it's an exciting development in this regard.

Good point: "if a few shell commands is all you want to do."

Agree totally. If you're doing something tiny, then a few shell commands are what is needed, not a CM tool.

I'm speaking mostly about serious systems that businesses run on.

Ansible is not for everyone. Each tool has strengths and weaknesses. I generally push Ansible because it's the easiest to get started with, but can also scale to 10K+ nodes. If something simpler/easier comes along, I'll recommend that instead.

I suspect some combo of Docker and Ansible to ultimately be the simplest set up (in the near-term), but I'm actively learning that and not confident enough in it to be able to suggest it to newbies.

Have to say I'm a bit puzzled by the claim of Ansible being "blow your brains out" difficult.

In many cases, Ansible is even easier than shell scripts. I wrote a post about this a few months ago: https://devopsu.com/blog/ansible-vs-shell-scripts/

I completely understand where the sentiment is coming from though. I wrote a comparison book on Puppet, Chef, Salt, and Ansible a few months ago and am currently finishing the 2nd edition ( https://devopsu.com/books/taste-test-puppet-chef-salt-stack-... ). Even for an experienced sysadmin, using Puppet and Chef to do even a trivial project (replacing a ~10 line shell script) took a painful couple of days. Why? They're overly complex and have confusing broken documentation (that mostly haven't been corrected even 6 months after I gave them a full breakdown on the issues). Salt was pretty smooth, but Ansible was downright easy.

Using a shell script to set up a server generally indicates that it will then be managed manually afterwards (sadness and despair!).

A huge advantage of using a configuration management (CM) tool is that they're "idempotent". Idempotency basically means that you can run the directives over and over again safely.

An idempotent command will verify that the system is how you defined it and will only make changes to bring the system back into alignment with what you defined. That means you can define your system in the language of the CM tool and use it not only for initial system setup, but also for monitoring, updating, and correcting a server's configuration over the life of the server.

A CM tool can ultimately act like a self-healing test suite for your systems - neat!

Your systems are the "app" that your app runs on. They're the foundation. Not using a CM tool is like not having any tests for your app. Sure, it might seem faster at first, but you'll pay for it in chaos, slowness, bugs, and misery later.

Shell scripts are a great first step, but if you're serious about your systems, you need to be using a CM tool. Modern ones like Ansible are simple, easy, and have great docs so there's little excuse left for not using one now.

Does anyone know if this affects any of the popular configuration management tools like Puppet, Chef, SaltStack, etc?

"Build a scalable service on top of something open source, rather than T&M services."

37Signals/Basecamp are an interesting example of this. Rather than building a business around supporting Ruby on Rails, they've built products on top of the framework they created. It's certainly an outlier and I can't think of many other examples like them who both built a popular tool, but then just used it as a tool instead of trying to monetize it directly.

DHH wrote: "External, expected rewards diminish the intrinsic motivation of the fundraising open-source contributor. It risks transporting a community of peers into a transactional terminal. And that buyer-seller frame detracts from the magic that is peer-collaborators.

It also holds the threat of corrupting the community at large. We plant the seeds of discontent by selective monetary rewards."

http://david.heinemeierhansson.com/2013/the-perils-of-mixing...

I've certainly seen this recently in some open-source communities. The peer-to-peer camaraderie transitions to a dictator-to-underlings relationship and people start defecting. It's sad to see, but happens when an open-source company isn't vigilant in nurturing the community that got them where they are.

Kudos to those companies that are able to pull off maintaining a strong community while still staying focused. It's a rare thing to be able to do well and behind the scenes there's a lot of work that goes into that kind of nurturing.

Haha, classic!

For those that didn't fully experience the 80's, "Jem and the Holograms" was a cartoon that took cheese and glam to a whole new level.

Here's the theme song, which will give you a good taste of that awesome 80's relic: https://www.youtube.com/watch?v=m6G_o1MYECg

I love this so much. It's freeing to think of English as just a hodge-podge of mismashed languages - because it is!

Current grammar is just a snapshot of the evolving mush of english frozen in time. I'm happy to let go of some of those awkward rules that sound so odd in our current version of english.

Grammar should always be the servant of meaning, not the other way around.

P.S.

If you're feeling truly adventurous, watch the 8 part historical documentary on The Adventure of English: https://www.youtube.com/watch?v=t32nCpzxHUc (part 1)

Images appear broken in this article. For a version of the article with working images, see the author's personal site: http://joshuaearl.com/selfpublishing/lessons-learned-from-a-...

For a higher-level overview of this release, see the blog post: http://blog.ansibleworks.com/2013/11/21/ansible-1-4-released...

A lot of great new features in this release - very excited to see the tool become more and more powerful.

I only came across Ansible this summer, but it really blew my mind. I've been doing systems contracting for a few years now and Ansible is the most valuable tool I've come across for systems management so far. Now, rather than have to use (and train my clients on) multiple tools, I can have them just learn one solid tool that can handle orchestration, deployment, configuration management, server inventory, etc. I can now do a systems project in days when before the same project would take weeks. Expect to hear a lot more about Ansible in the future.

Ansible isn't awesome because of a feature-list. It's awesome because it saves me time and gives me a lot of power for very little investment. Whether you're setting up a single server or 10,000 nodes, you can use this tool to give you a ton of leverage without having to spend weeks learning complex tools.

That's why I've been focusing on it so much in the last few months.

Shameless (but highly relevant!) plugs:

For weekly community updates and cowsay jokes, sign up for Ansible Weekly: http://devopsu.com/newsletters/ansible-weekly-newsletter.htm...

For a comparison of Ansible with other configuration management tools (implementing an identical project in each tool), see my book on the subject: http://devopsu.com/books/taste-test-puppet-chef-salt-stack-a...

For a brief introduction to Ansible in comparison to Shell Scripts, see: http://devopsu.com/blog/ansible-vs-shell-scripts/

Very cool! I'll add it to tomorrow's issue of Ansible Weekly: http://devopsu.com/newsletters/ansible-weekly-newsletter.htm...

This is a really great point.

I'm a big advocate of Kathy Sierra's "minimum badass user" strategy. Basically, try to write as little about yourself and write what will be most useful to improve someone else's abilities. I'm not amazing at this (yet), but it's my main goal.

Identifying too closely with being a "writer" or "coder" or whatever can be detrimental to doing more meaningful work. That's because it's all about you. But really, it should be all about your users and how you are making their lives better.

The more you can focus on "What will help my reader be awesome?" instead of "What will I write?", the better your writing will be.

Nobody cares about how awesome I am (and they shouldn't!), but everyone cares about becoming more awesome themselves. Resist the temptation to focus on yourself and shift that focus to them and you'll automatically be better than 99% of writers out there.

If you spend a week learning something remarkable and then show others how to do it in only an hour, it's a huge win for them. By focusing on learning and then sharing what you learned in a succinct way, you are accelerating the progress of others. That, more than anything else, will make you a great writer. The metric should be how much more powerful your readers are after reading your writing.

(Naturally, I'm talking about a specific type of writing here, not fiction or historical writing, etc.)

Remember that these aren't employees, but are temporary part-time contractors. It's not really "firing", but just ending the contract earlier for some than for others. All the contracts end ultimately, I just keep the best performers much longer than the lesser performers.

In practice, it usually means that I start sending more and more work to the better workers until there is little work left for the worker that doesn't perform as well.

I guess you could think of it like a load balancer distributing work. If there is a worker that is less responsive or sending poorer results than the rest, then you start sending less and less work to them. Ultimately, you run out of work for them since it's all going to the higher performers.

At that point, there's no point in continuing the contract with the lower performer. So, then I will let them know and end the contract and give them the best review and feedback I can while still being honest about their performance.

It's worth noting again that these are short-term part-time contracts, so this already fits within the contractor's expectations. They don't expect this job to last forever and they're generally not too heartbroken if a contract ends. They are often working on multiple contacts simultaneously, so mine is just one of several contracts.

Ultimately all contracts end, even for the best workers. The nice thing about hiring temporary contractors is that their expectations are already set that this is a temporary engagement.

"Gently" just means that you are sensitive to the feelings of the person you aren't going to continue sending work to. That doesn't mean you can't tell them why. I often do tell them why - it really depends on how they took feedback earlier in the contract. If they are defensive and don't take feedback well, then I generally don't give them final feedback since they've made it clear they don't like it. However, if they've taken feedback well during the contract, then I give them final feedback that will help them improve and win future contracts.

The feedback I give is also gentle. Well, I guess everything is done gently. These are all good human beings your dealing with, so why not be gentle? Gentleness doesn't mean you aren't honest with them. It just means that you are considerate of their feelings when working with them.

I can confirm this works in my own experience of trial-and-error interviewing over 2000 applicants and hiring over 200 for my clients and myself (since ~2006).

Nearly 100% of the applicants were remote, so I think that helped me from falling into traps of poor "traditional" hiring practices.

The point of hiring these remote folks was to help accelerate whatever team I was on. It can be a great way to scale your existing team very quickly if you know how to do it.

For example, I took over an iPhone app dev team and it was taking them 4 months to produce an app. These apps were all very similar in functionality, but the developers were spending a ton of time slicing images, testing, and other tasks that remote workers could easily do. So I hired some remote staff (via oDesk) to do most of that supporting work and we got the app production time down consistently to 1 month. That was a huge ROI for the business since the total cost for all remote staff was the same as for 1 additional on-site engineer.

There's nothing magic about hiring well, but I've watched others try to hire remote staff and the vast majority of them try once, fail, and give up on it.

They will approach the hiring process in a traditional way (personally interview them to watch how they handle puzzles, etc). It's a grueling process and then they still get really poor hires and conclude that "outsourcing doesn't work".

It's most helpful to think of the process as panning for gold. (Naturally, I'm not saying that some people are more valuable than others innately, just that you're looking for those who are most valuable at performing your given tasks.)

So, to find gold, you must filter, filter, filter. That's the exact process for finding applicants that are high performers. Most of your applicants will be pretty terrible at the job you're hiring for, so the filtering process is critical for success.

  - filter out the very worst applicants with a small easy question
  - filter out the remaining applicants:
     - pick a real-life production task you've recently completed
     - ensure that the task is *exactly* what they'd be doing in the job
     - have them perform the task
     - compare their task results to your task results
  - hire more than you need of the top performers
  - filter out (gently fire) the ones that aren't as good
  - repeat as needed until you have gold

When I see others attempt this, the most common problem is that they essentially go down to the river and just grab whatever pebbles they see in their first handful and hope there's gold in it (hire without filtering). Or they go down and carefully pick the prettiest pebbles hoping they will be gold (wrong filter / puzzle interviewing). But the only way to really find gold is to seriously invest in a filtering process that will yield actual gold. That means filtering based on their ability to do the actual tasks they'll be doing on the job.

The great thing about hiring remote folks is that I care 0% how they get the task done. I don't care if they've automated it, or have their mom do it for them, or whatever. If they provide the results I need, I'm happy, period.

There are plenty of other smaller caveats and gotchas to watch out for, but I'll try to cover those in a blog post sometime.

If you're a startup and want to go faster, try this out by off-loading some of the grunt work from your staff. It can be a big competitive advantage if you can do it right.

First, Continuous Deployment is really awesome.

Second, you should completely ignore Continuous Deployment until you have sane systems.

I do contract systems work for a living and have seen hundreds of live production systems. Nearly every system I see is in some kind of serious peril:

  - no backups
  - if backups, no docs on how to restore
  - no monitoring (or very little)
  - production passwords in the wild (former employees, etc)
  - no configuration management
  - no path to scale
  - no path to replace defunkt servers
  - etc... (I could go on for hours)

Often when I'm talking with a potential client, they are SUPER interested in Continuous Deployment (or whatever the hotness is at the time), but they start to yawn when I start talking about getting their core systems in order.

Your systems are the foundation of your application and your business. You wouldn't build your office building on top of an active volcano or below sea level on the coast, yet many businesses happily do this for their systems.

You know all those constant news stories about massive outages, security breaches, crippled sites, etc? I'd bet 99% of them are due to them failing at the fundamentals of sane systems. Having seen behind the curtains of so many companies, I'm honestly surprised there isn't more massive systems failure in the news.

This is a critical problem in the tech world. Does it affect you? A few of you will have amazing solid secure systems, but the vast majority of folks reading this won't.

If you're using a PaaS like Heroku or Parse, then you have most of the systems worries taken care of for you. You can breath a bit easier.

But if you are managing physical or virtual servers and aren't even using configuration management (puppet/chef/salt/ansible/etc), then your systems are probably in a very precarious situation. You might not think so, since everything just happens to be working at the moment, but you are essentially coding without using version control.

You would probably think that a developer that used email for his code version control is an idiot. You would be right. But consider that emailing code around for version control is actually better than what most companies do for their systems. Most companies set up their servers manually. They often don't even have any docs or even shell scripts.

At one client, it took them 2 weeks (!!!) to bring up a new server. The engineers thought it would take only 4 hours. Those engineers nearly killed that company.

Do you want to be a 10X or 100X systems engineer? Then use configuration management! After we put that client's systems in Puppet, we could bring up a new server in under 5 minutes. Yep, that's 4000X faster.

If you aren't using configuration management and don't know where to start, just use Ansible. It's by far the simplest and easiest to get going. I've written a quick intro to it here: http://devopsu.com/blog/ansible-vs-shell-scripts/

If you're curious about how the configuration management tools compare, check out my book: http://devopsu.com/books/taste-test-puppet-chef-salt-stack-a...