Hacker News Reader

Comment by pedoh | Hacker News Reader

I think tearing down walls between SysOps and Developers is important. I don't like the "this is mine, stay out" attitude. That being said, I'm all in favor of dependable and repeatable environments, and when you give people carte blanche to do what they want on the boxes, that can be a recipe for disaster.

"Oh, I just upgraded this package, yeah, can you put that in our master image?" sigh

Do it right, and you can get the developers all the info they need without having access to the box itself. Not only that, but you make it easy to spool up new dev environments for new employees.

If access does need to be granted today, then thought should be put in to see if that can be avoided tomorrow. Need a tcpdump today? Great. Access granted. Tomorrow, I'll have a script for you that takes a tcpdump and puts the file where you can access it. Access revoked.

View on HN

Dependable and repeatable environments are essential.

At my company the issue is that dev/qa/stage/prod are all not ===. We end up with two options to fix something. Make a fix and send it through the deployment process (over and over again). Or sit in a sysadmins lap and try to fix it. Both options are extremely frustrating, for both parties, and generate a lot of animosity.

It's not easy, is it? Imagine being able to spool up a new environment (on one server as opposed to, say, 5 in production) and feed it a copy of production data, run tests, and then tear it down. And imagine knowing that this environment was identical to production in every way except for minor configuration changes to get it to run on one box. And when you're done with it? Throw it away. That's really powerful. A new employee comes on board? Here, here's your environment. Mess with it as much as you want, we can just rebuild it if you totally hose it. All QA except for performance testing, which would need to be done on real hardware anyway, could be done in these environments.

I worked at a company where we had a similar system set up. We had automated database snapshots of production taken every two hours, and stored locally and in our account at Rackspace's Cloud Files. Eventually, I got fed up with people asking me to update staging with a copy of the production database, so I wrote a simple Python script and stuck it in the repository. As long as you weren't running in a production environment, it would give you a fresh copy of the database in a matter of minutes.

We were (slowly) working towards a system where the entire app (gems and all) was self-contained, and we had a list of all the system packages that needed installation; once we reached that point, setting up a new server would have been a five-minute job - a git pull, a package update/install, and a database pull. Sadly, we never did get around to finishing it (actual work took priority), but it would have been nice to have.

Reply on news.ycombinator.com