I'll Probably Never Hire Another Pure SysAdmin
peebs.org
peebs.org
I totally agree with proper deployment procedures. But having them doesn't eliminate the sysadmin at all. Sysadmin should be deploying standard packages that are necessary for the system, along with updates for them. Developers should be deploying in-house code, and updates for that.
I worked at a company that hired a sysadmin too early. When he quit, I ended up doing his duties and mine, too. Of course, I automated 99% of it and everyone was happy. That was really early in the life of the company.
Later, when it came necessary to hire one, they didn't. They'd expect a developer to deal with the issues, and were always dissatisfied with the time it took, and the lackluster results. Eventually it got so bad they hired a dedicated sysadmin again. Now, unlike the past, they actually needed one and he was busy all the time. Developers stopped being unproductive for weeks at a time, working on projects they didn't understand and couldn't complete satisfactorily.
For a startup, yes, skip the sysadmin at first. But once you have a complex system, hire one. It'll save your company.
Sysadmins will always be a necessary evil when you are dealing with hundreds of servers, physical, virtual, or cloud. The tools and process will change. We have gone from physical Unix boxes (Sun, HP, etc) to virtual Linux boxes, to cloud servers, but sysadmins are still doing the heavy lifting of designing and building highly scalable systems.
I would say that sysadmins should look at moving into systems architecture roles, since one sysadmin will be able to manage hundreds of systems rather than a few dozen in the future. You really need architecture, networking, and performance tuning experience in order to succeed when you are managing more than a handful of systems.
My read is that he's saying that the days of the "system administrator" with a job description that begins and ends with tar flags and Apache vhost configuration are numbered.
He's not saying sysadmins are being replaced by Puppet and Chef; he's saying they're being replaced by a new kind of sysadmin that can be expected to extend Puppet and Chef.
This seems closely related to a point Patrick McKenzie makes all the time, that being the difference between a "programmer" and "someone who provides business value by means of applying specialized skills with programming languages". Don't be a "programmer".
As a sysadmin, I'm fine with the concept of continual improvement, there is nothing worse than a static industry. The job is changing every day and this.is.good.
What I understand from this post is that startup people don't want to deal with sysadmin anymore. It makes a lot of sense, considering the amount of time/energy/money you need to run a stable infrastructure. But at the same time, heroku, ec2 & al. will never be able to cover everybody's needs. So while I'm convinced that a dev or devops will be fine deploying his webapp to some random cloud, I'm also convinced they will come crying when massive load hits them and they can't tune the system.
The sysadmin is not only the guy who configure new servers, it's also the guy who explains the new dev dude that, no, he can't do 15 inner joins on large tables to create the home page because it will stress the IOs so bad that the whole website will collapse.
Ease of use != flexibility.
Although in the case of SQL, specifically, database administration has historically been a specialized sub-category of system administration.
Yes, probably. That doesn't mean your role on the project, in the company, or in your job description isn't "Database Administrator." Or operations engineer or reliability engineer or whatever fancy name you want to call the fact that you are more responsible for running the system than building it.
A more realistic example, though, is that a DBA and developer collaborate; the DBA collects and analyzes data from the production system and shares this information along with recommendations to the developer, who is ultimately responsible for restructuring the query.
In a very small team there's probably no need for these roles to be divided.
When I was interning at Google I noticed that these roles were differentiated as SRE and SWE. I didn't notice a lot of competency difference, just different application areas of the same competence. I think many SREs and SWEs could probably switch roles successfully.
I never had a google internship or anything similar, and my first technical job out of college involved supporting an internally developed, browser-based ActiveX/.asp/vbscript tool for viewing and annotating loan documents. For a long time we on the systems team were unable to upgrade IE on user workstations because of a critical, easily reproducible bug in the document viewer tool. Eventually, I managed to get access to a snapshot of their codebase and tracked down the bug in a few hours (despite never having written a line of .asp or vbscript in my life and not having access to any of the documentation for the 3rd party libraries and activex components). The devs, of course, didn't listen to me anyway, despite my basically handing a solution to them. The bug remained for another year until that section of code was incidentally re-written to support some other feature.
The next company I worked for, a web company, had a much better relationship between development and operations. It might have helped that we had a lot of Indian developers, who didn't seem to have the same prejudices that western devs do: that system administrators are people who just aren't skilled enough to do development.
About ten years ago I remember talking to some old timers who were surprised that a lot of the younger sysadmins no longer even did any perl progamming, but rather did things like program Cisco rounters and set up Oracle servers.
At Google we don't have sysadmins at all, but rather "Site Reliability Engineers", and they are expected to be able to answer some fairly deep C/C++ data structure and algorithms questions. All of those skills are necessary to run a Google-sized infrastructure, and it's quite a far cry from the sysadmins that don't even write perl scripts any more -- and yet feel perfectly free to call themselves "sysadmins".
When I first started, I both wore a pager and wrote C programs --- including the 'ninit' program that was described in the Unix haters handbook. The fact that I was one of the people who might be woken up at 3am when the mail server melted down due to named core dumping informed how I wrote my C programs (i.e., very defensively), and is a prime example of the dev/ops concept.
But why are so many devs shooting at the sysadmins these days ? I never read anything from sysadmins complaining about the astonishingly low level of knowledge of a lot of devs. And they are numerous, willingly ignoring even the most basic performance issues and vomiting thousands of lines of codes, hundreds of tons of libraries, to simple problems.
It seems that devs are frustrated to not be able to push their random app to the web in 5 minutes, and throw their love to PaaS like an apple geek discovering the new iphone, blaming the sysadmins for not providing the same thing at their own company without even understanding (or trying to understand) how operating systems work, and how their own code should be built.
The real issue here is that when devs and sysadmins DON'T work together, and ignore each others problems, constantly blaming each other, then the result sucks. When the sysadmins are 20% devs, and the devs 20% sysadmins, and they drink their coffees together in the morning, you don't see that kind of problem. They built good, reliable, scalable infrastructure, whether it's in the cloud or in the basement of their house.
They should be. At my current employer we're pretty good about this, but in previous misadventures I've run into a lot of people who don't even understand HTTP or how a web server works (if I have to explain thread-per-connection to a guy who's been doing web development for five years, there's a problem!).
Remember, your developer spends their time making it work on a dev box, or even their own laptop. Performance testing and such is an art that if they had time to learn it... well I guess they'd be a sysadmin.
If you actually give a shit about web development beyond a paycheck, this is stuff you just gotta know. If you don't understand the system, your assumptions and assertions are so much more likely to be untenable that it becomes dangerous.
The below questions wouldn't fly at my current employer due to a lot of legacy concerns, but my own list of interview questions would include a number of stack questions in addition to programming questions:
-Explain asynchronous I/O to me. -How do you have to structure your code to leverage asynchronous I/O? (Alternately phrased: what happens if you write code that depends on synchronous I/O operations within an asynchronous I/O web framework?) -You've got 9 memcached servers with a cache hit rate of 80% resulting in a database load of 80%. What happens if you naively add another memcached server to your cluster? (a question our VP of engineering actually does ask, and I stole it 'cause it's great)
You don't have to be a hardcore nginx source code diver to understand your stack. You just have to understand the principles behind it well enough to make good decisions. A sysadmin's job isn't to audit your code for your terrible understanding of the model on which your stack is predicated.
First, I don't think it's good for a developer to have all the crap in their head that I do in mine. They'd be spending way more time pre-optmizing code than they would just getting stuff done.
A good developer will write a well thought out TDO that infrastructure and business units can provide feedback to.
Your questions about asynchronous i/o and memcached are great, and if memcached was a requirement for a sysadmin at a position it would be a good question to make sure they understand the software.
As a sysadmin though, you're also balancing
- Can the new system exist on an existing server/chroot/virtual machine - What monitoring is necessary for the application afterwards - Does the application meet all the standard requirements for database access ... out PFC is over 100 steps, this is just some of the bigger ones
- Can the sysadmin support the system after go live? The developers job is to create it, the sysadmins job is to keep it running long term.
Also, you are correct. The sysadmin shouldn't be auditing your code. The sysadmin should have a stack in place that they know works, and be able to help you do one of 2 things:
- Give advice on how your system can best fit in that stack - Help you change the stack to best fit your application
Which direction you go on those two choices depends on the nature of the application, how mission critical is, required timelines for launch, budget and more.
I don't say this stuff just for show--I've been there and done that, and it's stuff I've done to become what I consider an acceptable developer. I can, and have, set up a three-tier fully-enterprise-up-the-ass application infrastructure on my own (I have a couple machines on a VLAN at home as a VM farm) in order to learn this stuff. A developer who can't work the stack is a developer I don't want touching my code, because they're a developer that doesn't understand what they're doing actually does. Abstractions are great, but every abstraction leaks--they should be treated as productivity enhancers, not as insulation from knowing what you're doing.
Nobody can know everything about everything (there's a guy here at my current job who has forgotten more about the JVM than I've ever learned), but you gotta at least be able to talk about it. If a developer can't explain to me high-level concepts like how your web server handles threading, I don't even know what to say. This stuff is just so integral it's not even funny. You just gotta know it. Specialization is for insects.
.
EDIT: I think you and I are coming at this with very different philosophies. At my current employer, we don't have a PFC. We don't have TDOs. We have two live-site guys and a few operations engineers who are able to jump into our codebase (one is probably the most knowledgeable developers at the company who wanted a change). We have developers who can be trusted to make good decisions, and we have senior devs and managers who are very well equipped to analyze the plans and ideas that come from junior devs to say "hey, that won't work." But to be able to trust developers, they've got to understand how stuff works. (We don't ensure full-stack knowledge as well as I'd personally like, but we do a pretty good job of it--at the least, there are no "front-end engineers" who can't sling Java and no Java guys who can't write HTML/CSS/JS if a project comes along that requires it.)
The deck is stacked against sysadmins - developers are the people who deliver business value to leaders, and sysadmins are the ones who cost money and say 'no'. When you are on a tight timetable, the person who says "sure" is the one who gets the high fives... assuming things work out, of course.
This happens when the company discovers that when the system goes down they stop making money. If you've managed to design and build a system to be very resilient and reliable on the chosen platform and has proved to need minimal maintenance, then chances are good you have a very small ops team who doesn't get in the way much.
There are plenty of DBAs complaining about the low knowledge level of a lot of devs, e.g. using an ORM instead of writing SQL and wondering why the app runs like a dog.
I would say, 25 years later, the need to fix C bugs is far less relevant. That said, I'd say around 50% of the system administrators I've worked since moving into the Unix world know C and can at least read the source code of the operating system they're supporting. Even if there's not much need to write anything or even submit bugfixes anymore. Also, the number of unix admins I know who don't know a scripting language well enough to write a reasonably-sized, useful tool is vanishingly small, and those few have been very solid with some other technology(network, storage, etc.) and valuable anyway.
"System Administrator" is a broad term, just like "Developer." I'm not sure google has either, right? If we're going by fancy titles, Google has "Software Engineers." I'm not going to criticize Google's culture but the fact is if I say "Software Developer" you know what I'm talking about and from context should be able to tell the difference between Linus Torvalds and a guy who creates wordpress sites for his friends.
There are pure SAs around, and I don't just mean the caste system. The entire sysops organization was just one example. They may have been rolled up into another organization but the people still exist.
tptacek's comments sum up the point - hence the qualifier of "pure". I think the SysAdmin role is rapidly evolving to look more like a programmer's job description from several years ago.
Sure, someone still has to write the puppet configs and monitor and manage the deployment, but that person is a far cry from the skillset and job description of 5 years ago.
Who will make sure your provisioned configurations have proper redundancy?
Who will move your domain when an availability zone has an outage?
Who will work with upstream networks when you're getting DDoS'd?
Who will wake up in the middle of the night when a service you're integrating with is not working?
Maybe you've never run a large system at scale, or maybe ou just had bad experiences with "SysAdmins" that didn't, but the role of hand-holding a complex, integrated service is not going away. The fact that you may not need screwdrivers for certain systems is almost beside the point.
Nevermind that, when you're at sufficient scale, you can run your own data centers cheaper than Amazon. Break-even happens well before a thousand nodes.
(I'm not a sysadmin; i'm an engineer who is very glad we have good sysadmins around :-)
Sysadmins do a lot of stuff, and this is only somewhat reduced by cloud services. Or is he expecting to get the developers to maintain OS versions, security patches, log monitoring, proxy servers, database replication, backups, firewall rules, failover testing and so on across dozens or hundreds of systems? Right.
Absurd.
Currently, at $DAYJOB, our database is growing by about 1GB/day, and our storage is growing about 4GB/day -- in a downward housing market. If/when the housing market kicks up, I expect to see those double, if not triple. Having someone in house who can keep the right people informed of the risk (of running out of space, of buying the wrong equipment) who is focused on keeping of the needs of the company balanced with the needs of the software is critical -- do you honestly think your cloud provider cares if your customers can't access their data during an outage? If you're a small fish...probably not...
EDIT: Spelling/grammar change.
So there's less hardware replacement and more DNS configuration but none the less there's still a job for system administration.
In fact, I'd say the role of administration becomes even more significant because the cloud industry is so new. Hardware has, in many ways, become pretty rock solid in the last decade. We're to the point where a hard drive in a RAID setup can go out and the users won't even notice.
The same can't be true for things like SQL Server Federation and caching servers which used to be the exclusive domain of large web sites but are increasingly being used by smaller companies who don't have a whole staff to dedicate to their upkeep.
(One Caveat: If cloud models like Microsoft's Azure take over this could change. By that I mean clouds that are literally frameworks in themselves and not virtual machines)
I manage 2 dozen Ec2 instances of various types as a side task to my regular development job. I script everything. When things are scripted they are consistently reproducible and fast. Humans make mistakes (including me).
I would prefer there to be a dedicated SysAdmin but there just wouldn't be a full time job for him to fill.
I have apparently been spoiled somewhere along the line, because I really don't like having clumsy manual processes that involve software/computers. I was recently tasked with figuring out how some researcher's code worked (with an eye towards automating the use of their tools), and it took 8 pages to get most of the steps documented well enough that I could follow them. Steps like, "edit the path in line 977 to point to your input files, recompile, run, click button A..." -- i.e., things that should never have been done manually to start with.
How on earth people whose primary job involves writing code to do things for them can fail to automate tedious (and frequently error-prone) parts of their job is beyond me.
That said, many companies are still going to need to hire somebody to write the one-click deployment. Somebody will have to know how to interact with the cloud services, and how to get around all the little minefields and gaps left in the cloud service offerings, and odds are you aren't going to want to pay that person as much as you pay your developers that do the "real work." That's probably where the sysadmins will go.
It is not until you appear that it's time for the app to be deployed elsewhere.
Reminds me of the 'No true Scotsman' thing: http://en.wikipedia.org/wiki/No_true_Scotsman
This is true, but it's also a lot like saying you don't need to hire developers because you can outsource your development to Lithuania. Really what it means is that it's possible to outsource systems. If you're the little guy and your needs manage to mesh well with what you can get off the shelf from a cloud provider, this is true. Otherwise, this isn't true. If you're the little guy and your needs don't exactly fit what you can get off the shelf, it's going to be trickier. If you're the big guy, you can afford to cut out the middleman.
However, I agree with the basic contention - that the role of a pure, old-school, risk-averse sysadmin is obsolete. In a world of Agile developers practising continuous delivery (or hoping to), the system engineer is now another cog in the development machine; not part of a separate machine.
The resulting closeness does wonders for understanding and quality of the result. Unfortunately, at the time of writing, system engineers who speak Developer are relatively few & far between.
I'm a sysadmin in a 100% AWS shop. Before I showed up, Dev was running things. It was horrible.
Versions differed greatly, they didn't use init scripts for anything, deployment was pretty manual, and their release cycle was really slow because they spent too much time in operations where they were clueless.
My job is to keep production rock solid and provide Dev with the resources they need to write code.
If Devs aren't writing code, I failed.
a) Deployment of 'standard' applications like Drupal, Joomla!, Wordpress, etc. It makes sense to automate it once and let people "self service" In this case either the sysadmin is responsible for packaging those apps or they can get a third party "catalog" that you can deploy to your virtualized/cloud infrastructure (that's exactly what we are aiming for at BitNami, you can check it out http://bitnami.org/stacks/)
b) Complex multi-tier, in-house applications. Those are much harder to automate because they tend to be "one off" systems and will require 'traditional' system administration for a while.
c) Web companies that run a logical application across multiple servers (like Zynga, Netflix, though obviously most people will do so at a smaller scale). Here you need to use something like Chef or Puppet, there's simply no other way of doing so.
In other words, before the advent of cloud it was 'optional' for system admins to automate their work. Now it is expected. In that regard, I think automation is the real benefit that cloud computing has brought us (rather than the capex/opex and autoscaling that people often mention)
I'm building a web based company right now with a good friend of mine and we've gladly chosen GAE because of this single attribute. Both of us are 15+ year sysadmins as well as software engineers. That said, instead of worrying about how we are going to scale and run everything, we are able to focus entirely on the development of features and building the company. As a result, we hope to never have to hire any IT positions.
In the end, I don't think the roles will go away since someone still needs to run Google. ;-) I just think that the roles will eventually morph a bit for the companies that can take advantage of PaaS solutions.
From my experience, sys admins are important. If you are doing anything of scale, knowledge on how to setup, tweak, and debug systems is critical. It is helpful if the admins can stretch a bit into the app layer, just as it is helpful for developers to be able to stretch a bit into the systems layer, problems come up and knowledge of the parts helps.
As an engineer that can also setup systems for scale, I know my limits and really appreciate help from and learning things from good sysadmins.
If your business only requires a standard Rails site, there may standard tools. However, when you develop new technology, someone has to deal with deployment and scalability issues. This may be more of a developer job, but there are significant sysadmin issues.
Faster: Cloud adoption will be huge.
Slower: Cloud technology will increase the amount of apps and data used. Low end jobs will exist helping firms get on the cloud. High end jobs (fewer though) will exist at the cloud providers.