Hacker News Reader

Comment by Moissanite | Hacker News Reader

Very heartening to know other people are thinking about this. For about 4 or 5 years I've been warning about an equivalent problem in the HPC space, whereby even if on-prem is cheaper, companies will be forced to adopt cloud due to the lack of skilled bare-metal sysadmins.

I should note that these warnings aren't accompanied by much sympathy; companies and academic institutions both do a profoundly terrible job at incubating the skills they need, then bleat at their local governments about new graduates not having the skills to be employable.

View on HN

It will get worse when the manufacturers of the best performing hardware will be the cloud providers themselves :) I am afraid Amazon will slowly drown out independent CPU (and eventually other gear, but with CPUs the playbook is most evident) manufacturers.

Today, Intel (and probably also AMD et al.) offer them exclusive deals on and even exclusive models of their hardware, due to the insane volumes "hyper-scalers" procure. In parallel, Amazon offers their own CPU arch (Graviton) cheaper than x86 cores to users of their platform in order to make it more attractive - at the insane margin AWS operates at, it can make these instances a loss leader, no problem. As a result, those in the x86 market that do not have a cloud business attached will grow weaker and weaker in comparison, and most cloud-native developer mindshare will shift to the cheaper option (since most engineering work in cloud envs is about reducing cost, period), which is AWS/GCP/... custom CPUs. And you, the random shmuck with an appetite for powerful on-prem hardware, will be left with worse products to buy for a higher price.

I'm not sure this is actually true. Intel and AMD are not thrilled that Amazon and Google have so much market power as their buyers. They are willing to give you a lot of other value-added products and a TON of engineering help along with their CPUs if you are buying a cluster. They will also give you a decent price (not as good as they have to give to hyperscalers, but good), and try to establish a long-term relationship with you.

Also, cloud environments are rarely cost reductions. They are very frequently more expensive than on-prem deployments, but they are used because they are easy to spin up and down and they allow you to engineer things faster.

The 'cool' thing - from a vendor's perspective at least - is that running your applications locally gives you a lower confidence interval when production is a different chip architecture than developer machines, so you have to spin up more pre-prod machines to be really sure your code actually behaves. Or use some other strategy that all ends in the same place: more servers to concurrently run more revisions of your code.

> companies will be forced to adopt cloud due to the lack of skilled bare-metal sysadmins.

More likely that they're just not paying enough, and those skilled sysadmins would rather go to any other company that will pay them appropriately for their skills.

This is just another take on the "nobody wants to work anymore" rant that happens when companies can't understand why nobody is accepting their open positions.

I think both are actually true.

Companies are not paying enough, but we also do not have as many opportunities (for SysAdmins) to learn things.

Paid Internships, Jr Admin Roles, etc are fewer and far between, companies are looking for "experienced" people in a time where alot of the experience is retiring.

Dont get me wrong there are plenty of us Gen X's out there with lots of experience but most of us are also massively over worked with limited opportunities given for us to train the next generation, so pay is only 1 part of this problem, head count is the other

Even if you find a company that has good pay, they almost always under size their IT dept by 25-50%, and rarely promote internal training or continuing education, or offer entry level admin positions with the goal to skill up people.

It's worse than that, he's dead, Jim.

I think if you want to go back into history and look at things, we had sysadmins who were not paid enough, but fell into "give a man a little power" trap and showed a side of themselves that we would rather not see, fed and reinforced by getting blamed for anything bad that happened 'on their watch'. The cumulative generational trauma on both sides of that relationship sets us up for a situation where outsourcing IT to someone who can't sass back or harass us about our deployment plans seems like a good deal. Yeah it costs about the same and it'll interrupt our roadmap but I get to 'stick it' to the IT department. Fuck those guys.

Now it's been a while and people are starting to feel nostalgia for the old days.

That is the perception (an incorrect one) that alot of devs have, even though they have never done, and refuse to do any sysadmin work.

It was never, has never, and could never be about "power". I have always been a generalist, with one foot in both roles of dev and admin.

I am very good dev in 3 programming languages (not including scripting languages like powershell and bash.. php, C#, and JS are my strengths), and passable in about 8 or 9 others (go, python, learning rust now, etc). I hold multiple certs and years of experience in sysadmin roles.

There are very few of us out there, people with extensive experience in both. I can tell you, once you have seen both sides of that wall you come way with the clear view that neither understand the other.

DevOps was proposed as a solution to that, and if Devops would have been implemented as a Management sytle not a role it might have accomplished that goal but sadly most orgs just eliminated the valuable perspective of sysadmins, and as you said "outsourcing IT to someone who can't sass back or harass us about our deployment plans" which from my experience many devs need to have their plans pushed back on because often time they are widely inefficient, insecure, ill thought out, and expensive

I have to wonder, if many of the cyber security incidents we have seen over the last 10 years could be avoided if Dev's lost the chip on their shoulder, and listened to the sysadmin more, instead of just rejecting their point of view as a "god complex" like you seem to

It’s not incorrect. Inaccurate maybe, but not incorrect.

You know why all internet traffic goes over port 80 and port 443? Entire generations of admins who learned that their job was to say “no” instead of “how”. By the time the fingers got peeled away from that issue the damage had already cemented. Similarly ORMs also got a healthy nudge from DBAs who decided it was their job to protect the business from ever doing anything new. Today the priesthood of the DB is all but gone. Devs are expected to do those roles because otherwise nothing gets done.

And similarly, the Cloud doesn’t say “no”, it says “how much”.

Are bad things happening now? Bad things always happen. I read Applied Cryptography while I was still in college, and have worked at a lot of startups and in other small projects, so I am biased toward thinking devs should not be making other teams babysit them. Advice and control are very different things.

Edit: I’ll note this isn’t my opinion, but my opinion plus decades of listening to devs vent about people out of earshot. Sometimes to the point of sedition. There are some big frustrations out there and creative people steer solutions away from frustrations. It’s not an accident we are where we are. People voted with their feet.

>>You know why all internet traffic goes over port 80 and port 443? Entire generations of admins

Now we have conflated network admins and sysadmin. Different things.

the reality is that was largely ISP's not sysadmins that caused that by throttling, shaping and in some cases out right blocking traffic.

Throw in some laziness by devs as well, it is widely inaccurate to lay that out the feet of sysadmins.

> Similarly ORMs also got a healthy nudge from DBAs who decided it was their job to protect the business from ever doing anything new.

Lots of ORM's were written by people that were DBA's because they were tired of devs that did not know how to construct a proper SQL Query, and thought that Select * was the always the correct way to return data.

This idea that devs had to ride in to save the industry from evil DBA attempting to block everything is a complete revision of history

>>my opinion plus decades of listening to devs vent

I am not sure how a circle jerk is proof of anything? devs in many cases may have gotten rid of sysadmins, but now a new beast is emerged that is making life hell for both sysadmins and devs alike. The "Cyber Security Analyst" and if the devs thought sysadmins saying no was a problem, you have not seen anything yet. Unlike the sysadmin cyberSecurity will be backed by compliance, insurance companies, and standards organizations. They will have actual teeth, and actual power something the sysadmin never had

You're right about attitude. Internally, the Google SRE attitude is to never say no. Instead, what's drilled into you is "Yes, but have you considered...". Still, that only works when the sre org has the support of management, or that there even is an sre org to be supported. Anyone smaller and organized different doesn't have such luxury.

Every architecture I've been responsible for has been better because of having conversations with operational people in which they tell me how divorced one of my ideas is from operational reality. Nobody would build a cluster that way because X.

And then they started taking away my Ops people.

> even if on-prem is cheaper

Surely this calculation should include the cost of people with the required skills?

To quote TFA: "Some things are simpler, others more complex, but on the whole, I've yet to hear of organizations at our scale being able to materially shrink their operations team, just because they moved to the cloud."

Something I've observed for some time as well. If Ops was truly outsourced to the cloud service providers then why is devops such a buzzword/trend, hmm?

The complaint in the comment above was that on-prem was chaper but they couldnt hire people with the necessary skills. Obviously it is not actually cheaper if you don’t factor in hiring the necessary skills.

I suppose that is just another way of expressing my point; comparing hardware on a like-for-like basis, on-prem is cheaper. Once you factor in the hiring cost and adjust for the availability of those skills (since that is never truly reflected in the salary, in my experience) - then cloud becomes "cheaper" not in money but in opportunity-cost and reduced staffing-related business risk.

Generally, with cloud, you still need people with required skills. Engineers who write Terraform ("DevOps") aren't cheap. They certainly cost more than the old-school sysadmin who will set up a bare metal server with shell scripts.

You definitely don't want them using shell scripts. That's cloud-init/Ansible work. And you can do all of those through terraform as well.

Now your cost has gone up. There’s a reason I said “old school.” Shell scripts are perfectly viable for setting up a handful of bare metal servers.

curiously at the small chair I work, the professor goes against that flow and we do our small cluster all on our own. We also don't have much money (small), so dedicating probably half a year of state-funded PhD-time paid off (still quite academic around here) as we now have an environment newcomers can get work with in week 1. On top me and a colleague have a niche but interesting skillset pxe-booting a slurm cluster, running RoCE and NFS. Our softwaredistribution (as in "compiled sensibly"...) is arguably better than on some parts of the HPC-center (continent-wide academic TOP10 iirc) nearby. Also a first year PhD has some interest in that and will probably carry this on.

I don't have a lot of insight in academic funding and I know the meme that PhDs don't get paid a lot, but is it really better to use presumably very highly trained PhDs to do the grunt work of systems administration rather than actual research? Honest question here, it just seems logical to hire dedicated technicians for the not-research work?

there is no money for dedicated techs nowadays. Also hiring a competent sysadmin to do on-prem HPC-Linux might prove challenging. And while it is grunt work sometimes (but more so is academic research in lab, which to 90% could be rationalized by a labtech...), it's not too stupid if the people programming the system know how the system works. Aside from racking severs, that's essentially what SRE does, no?

> hiring a competent sysadmin to do on-prem HPC-Linux might prove challenging

Anecdata: I have 2 customers who have had open roles of this type for more than a year each. I'm pleased to report that they are edging their salary offers up into "reasonable" territory - but the fact remains that someone with that skillset would ordinarily be able to get paid twice as much at any of the 3 major clouds.

yes, pay is obviously an issue (likely also for the HPC-center - though I would take a job if they agreed that I can hire/apprentice people properly (a lot of the grunts education focusses on clicking through menus and nothing more today...) and then they let me alone doing whatever the current people did. Probably a fast road to burnout though)

Also noone is doing this then eventually (deskilling is real) and as I really don't want to work for any of the evil empires...

Reply on news.ycombinator.com