Hacker News Reader

>In 200X lots of companies were able to maintain their own infrastructure, just like Facebook and Amazon did at the time.

But my point is that companies' IT departments did not maintain infrastructure just like Amazon did.

In ~2005 when companies were first experimenting with AWS cloud, they might start with dev & test servers. They click a few buttons and are amazed when new servers get spun up in minutes and their programmers are productive immediately. The natural question that company execs ask is, "why can't our own internal IT department spin up servers for us in 10 minutes?!? Why does it take them so damn long?!?"

They wouldn't have asked those hard questions if their internal IT capability was equivalent to AWS. Eventually, their improved experiences with AWS on Dev&Test&QA convinced them to migrate mission-critical Production workloads to AWS as well.

>We have cheaper hardware. We have (supposedly) better software.

You're still focusing on on hardware+software and not considering the IT employees' speed of execution in how company executives compare the situation.

Even Netflix as a tech company maintained their own datacenters for over 10 years but ended up migrating to AWS. Their "Guardian" moment was a big database corruption in 2008. They also had ambitious plans to expand into countries outside of USA. Those were some of the factors that convinced them they didn't want to invest anymore in their own datacenters and preferred AWS take care of it. Amazon employees iterated on datacenter capabilities faster than Netflix engineers could do it.

netflix does not depend on aws to deliver their video bits. they do it themselves based on a big network of core and edge pni and caches. this infra, that netflix has built, in datacenters, is a big competitive advantage. it would be worse performance and much higher cost to do this over something like aws cloudfront.

>this infra, that netflix has built, in datacenters,

Unless things have changed since 2015, reports say Netflix eliminated their last datacenter already.[1]

The Netflix "edge appliances" for CDN streaming are located in others' datacenters owned by Verizon,Comcast,ISPs,etc.

[1] https://arstechnica.com/information-technology/2015/08/netfl...

That's still in datacenters that aren't Amazon's, Google's, etc. It's the most critical component of netflix (the actual video delivery) and it's not "in the cloud".

>That's still in datacenters

The point isn't that they are still _in_ datacenters. Yes, of course, they are. Even the "cloud" ultimately resolves down to somebody's datacenter somewhere. The point is that Reed Hastings & Netflix wanted to get out of managing their own datacenters.

Putting their Netflix appliances inside of ISP owned datacenters still lets them avoid managing their own datacenters. Their critical user accounts signup, monthly billing, and analytics, etc workloads are at AWS. And as the article mentions, even updating the cache on the Netflix appliances is coordinated through AWS. The combination of those strategies keeps them out of the "datacenter business" and let's them stay focused on their core competency of "video content".

The distinction makes important fault domains and whose data centers.

If you are a Comcast customer and your internet goes down, and Netflix is unavailable, who do you complain at? If you're smart enough to notice that both are down, the answer is almost certainly not Netflix. That does not make the Netflix workloads in Comcast data center any less critical, they are core business functions. But they are well aligned with Comcast, who also depends on the proper functioning of those datacenters.

It makes sense for Netflix appliances to be in Comcast datacenters then, especially given that Comcast cannot outsource their data centers any more than the Pentagon can reasonably do so.

Joe Company from off the street can outsource their data centers and derives no competitive advantage from maintaining their own private data centers. Netflix in that sense is closer to Joe Company than they are to Comcast, I guess. I'm not sure what all we can learn from this, but it's interesting.

Netflix employees still walk inside those datacenters.

I’ve avoided my own datacenter by using Equinix, for example. I wouldn’t call that “cloud” though.

The 'web' and browsing part is on AWS, the CDN is their own not on AWS.

The Netflix example is a poor one. They have a friggin fleet of CDN-boxes around the world that are as locally close as possible to eye-balls. To say they are "100%" on AWS is disingenuous - they still have their own hardware deployed all around the world - just closer to the edge.

AWS is used for control-plane / management-plane tasking - but the bits are pushed from locally-peered CDN boxes at all kinds of ISPs.

>In 200X a lot of companies were able to maintain their own infrastructure, just like Facebook and Amazon did at the time.

I worked at a decently large tech-ish company at that time. It took 3 months to provision a handful of dev servers in the data center. After filling out forms in triplicate. We asked for extras since we had almost zero ability to manage them (reboot, etc.) or get them re-built if something broke. No backups of any kind as far as I knew. We once had one break at night and apparently someone had to psychically drive there at 3am to reboot the thing after we filed a ticket. Used some sort of in-house distribution of linux that had it's own quirks.

I'll take the cloud any day of the week.

That sounds like a problem with process, not technology.

I can provision a handful of production servers in 5 minutes.

> In 200X a lot of companies were able to maintain their own infrastructure, just like Facebook and Amazon did at the time.

I would question that equivalence. Speaking as someone who was trying to get what we now call devops going at the time, there were a _few_ companies starting big wins from automation and a ton of places which were content pouring huge amounts of human time into doing it the hard way and slow metrics for time to patch, provision new services, recover after failures, etc. When they had automation, it was known-bad practices like cloning VMs and had accordingly greater cost and lower benefits.

The companies in the latter groups were the ones who were faced with either spending a huge amount of time and money catching up or cutting a cloud contract and not having do a large percentage of that remediation work at all, while being able to deliver results immediately.

Ok, I will try to counter that. The stack grows higher. More specialized employees become more effective. In 200X you just needed a Linux admin. Now on top of being good a good Linux admin you also need to know docker, kubernetes (the admin part, not the user part) etc. If you don’t need that daily, because you also need to maintain some apps, you will be less professional than a cloud provider.

Of course it’s a trade-off.

If you’re large, with constant high resource needs and benefit from custom optimization, you may be better off with your own team/infrastructure.

If you’re small (2 it guys), fast changing, normal speed requirements / easy to scale out then the cloud should be interesting to you.

Really good points. Maintaining your own infrastructure may be "boring" with "old" technologies. But its a known process. Its really easy.

But with the cloud its a whole different domain. It may be easy too, but I don't think its as easy as hosting your own.

And your last two lines are great. It really comes down to the company about which tradeoff is appropriate.

I worked at a couple of those self-hosted companies in the 200Xs. For me, as a developer, it was painful and way, way, way worse than AWS or GCP. Planning for big expected spikes of traffic was extremely painful, and once the cloud vendors started adding new services we felt continually behind the times.

I can only think it was "interesting" if you've forgotten how difficult and expensive managing your own infrastructure can be.

It's a false dichotomy (or at least a false equivalence).

So, there were a lot of, and still there a lot of corporations running their own stuff. Some of these are still running their own stuff well. Some got big, some stayed small in the past decade (some downsized).

It's not necessarily hard to run your own stuff either.

Of course, of course, managed platforms are easy (or easier) from a lot of aspects. (That's their value offering after all.) And the IT expertise market was never exceptionally great in a lot of places. Meaning it was hard to find good sysadmins, good ops people, good devops folks, good programmers, leads, tech PMs, POs, etc. who were able to work together and effectively run their shit, build their product/service, run with the flow (of open source or whatever vendor they used), and so on.

AWS/GCP/Azure is a big paradigm change. No longer do you need to argue about what and how, at best you pick one of the three, and that's it. No longer you need to think about hardware, colo, uplink, switches, peering, multi-homing, picking the right dedicated server provider, and so on. (Sure, you need network guys, who at least have some basic understanding of VPC - overlay networking, but the basics are easy, and the rest is YAGNI anyway.) And this is due to their size. They are big enough, that their price premium - compared even to whatever random small hosting provider you might find on the 'net - is irrelevant, because they are usually more efficient, more secure, more geo-available than the small ones.

But, ... of course, while generalizing all self-hosted companies (meaning that all self-hosted is a pain) leads to a falsehood, on the margin, especially during the IT consolidation of the past years, most companies moved away from self-hosting. And developers usually rejoiced. And similarly, generalizing the problem of self-hosting from your experiences again leads to a flawed conclusion (that it's necessarily hard and expensive to manage your infra), on average, outsourcing a cost-center in an ever increasing complexity world (that is potentially unbounded costs) and focusing on profit-centers is a sane decision.

> it was hard to find good [key job roles] who were able to work together and effectively run their shit

Having had to interact with sysadmins and others, first at at a university and then in business, "work together" is too frequently overlooked.

The crusty, angry and rude stereotype proved right. I've never met a bigger group of downright bastards. Those who were nice people usually couldn't get the servers/software to work reliably. So you were forced back to group 1.

Spiky usage patterns are great for cloud.

Otherwise, I’ve did some tests years ago and what I spent over 5 years would have gotten me only a few months on AWS. And I think I only walked into the data center once in those five years.

When I did tests and studies for our new infrastructure, AWS was at least 2x as expensive per year. And in the 4-5 year range, it made it 3x+ expensive.

His whole comment was about how it's not about hardware cost, but staff cost.

> It's in-house IT employees' speed of tech innovation vs Amazon's engineers'

The cloud shrunk the cost of IT team. Sure hardware is cheaper, but that cost is marginal versus employee cost.