Hacker News Reader

Comment by paol | Hacker News Reader

I have a pet theory that the recent focus on CPUs with a high count of low-perf cores (this article is about intel but AMD has an equivalent offering) is being driven by hyperscalers like Amazon and the perverse incentive caused by their pricing structures.

Cloud providers have structured their compute offerings around dollars per core. This creates the incentive to deploy the densest possible core count processors, and zero incentive to make those cores faster.

It can't have escaped people's notice how slow cloud VCPUs are. My 6 year old crappy laptop CPU has the same single core perf as a cloud instance that costs hundreds of dollars per month. And the frustration is not just cost, when you actually need good performance for something that doesn't scale so well horizontally, it's hard to get it.

View on HN

The thing is, most cloud systems sit at idle-ish loads, sans a few select components (file inegestion, file conversion, AI inference, high load databases). These idle microservices needs bursts, not long strips of straight road.

Hence having a lot of low power cores which can be quickly scheduled and have ample bandwidth to the rest of the system is what makes them feel snappy on the cloud.

Another fact is, currently processors are really way more powerful and have more capacity demanded by the tasks. I have a new N100 NUC system. It has 32GB single channel memory, a paltry 500MB/sec SATA and another paltry 800MB/sec NVMe disk, and it's way snappier than my old desktop, which is more than enough for me (It's running Debian & KDE).

So having tons of E cores on a bog standard server won't hurt people who are trying to serve CRUD apps, but they may need a couple of P-core filled servers for DB, etc.

It's not very different from what Sun did with Niagara processors which moped the floor (and moped again just for the giggles) with Intel in web serving and small file processing tasks (500% difference, c'mon now).

The only downside is, we're going deeper on specialization cycle. Now, processors are the new accelerators, which are task specialized, and your server may not be just repurposed for something else on the other rack.

P.S.: I'm not a big Oracle fan, but the ARM servers they offer at OCI are way more snappier than the Premium Intel server I have at DigitalOcean. Result may change if I test their "long-straight performance", but ARM one works like local machine response wise.

> being driven by hyperscalers like Amazon and the perverse incentive caused by their pricing structures

Not really. They can and do price it otherwise e.g. with different instance types.

If we had to blame something then it's the move away from the traditional software licensing model. In the past, self-install software e.g. databases, OSes, etc would charge per core / CPU and that would be a huge chunk of the expenses added up. Now that we're free from this things are a lot different.

Having said that, the desktop side is still aiming for high-perf cores for e.g. gaming. It's not lost - just harder and harder to do.

> It can't have escaped people's notice how slow cloud VCPUs are

It's not that slow. It comes down to choosing the right instances and staying current. EPYC 4th gen single core on e.g. Amazon i.e. c7a is not the same as a 6th year old laptop CPU? It's also not hundreds of dollars per month (reserved / spot instances are the way to go).

Also this is nothing new. I remember the days of companies running their own VMWare data centers and those server Xeons were a lot slower. There are just heat and density concerns even if you had to do it yourself.

Funny enough c7a is just what we chose recently for our CI runners precisely because of the better CPU performance. It's only better relatively to the terribleness of the t* types though.

And it definitely costs hundreds of dollars per month. Just ran the calculator and 8 VPCUs (to keep the rough laptop analogy) comes out at $316.

> It's only better relatively to the terribleness of the t* types though.

It's never a value add to use the t* types. That's poor misinformation or at least outdated information that people keep spreading.

The advantage of c7a or Graviton 3 though is you get real cores. Most of the t* get you vCPUs, and 1/2 of it are just threads. It can be up to 2x faster in that sense.

> Just ran the calculator and 8 VPCUs (to keep the rough laptop analogy) comes out at $316.

Does your laptop have 8 real cores not threads that can 24/7 keep its peak speeds? Most overheat and throttle... are we comparing the same thing?

??? USD?

c7a.2xlarge (8 vCPU) spot instance pricing is $0.1032 = ~$80/month (us-east-1)

Even on just a 3 year compute savings plan and we're on $144.71/month. You're not required to commit to a particular instance type - just the spending, so all good?

This is excluding storage. Either way there's no reason to use EC2 On-Demand without a savings plan or spot. At least commit some of it a year and it's a big saving. Otherwise don't use AWS.

Side note: EPYC 5th gen has been announced and we'll likely see way more improvements on top. Pricing will definitely go down.

Also: this is just a rant on AWS. Hetzner, OVH or even Oracle cloud is way cheaper, like maybe 1/2 to 1/10 of this $316... it's not a cloud issue - just AWS.

You're slightly off in your understanding of VCPUs. All AWS instances "get" VCPUs. A VCPU is a single thread in architectures with SMT and a single core in architectures without. So for example c7a instances which are AMD EPYCs have 1 VCPU = 1/2 core.

They have is a page that lists this for every single instance type which I can't be bothered to search for right now.

As for spot instances, they're a great if you can use them, but you can't hold their price as being representative. Now using savings plan prices as representative is defensible, but it won't bring the cost below the "hundreds per month" order of magnitude.

Hyperthreading is turned off on m7a/c7a/r7a.

"One of the major differences between M7a instances and the previous generations of instances, such as M6a instances, is their vCPU to physical processor core mapping. Every vCPU on a M7a instance is a physical CPU core. This means there is no Simultaneous Multi-Threading (SMT). By contrast, every vCPU on prior generations such as M6a instances is a thread of a CPU core."

https://aws.amazon.com/ec2/instance-types/m7a/

> Hyperthreading is turned off on m7a/c7a/r7a.

You made me go check and you're right! Thanks for the correction.

I think it's far more likely that it's driven by the fact that most CPUs are sold in laptops; and P/E core split lets them have better battery life.

Those design decisions then bubble up to server space, not the other way around.

But the decision to make server chips entirely of E cores must come from some customers demanding core count density above all else. I'm suggesting where that demand comes from.

> (...) must come from some customers demanding core count density above all else.

I don't think you realize that that's not a niche, whimsical requirement that you can only explain with conspiracy theories.

For decades, all hosting providers have been paying a premium to have more computational power with a smaller size and power footprint.

This is not a AWS whim. All cloud providers want this, and have been paying a premium for this. If anything, AWS had the budget to roll out their own CPUs.

How long ago did AMD launched its Bulldozer line?

Modern CPUs are mostly limited by their energy budget/heat and smaller cores are more efficient. Also, if you have a workload scaling to, say, 8 cores chances are good that it also scales to 20+, allowing the usage of many small cores. Hence, having a few fast and then several smaller, more efficient cores should be also useful for workstations and servers.

There's another aspect to this, which I'm not sure even has a proper name. Misaligned incentives isn't quite it, more like... incentive scale misalignment.

It goes like this:

Let's say you run a company that has a critical database that cost $10M in software licensing, $100M in software development, and is central to your $1B business. Now go host it on a cloud server that has margins of $10 per core per month. If that cloud vendor can cut your performance by 10% to save $1 without you switching providers, they will do that, because across millions of cores and millions of customers, that's huge money... for them. Sure, you've lost $111 million worth of performance on $1.11B of your stuff, but that's your problem. The cloud vendor's executives are whistling a happy tune while flipping through a yacht prospectus. They don't care about your lost efficiency, they care about their single digit dollar margins.

You'll see this any time some huge company uses a platform, input material, or service that is very low cost relative to the value of the service or product that it goes into.

As another random example: the quality of the cotton cloth that goes into name-brand T-shirts. You can buy a $100 shirt that has 10c of material in it because some guy worked very hard to make it as thin as physically possible to cut that down from 11c. He doesn't care if your nipples show through.

Conversely, the companies with the best products in the world typically have good vertical integration. E.g.: Apple.

PS: All major cloud providers use processor "C-states" to save a few dollars per server per month on their power and cooling costs. Turning that off is the magic turbo button on database servers. I do this for kit that I have direct access to. You can't do this in the cloud, and never will be allowed to, no matter how many millions of dollars this costs you.

PPS: All cloud SLAs refund the cost of the service you purchased, not the value of the service you were unable to sell due to the outage.

That's just physics/scaling and pricing to represent it

Space is practically finite, if more of the same performance can be crammed in, that's still a win! Your laptop and servers are solving for entirely different workloads and constraints.

The egregious pricing has always been there, but the proportional nature makes sense. It's less perverse incentive and more... natural characteristic

DHH made this point a while ago when talking about this potentially springboarding a walk back to self-hosting/on-device software for companies. I think the main point was that running CI on a MacBook PRO happens about 3-5x faster than CI on the cloud. The benefit of things running faster on the cloud is tearing away.

> The benefit of things running faster on the cloud is tearing away

Was that the benefit? I thought it's reliability, scalability, etc.

I remember having a FAST shared server for CI. "Someone" was in charge of it. Every time it went down everyone lost 1/2 a day trying to ask "someone" to reboot it.

It wasn't switching to the cloud that saved your dev team. It was getting rid of policy/thinking that allowed the "ownership" of a critical service to be so opaque/siloed that it took half a day to reboot a computer.

> It was getting rid of policy/thinking that allowed the "ownership" of a critical service to be so opaque/siloed that it took half a day to reboot a computer.

No 1 got rid of any policy. Cloud meant any team could just take out their company card and sign up. Same as how SaaS got popular.

The problem is, what is critical? That server was non-production and there were 100s to 1000s of things more critical to deal with. It was low priority.

Maybe you know but to many up the chain in many orgs - they don't know. I've seen way worse scenarios - things on someone's laptop with ports opened that were production critical.

Cloud isn't the cure, you're right, but it forced a lot of change.

> Cloud meant any team could just take out their company card and sign up

> Cloud isn't the cure, you're right, but it forced a lot of change.

That's the thing - there are plenty of organizations that operate solely in the cloud but do not give their teams the authority/ability to spend without prior approval. And there are plenty of organizations that have no cloud presence that allow their teams the discretion to spend money when spending money is the right answer.

For a lot of companies, the move to the cloud was a super large benefit. I'm just arguing that the real benefit to those organizations was a side effect and that many organizations probably still don't realize this.

I meant for CI and other software that your company uses internally (e.g. not your own software offering to customers/clients). Yes for other purposes, lile hosting your software, the cloud provides reliability/scalability. But for Dev compute. That was the context of the DHH conversation. I would say the cost value still leaks in though: I have never seen a big three providers cloud costs per $ for a given machine ever reduce (excluding Spot bidding etc) - despite the fact the efficiency gains in recent years have been so fast it is making more and more sense to self-host either on prem or in a data center. Cloud was a lot cheaper in 2008 compared to buying off the shelf, not so much today.

> But for Dev compute. That was the context of the DHH conversation.

My point was exactly that internal use cases still has a reliability/scalability factor.

> it is making more and more sense to self-host either on prem or in a data center

I'd say it's the other way around. Automation improvements, infrastructure as code and less knowledge has meant Cloud now makes sense. Back in the days it was a lot easier to find people skilled in self-hosting and the common language was something like Redhat and not AWS. Self-hosting didn't meant getting internal networking and stacks enough to run kubernetes. You'd have a single server or 2 and that was it. It's a totally different game.

> I have never seen a big three providers cloud costs per $ for a given machine ever reduce

It has in a lot of ways, e.g. similar prices for faster CPUs (newer generation) and e.g. with AWS getting 2 real cores vs 1 (core) + 1 (thread).

The problem is server hardware is cheaper. Electricity, rent (data center) and wages is not. This growing cost eats into self-hosting a lot more than the hardware savings. The prices not going up is actually already a cost saving. Most data centers want to up your contract by at least inflation % per year if not more.

Thank you for the point on newer generations costing similar but delivering better; I hadn't considered that, I suppose my mind was on the serverless and Cloud API services, not machines under infra-based scaling.

Electricity of a Macbook M1 Pro running my build locally is cheaper than self-hosting or Amazon, as far as I can tell so while I get your reasoning - I'm talking about Dev compute.

Also MRKL and other tools (even yes for Infra as code) are making it to my mind more viable and easy than the old days. Good times are ahead.

Last I checked it is even worse if you need a fast NVMe. 120000 IOPS will cost like $7,800 on Google Cloud.

Pretty sure "one VCPU" is scaled to the core performance. So if they have a powerful core that has 1.5x the instruction throughput, it gets sold to users as 1.5 VCPU's.

At least with Azure that's not the case. Depending on the VM family one vCPU is one core or one thread.

Not in AWS. 1 VCPU = 1 thread (so half a core in architectures with 2-way SMT, which is most of them).

It is, except when it isn't. T2 etc are burstable and each vCPU seems to be guaranteed only 10% of a CPU thread(/core), even while it's called a "vCPU".

So, in AWS (and most others), a vCPU is something between nothing and one single thread, depending on how much you pay for it.

I wonder why IBM doesn’t do more with POWER10 in their cloud, because each core can go up to, IIRC, 8 threads.

Same for Oracle, BTW, but they are no longer investing on SPARC, just milking until it dries up completely.

This was tried before: https://en.wikipedia.org/wiki/SeaMicro

or it has escaped people -- how CPU's have gotten blazing fast.

we can't notice of coure due to how software is these days. your average desktop CPU might have 8Cores / 16 threads.

if you rent a dedicated machine n skip the 'cloud sharing' your average web app -- won't be able to exhaust a single machine with 128GB Ram to 2TB Ram and 64 core CPU. Now we're talking about 288 Cores with the Intel Efficient CPU's.

CPUs have had an enormous performance boost over the last few years and I do indeed think it's happened without many realizing.

For a big part of the 2010s CPUs were kind of stagnant since Intel didn't really have much competition and was happy resting on their laurels and dropping slow incremental upgrades (like a 2012 CPU and a 2016 CPU were basically the same), and AMD were floundering and playing catch up, but when they did catch up and really blow past Intel, it really lit a fire under Intel's ass and now there's real competition for x86 CPUs again.

I think it's a pretty favorable description of cloud providers that they will sell you 2015 performance for 2024 dollars.

It’s quite incredible how the cores themselves got faster - I was betting on that stagnation and core count increases, but, it seems, they have managed to delay that point for a very long time, and with no wall in sight for now.

Having said that, making software more parallel is still a good investment.

> Cloud providers have structured their compute offerings around dollars per core. This creates the incentive to deploy the densest possible core count processors, and zero incentive to make those cores faster.

I think your take is too conspiratorial. There are plenty of good justifications for this sort of approach. One of the selling points that stand out is that the bulk of the threads/processes don't have high computational needs and act more as event handlers, and typically they stay idle while waiting for IO, which represents the bulk of their workloads.

Why would you waste your budget on high-performance cores if most of what you do is sitting idle while waiting for data?

There's a reason why things like node.js are a success.

You may be right in general about my take being too cynical, but I have to counter this in particular:

> and typically they stay idle while waiting for IO

That's just not how you size servers. If most of your cores are idle you will add more work to the server, or you will have bough one with fewer cores to begin with.

There's plenty of work that will max out your NiC before maxing out all cores, or even one, with reasonably efficient code.

At some point youre just actually pushing 10 Gbit/s through a process that is using a few 10-15% per core.

You're missing my point. Servers are very expensive and are sized to task. If your NICs (or whatever) bottleneck the server with 32 cores worth of load, then you buy a server with 32 cores, not one with 64 cores.

Fair

> If most of your cores are idle you will add more work to the server, or you will have bough one with fewer cores to begin with.

I don't think you understood the issue.

It's not that cores will be idle. It's that the bulk of their workloads is not performance-oriented. They are there to context-switch while handling IO-bound tasks. You'd be wasting money if you pay a premium on high-performance cores if you're using them in IO-bound workloads. You waste money buying the processor, and you waste money powering performance-oriented cores to run workloads that do not benefit from this performance. For the same budget, you are far better off having a larger number of efficient cores than a smaller number of high-performance cores. Because you are running more efficient cores, you can cram more of them in the same package without ramping up cooling needs.

Think about it for a second: where is the performance bottleneck of your server if the bulk of its load consists of IO?

You should turn your conspiracy theory upside down and ask yourself how come a cloud provider is interested in running more cores in smaller spaces and in spending less energy cooling them.

> when you actually need good performance for something that doesn't scale so well horizontally

Truth is, you don't have the right to need fast cores any more. Those times are over, when you could write poor code and buy a more expensive CPU. Now, if you want your software to scale, it's up to you to make it scale horizontally. Either this, or suck it up!

Reply on news.ycombinator.com