Hacker News Reader

Comment by modzu | Hacker News Reader

how is a 300W cpu cooled in a server environment? just high rpms and good environmentals? ive stayed on intel with my workstation so i can keep a virtually passive and quiet heatsink without having to go water

View on HN

When I worked in HPC in Tromsoe, arctic Norway, one of local supercomputers was liquid cooled and the heat used warmed up some buildings on the university campus.

Servers are in 4U rack mounted enclosures with (relatively) low height heatsinks and huge amounts of airflow. Intake in front, exhaust in rear. Most clients will have beefy air conditioning to keep the ambient intake temp and humidity low.

I think 4U servers are quite rare these days, except for models designed to accommodate large numbers of either GPUs or 3.5" hard drives. Most 2-socket servers with up to a few dozen SSDs are 2U designs.

Y'all can try to pry my quad-cpu 3-upi 4U HPE DL.580 from my cold dead hands.

Nobody wants to throw their back out lifting that monster, so you win.

:-) thanks, gunning for the 16 T4s to put in there, then.

Agreed. My last company we used a 1RU with 2 blades (servers) in that space. Front to back cooling, quite loud. No need to worry about the noise in the DC. Power density in a rack vs computer density is all that matters.

Yeah, noise is relatively unimportant so they tend to use a ton of extremely high RPM fans and big hunks of copper, from what I've seen.

And the DC staff wears hearing protection when they're working among the racks.

That still blows my mind. Coming from telecom where everything prior to the #5 ESS was convection cooled, a happy office is a quiet office.

Data got weird.

I'm assuming telecom had very different volume-power requirements. Where I grew up there were many mid-city phone switches that were large, concrete exterior, almost windowless, buildings.

I wish that hearing protection had been required or at least offered when I used to visit data centers frequently. They made a big deal of the fire suppression training, but never even suggested ear plugs. 20 year-old me had no idea how bad that noise was for my ears. I hope the staff there were wearing plugs, but it was never apparent.

You only get one set of ears.

I suspect I still have the record for most expletives used in front of the headmaster at my old school because someone turned a few kW speaker on while I was wiring something under the stage - i.e. I wasn't pleased.

To put it in perspective, the Intel Xeon 6258R is rated at 205W. Servers have oodles of (likely airconditioned) airflow so this isn't a problem.

Performance per watt is better on amd right now, at least last I heard.

Performance per watt is better but unfortunately the idle power consumption is fairly high.

Oh I see that in the article now. It also seems like the performance gains you saw on other Zen3 chips doesn't correlate with EPYC due to the sheer number of cores and other components on chip.

Epyc's idle power consumption is fairly high but Ryzen's isn't. The more workstation-focused Threadripper & Threadripper Pro is also still significantly better than Epyc here.

Zen2/Zen3-based, non-APU Ryzen still has pretty awful idle power consumption compared to Intel desktop CPUs.

My 5900X at idle draws around 22W. Modern Intel desktop CPUs (which are basically all Skylake refreshes right now) are 8-10W at idle.

Sure, but 22W is still a trivial amount of heat & power in the form factor of a desktop system. Fans don't need to spin for that (or if they do it's at the lowest RPM - still silent). Fans do need to spin, and with some moderate oomph, to handle 80W, though.

Is TR Pro really better? I suspect that it's just difference between motherboard because it's basically same as EPYC.

TBD on TR Pro using Zen 3, but TR Pro on Zen 2 appears to be a bit better than Epyc currently. I'd guess because optimizing idle power draw on TR Pro is a more worthwhile use of time vs. servers, where your fans are running at a bajillion RPM always anyway and you're not idle as much.

By an array of fans specced like 8cm 10W 10krpm, like six fans wide two fans deep, blowing into a passive heatsink, with help of air ducts inside the chassis

Intel or AMD or SPARC or ARM the setup is all the same for rackmount hardwares, pure copper passive heatsink and high power axial fans

Typical servers are 1U or 2U high boxes. 1U=44.5 mm and each box is "full depth", so around 700 mm long (sometimes more), and ~450 mm wide. The 1U boxes typically just have a block of copper with a dense bunch of fins on them as a CPU cooler, while 2U designs usually incorporate heatpipes.

Lots of airflow.

High-density systems are usually 2U, but with four dual-socket systems per chassis.

Servers are horrifyingly loud, and datacentres will destroy your hearing in no time flat. Airflow management in datacentres is quite the art form, as well: proper packing of the racks for airflow, height of raised floors to accommodate blown air, and so on; some people are starting to go back to the future with things like liquid-cooled racks as well.

Don't forget that you could feasibly use a 280W 64-core processor to replace a dual-socket 2x205W 2x28-core Intel box with 8 cores to spare. Obviously, there will be performance deltas based on specific use cases, but this should help frame the comparison.

a cooling unit provides a delta between surface area temparature and the ambient temparature. Epyc chips are significantly larger and more spread out with the chiplets so the heat density of these chips are relatively similar. So the cooler doesn't have to provide a larger temparature delta and ony has to be slightly larger if at all.

Overall heat density on chips has increased due to litography changes so the chiplet architecture in a way is only a stopgap.

Reply on news.ycombinator.com