Hacker News Reader

Hacker News

Comment by suprjami | Hacker News Reader

Full thread

suprjami

With an impressive 1.5TB/s RAM bandwidth you'd think they'd advertise that more.

Marketing department asleep at the wheel.

View on HN

This is a very low number for a server GPU. My V100 from 2017 is 0.9 TB/s.

A single B300 has almost 8 TB/s.

AMD MI300x from 2023 is 5.3 TB/s

Edit: my bad you are saying that this is an impressive number precisely because it is more or less normal RAM, not VRAM

VRAM isn't a type of memory, it's just RAM used for the GPU. So this LPDDR5 is VRAM, just like it can be with my Strix Halo machine.

Very true, though traditionally it did refer to GDDR.

That's only partially true - the GDDRX you'd find on say a 3090 is physically very different than what you'd find on a strix halo or mac or vanilla ram. e.g. runs 4 voltage levels not 2 so that it can transmit twice the info per cycle

Well, what they don't tell you about GDDR6X is that it's 4 voltage levels but it never goes from min to max or vice versa in a single transition (PAM4 MTA). It transfers 1.75 bits per cycle.

And then GDDR7 backed off to PAM3 and sends 1.5 bits per cycle.

There's a good chance [LP]DDR will use the same techniques once it gets into the same speed range.

Yes the cells are pretty much the same as they would be with hbm as well all the magic happens at the bus and the signaling, there is both a reason hbm and gddr have higher bandwidths and tradeoffs.

DDR has better latency hence the preference in CPU workloads where lower latency is more important due to variable workloads.

Its bandwidth is around that of 32 DDR5 DIMMS (50ish GB/s). These boards should have built in NICs but they don't because Intel also wants to sell more servers.

And a 5090 has 1.79 TB/s.

So yeah on its own not very impressive.

The large amount of memory means you can serve larger models, or less quantized, but that also means they're going to be slower due to the low bandwidth.

https://www.techpowerup.com/gpu-specs/geforce-rtx-5090.c4216

I'm pretty sure the D270 is meant to be a competitor to the RTX Pro 6000, which also has 1.79TB/s memory bandwidth like the RTX 5090. And it will potentially be a competitor (depending on what the MSRP is) to Nvidia GB10 based systems for 'small' scale AI development. It's not meant to run frontier-size models at the speed you'd expect from something like a cluster of H200 cards.

D270: 1.5TB/s memory bandwidth, 480GB memory capacity, 350W power draw

RTX Pro 6000: 1.79TB/s memory bandwidth, 96GB memory capacity, 600W power draw

Slightly less memory bandwidth, but 5x memory capacity and nearly half the power draw.

Sure, but if you use that memory for the weights of a single model, it'll be ~5x slower.

Yeah but I think this accelerator is more meant for a small scale deployment. This could probably serve a medium moe llm model at reasonable tk/s plus you could in addition to the llm also run a couple of other models on it in parallel like an embedding, reranker, STT, TTS, decision, image, video, … if you deploy this for a small business they might make use of all of these models but not all at the same time and not with 100s of users but this card could bring a serious broad range of AI capabilities for them.

> Marketing department asleep at the wheel.

The marketing department doesn't have to lift a finger in this market. They could quietly drop it for sale without a single announcement and it would be backordered for a year.

Companies nowadays care more about valuations and stock spikes than sales. Curiously D270 announcement was followed by 3% INTC drop.

everybody is long everything that may touch sand in the next couple years, no buyers left.

These things will fly off the shelves with 0 marketing

Source? Extremely impressive if true

https://www.intel.com/content/www/us/en/newsroom/news/artifi...

It's not impressive? Thats about "as slow" as M5 Ultra Mac Studio. That's just speed here is prettu much limited by bus width. You get 1.8 TB/s with 5090 or RTX PRO 6000.

> Marketing department asleep at the wheel.

They tasked an AI agent to it, like everyone else.

Xeon Phi when introd in 2013ish had that much bandwidth.

Xeon Phi's ranged between ~100-350 GB/s depending on the models.

The thing to note here isn't supposed to be that 1.5 TB/s is record breaking/astounding though, even the consumer 5090 GPU has higher bandwidth than that, it's just capacity@bandwidth is what defines the usefulness of an AI accelerator and this is good on both sides instead of just one. I.e. there are a lot of "cheap" low-capacity@high-bandwidth devices and a lot of "cheap" high-capacity@low-bandwidth devices. What makes this accelerator interesting is not that it has 480 GB of LPDDR5X, it's that it has 480 GB of LPDDR5X at a speed worth having that much for, which has normally been the realm of the likes of Nvidia/AMD and their AI/DC options.

Sorry I was referring to the internal bidirectional ring between all of the cores. Cache != external ddr!

Reply on news.ycombinator.com