Hacker News Reader

Comment by Yokolos | Hacker News Reader

I wouldn't be so sure. I've seen analyses making the case that this new phase is unlike previous cycles and DRAM makers will be far less willing to invest significantly in new capacity, especially into consumer DRAM over more enterprise DRAM or HBM (and even there there's still a significant risk of the AI bubble popping). The shortage could last a decade. Right now DRAM makers are benefiting to an extreme degree since they can basically demand any price for what they're making now, reducing the incentive even more.

https://www.tomshardware.com/pc-components/storage/perfect-s...

View on HN

The most likely direct response is not new capacity, it's older capacity running at full tilt (given the now higher margins) to produce more mature technology with lower requirements on fabrication (such as DDR3/4, older Flash storage tech, etc.) and soak up demand for these. DDR5/GDDR/HBM/etc. prices will still be quite high, but alternatives will be available.

> produce more mature technology ... DDR3/4

...except current peak in demand is mostly driven by build-out of AI capacity.

Both inference and training workloads are often bottlenecked on RAM speed, and trying to shoehorn older/slower memory tech there would require non-trivial amount of R&D to go into widening memory bus on CPU/GPU/NPUs, which is unlikely to happen - those are in very high demand already.

Even if AI stuff does really need DDR5, there must be lots of other applications that would ideally use DDR5 but can make do with DDR3/4 if there's a big difference in price

I mean, AI is currently hyped, so the most natural and logical assumption is that AI drives these prices up primarily. We need compensation from those AI corporations. They cost us too much.

It is still an assumption.

> The shortage could last a decade.

Do we really think the current level of AI-driven data center demand will continue indefinitely? The world only needs so many pictures of bears wearing suits.

The pop culture perception of AI just being image and text generators is incorrect. AI is many things, they all need tons of RAM. Google is rolling out self-driving taxis in more and more cities for instance.

Congrats on engaging with the facetious part of my comment, but I think the question still stands: do you think the current level of AI-driven data center demand will continue indefinitely?

I feel like the question of how many computers are needed to steer a bunch of self-driving taxis probably has an answer, and I bet it's not anything even remotely close to what would justify a decade's worth of maximum investment in silicon for AI data centers, which is what we were talking about.

Data center AI is also completely uninteresting/non-useful for self driving Taxis, or any other self driving vehicle.

Do you know comparatively how much GPU time training the models which run Waymo costs compared to Gemini? I'm genuinely curious, my assumption would be that Google has devoted at least as much GPU time in their datacenters to training Waymo models as they have Gemini models. But if it's significantly more efficient on training (or inference?) that's very interesting.

My note is specifically for operating them. Training the models, certainly can help.

A decade is far from indefinitely.

AI is needed to restart feudalism?

No, the 10% best scenario return on AI won't make it. The bubble is trying to replace all human labor, which is why it is a bubble in the first place. No one is being honest that AGI is not possible in this manner of tech. And Scale won't get them there.

There's not a difference between "consumer" DRAM and "enterprise" DRAM at the silicon level, they're cut from the same wafers at the end of the day.

Doesn't the same factory produce enterprise (i.e. ECC) and consumer (non-ECC) DRAM?

If there is high demand for the former due to AI, they can increase production to generate higher profits. This cuts the production capacity of consumer DRAM, and lead to higher prices in that segment too. Simple supply & demand at work.

Conceptually, you can think of it as "RAID for memory".

A consumer DDR5 module has two 32-bit-wide buses, which are both for example implemented using 4 chips which each handle 8 bits operating in parallel - just like RAID 0.

An enterprise DDR5 module has a 40-bit-wide bus implemented using 5 chips. The memory controller uses those 8 additional bits to store the parity calculated over the 32 regular bits - so just like RAID 4 (or RAID 5, I haven't dug into the details too deeply). The whole magic happens inside the controller, the DRAM chip itself isn't even aware of it.

Given the way the industry works (some companies do DRAM chip production, it is sold as a commodity, and others buy a bunch of chips to turn them into RAM modules) the factory producing the chips does not even know if the chips they have just produced will be turned into ECC or non-ECC. The prices rise and fall as one because it is functionally a single market.

That makes sense, thank you.

At the silicon level, it is the same.

Each memory DIMM/stick is made up of multiple DRAM chip. ECC DIMMs have an extra chip for storing the error correcting parity data.

The bottleneck is with the chips and not the DIMMs. Chip fabs are expensive and time consuming, while making PCBs and placing components down onto them is much easier to get into.

Got it now, thanks!

Yes, but if new capacity is also redirected to be able to be sold as enterprise memory, we won't see better supply for consumer memory. As long as margins are better and demand is higher for enterprise memory, the average consumer is screwed.

Does it matter that AI hardware has such a shorter shelf life/faster upgrade cycle? Meaning we may see the ram chips resold/thrown back into the used market quicker than before?

Is there still a difference? I have DDR5 registered ECC in my computer.

I mean, the only difference we care about is how much of it is actual RAM vs HBM (to be used on GPUs) and how much it costs. We want it to be cheap. So yes, there's a difference if we're competing with enterprise customers for supply.

I don't really understand why every little thing needs to be spelled out. It doesn't matter. We're not getting the RAM at an affordable price anymore.

Anytime somebody is making a prediction for the tech industry involving a decade timespan I pull out my Fedora of Doubt and tip my cap to m’lady.

Maybe we'll get default to ECC in everything with this?

A LOT of businesses learned during Covid they can make more money by permanently reducing output and jacking prices. We might be witnessing the end times of economies of scale.

The idea is someone else comes in that's happy to eat their lunch by undercutting them. Unfortunately, we're probably limited to China doing that at this point as a lot of the existing players have literally been fined for price fixing before.

https://en.wikipedia.org/wiki/DRAM_price_fixing_scandal

It seems more likely that someone else comes in and either colludes with the people who are screwing us to get a piece of the action or gets bought out by one of the big companies who started all this. Since the rare times companies get caught they only get weak slaps on the wrist where they only pay a fraction of what they made in profits (basically just the US demanding their cut) I don't have much faith things will improve any time soon.

Even China has no reason to reduce prices much for memory sold to the US when they know we have no choice but to buy at the prices already set by the cartel. I expect that if China does start making memory they'll sell it cheap within China and export it at much higher prices. Maybe we'll get a black market for cheap DRAM smuggled out of China though.

I think in part it is a system level response to the widespread just-in-time approach of those businesses' clients. A just-in-time client is very "flexible" on price when supply is squeezed. After that back and forth i think we'll see return to some degree of supply buffering(warehousing) to dampen down the supply levels/price shocks in the pipelines.

I thought that, too, but then the Nexperia shitstorm hit, and it was as if the industry had learned nothing at all from the COVID shortages.

In that case it's far simpler - even IF they wanted to met the demand, building more capacity is hideously expensive and takes years.

So, it would happen even with best intentions and no conspiracies. AI boom already hiked GPU prices, memory was next in line.

Reply on news.ycombinator.com