GPU seem to last lot longer than expected.. And have good resale value for other use cases. The $2 for every $1 inference cost quote from Anthropic I presume includes all opex.
Defraying of capex also depends on how the hyperscalers scrap used hardware.
AFAIK, all hyperscaler hardware is effectively a consumable. Disks, memory, boards, chips; all the high-value pieces do not get refurbished into new computers for sale into secondary markets. The commercial and legal liability is too high---leaking their own proprietary IP as well as leaking customer IP, not to mention secrets and hardening configs and so forth.
I can't recall where I read it, but if a datacenter is modeled as a bunch of black-box hot-swappable shipping containers, then for pre-LLM workloads, per-container unit economics works only as long as a container delivers over 30% capacity. After that you got to swap it out, and send the black box into a furnace to recover them rare earth metals for the next generation of chips.
To a first approximation, high-throughput, gpu-maxxing math workloads (LLM inference) are surely harder on chips and memory, than are bread-and-butter SaaS / data lake / ETL / RDBMS workloads. Speculating further, I'd guess that the ~40% operating margin of datacenters is due to the relative compute-friendliness of non-math workloads for lack of better words.
Could it be that they _have_ to run them like that because of the shortage of compute volume and once there are sufficient data centers, they can be run relatively cooler and thereby extending their lifespans? That doesn't bode well for NVIDIA/AMD.
> AFAIK, all hyperscaler hardware is effectively a consumable. Disks, memory, boards, chips; all the high-value pieces do not get refurbished into new computers for sale into secondary markets. The commercial and legal liability is too high---leaking their own proprietary IP as well as leaking customer IP, not to mention secrets and hardening configs and so forth.
Maybe that could be a catalyst for going for more "commodity" approach to hardware instead of highly spcialized (atleast for some sections of the hardware) so that the IP is not the hardware but the model/software and they can re-coup the costs by selling the hardware to "tier-2" providers that might emerge based on using cheaper second-hand hardware?
[Type A] Constant use a constant power level: Quantum world rules would dominate.
When run hot, electron tunneling will kill off individual transistors, and AFAIK that is a cascading failure mode once it takes root. A big enough failed spot, even if localised, could effectively kill a core (if not the whole chip---too many errors). A local hotspot will directly affect neighbours, and indirectly the chip as a whole because the same work is being pushed on fewer cores.
[Type B] Thermal cycled: Mechanical and electrical factors would dominate...
Microfractures / mechanical damage in weak areas of whole chips, and motherboards. In a hyperscaler server rack, if a board dies that part of the rack is dead---micro-maintenance is probably too expensive.
And speculating further, it stands to reason that Type B failures would likely cause Type A failures, and vice-versa.
server parts are tuned for power efficiency and longevity
B200 has an operating range of 10-35C. https://resources.nvidia.com/en-us-dgx-systems/dgx-b200-data...
RTX 5090 (and most consumer CPU/GPUs) have thermal limits at 90-100C.
No, it does not, its just revenue minus compute. Per their S1, Anthropic has never turned an operational profit.
Older GPUs can still be used to run model inference, albeit more slowly than newer GPUs. We already see frontier labs turning this insight into an opportunity to charge a premium on token delivery speed, when models run on newer hardware. As AI becomes more widely spread throughout the economy, not all workloads will require fast inference, and there will be many use cases for previous generation hardware. The only exception is if the latest batch of specialised chips become ubiquitous very quickly, but then price of compute AND intelligence per watt will both go down dramatically, which would reduce the need for sustained high capex and opex
- Demand drops like a stone: raw TFLOPS / per model. This will happen if someone figures out insanely fast and power-efficient math (specifically, multiplication) for existing chip architectures, that makes it dirt-cheap to keep re-computing ever-better SOTA LLMs, as well as to on-line compute inference with ever-bigger models (scaling laws rule). These are different workloads, AFAIK.
- Moore's law re-starts, because someone figures out entirely new transistor designs that can deliver at least up to 100x per-core frequency gains over Si transistors (which would be nothing considering that Si transistor tech delivered ~1Mx gains since invention) and which can be manufactured at similar bin sizes, at similar scale (i.e. equivalent per-die yield), price-competitively (no more 10x the equivalent in SOTA Silicon compute).
I'd say that a math breakthrough might be more accessible than the transistor-manufacturing breakthrough.
That assumes that there is no hard, physical limit on Moore's Law, which is obviously false. There is a limit, and as long as we depend on mass flow of charge through conductors, it's nowhere near quantum sizes.
Where that limit is has not yet been determined, but at this point the support structure to write transistors is the size of a house trailer, using a staggering amount of toxic chemicals (just to run the exotic lasers). So, advancements today may scale very nonlinearly in cost.
The very phrase "Moore's law restarts" acknowledges that we are no longer on-track.
Do they? Is H200 $-competitive with an ASIC for crypto? Bearing in mind they're 4 figures.
Neural networks in general are here to stay. Whether that is the current day frontier-model style LLM or not doesn't really matter. We've already had multiple different Deep Learning/Machine Learning architectures that have been incredibly applicable to multiple different domains.
Especially right now in the RAM-pocalpyse where it is difficult to get any machine, yes, these older GPUs are still very very useful. A guy at my company was having trouble getting compute allocated, budgeted, and approved - so eventually he went and dug up a bunch of old Volta-based GPUs, setup a local compute farm, and is getting work done. They are absolutely slower than current state-of-the-art, but they are still better than no compute at all.
I think one of the silver-linings of the current hardware high prices and shortages, is that people are going to (A) focus on software efficiency (B) not be so obsessed with having the latest and greatest hardware. Even on the consumer side, how many games ACTUALLY can utilize a RTX 5090?