Hacker News Reader

Comment by dlevine | Hacker News Reader

It has been a decade or so since I read The Singularity is Near. From what I remember, Kurzweil said that technologies will increase exponentially once they become linked to the exponential increase in computing power.

Now that Moore's Law has leveled off, I do wonder what will keep the exponential increase going. For example, LLMs are increasing exponentially in size each generation, but that also involves an exponential increase in cost, and will only be sustainable up to a certain point.

View on HN

> but that also involves an exponential increase in cost

Not necessarily. For training, yes. But there surely will be custom hardware specialized for the inference of "1-bit" quantized models, which could be orders of magnitude more energy-effective than GPUs.

Hell, you probably can even bake trained models in silicon. Maybe instead of digital logic, use analog circuits. Photonics?

I believe there's so much room for improvement and I feel that GPUs are just the "training wheels" (lol, quite literally) — there will be exponentially more effective hardware for running LLMs.

LLMs aren't even relevant in Kurzweil's Singularity vision, they are just a current trend. They may pan out, or something else will.

Even asic will imo only give us one step-function jump. there will be a huge jump and then it is mostly back to moore's law-ish. maybe the gradient will be a bit steeper but i doubt it.

If blockchain is any indication, ASICs give two jumps (2 orders of magnitude) over GPUs. But yeah, then further jumps will have to be via parallelism; can we put 100s or 1000s of these ASICs in a single computer?

In Kurzweil's version, Moore's law is not a metric of transistor capacity per dollar, but instead is an exponential growth of the amount of information that humans can leverage. He claims that this has been going for thousands of years and continues (quite vertically) now.

I would argue that there will be a horizontalization of the curve at some point. Meaning, we cannot continue exponentially forever.

Unless we discover new physics we will be bound to cubic growth.

This ex-OpenAI engineer’s recent essay makes the case that LLMs are progressing at 5x the rate of Moore’s Law across a number of dimensions: compute, efficiency, and capability: https://situational-awareness.ai/from-gpt-4-to-agi/

I don't know but in another HN thread today they discuss a way to make the things much more efficient https://news.ycombinator.com/item?id=40794564

It's surprising compute/$ has increased in a fairly smooth exponential way for more than a century, long before Moore. I guess if the financial incentives are there people find a way.

Reply on news.ycombinator.com