Hacker News Reader

Comment by redanddead | Hacker News Reader

More and more I think about the article asking why we didn't have LLMs in 2005

I don’t understand why this article makes you ask that, can you elaborate? The simple answer to why we didn’t have LLMs in 2005 is that nobody had the requisite ideas that make them work effectively (independent of any agentic harnesses and Schmidhuber notwithstanding) and there wasn’t sufficient clarity in the direction to scale for anyone to build anything at scale (also a requisite for them working well).

Surely the only problem isn’t the knowledge of how to train and the confidence it would work. Wouldn’t it also have required a 1 or 2 order of magnitude increase in computing power over 2005-era supercomputers to be able to train a modern LLM in a reasonable time (“reasonable” meaning “finish training before GPT-3 comes out in 15 years”)?

And also "the whole Internet" (including scanned books etc.) on which to train was much smaller.

On the other hand, Google was openly collecting data with a view towards building AI.

I think it’s plausible that attention could have been invented earlier and GPT demonstrated at smaller scale on earlier hardware. Then hardware timelines might have accelerated (or alternative approaches explored) and we’d have gotten to where we are earlier.

NVIDIA moved as fast as possible back then, there were huge performance gains with each new GPU generation, hard to imagine they might have gone even faster.

We had genetic programming though, and with the right harness you could accomplish impressive results: https://dl.acm.org/doi/abs/10.1145/2771783.2771796

You could view LLMs as a super-charged mutation function inside a genetic programming harness.

>You could view LLMs as a super-charged mutation function inside a genetic programming harness.

There's probably something there, though im not sure what. I'm thinking about LLMs token-filling and the 4 amino bases

Reply on news.ycombinator.com