Can you elaborate on fundamental limits?
Clearly this cannot continue, as the training costs will exceed all the compute capacity in existence.
The other limit is training data, eventually you run out of cheap sources.
There could be huge efficiency savings within the implementations but when training costs are already this high it seems naive to think the low-hanging fruit is still there.
I was not familiar with Edge AI, interesting concept. I feel like improving the efficiency of very large models is much more likely.
The recent successes will lead to an even larger influx of $ in the short term --- this is an existential threat to Google, after all. We will see where things go!
I don't know why people just keep repeating this complete fantasy as if it were true. Where does it originate from, I wonder? I suspect someone said something like that on social media, their post went viral, and now all of the internet is reverberating with this thing. It's a meme, yes?
I don't have social media outside of HN.
It comes from a few observations:
1) Large models are still improving with increased parameter counts (we do not know where the ceiling is yet; it could be low but it could also be high).
2) Most current architectures train by using all model parameters to produce an output, which is vastly inefficient. While it is not clear how to improve on this in the general case yet, in the simpler problem of NERFs, sidestepping this issue has led to a ~100x improvement in training time.
3) https://mingukkang.github.io/GigaGAN/ very recently increased the parameter count of StyleGAN by selecting parameters dynamically at runtime. They improved on previous results by a very, very large margin, at somewhat comparable training times.
I stand by my claim: "neural network architectures are still in their infancy, it is very likely that more efficient approaches exist". I am not claiming that AI will become sentient or anything crazy and do not understand why you are associating my point of view with other people. I just said that it is likely that a novel technology will continue to improve (has this it ever NOT been the case for any new technology?).
Well, if you want to know whether neural nets are in their "infancy" you shouldn't make "observations", you should read the literature. It goes back many years. Go to the primary sources, why try to guess and risk guessing wrong, as here?
>> I stand by my claim: "neural network architectures are still in their infancy, it is very likely that more efficient approaches exist".
Half of your "claim" is incorrect. Don't just double down on it! There's really nothing to "claim" here, the "infancy" or not of neural nets is not a matter of claiming or guessing. Either you know what it is, or you don't.
"if you want to know whether neural nets are in their 'infancy' you shouldn't make 'observations'" -> I think you have understood this as "neural networks did not exist before" when what I meant is "neural networks will still change a lot in the future"
I was interested in discussing, specifically: will NNs continue scaling up in size in the near/long term. In answer to the statement "the training costs will exceed all the compute capacity in existence", I added "it is very likely that more efficient approaches exist.". You replied that NNs have existed for a long time, and that because of this, it is a complete fantasy that more efficient approaches exist. Do you feel like this is a fair assessment or no?
Well, my mistake then for taking you seriously. Many apologies. You can rest assured it won't happen again.
You are right that neural networks are not in their infancy. Best wishes.
LLMs of this type will just start making up shit when they dont know something. because they simply generate the most probable next token based on previous x tokens. this is not fixable.
this alone makes these LLMs practically unusable in vast majority of real-world applications where you would otherwise imagine this tech to be used.
for instance, your comment confidently states this is unfixable - presumably based on the frequency you've seen similar text on the internet. why should anyone believe the veracity of your statement? These things didn't have any of these emergent capabilities one year ago, why are you so sure you understand their nature one year from now?
no its because GPT is based on transformers.
Aren't you just a function of your input and memories (stuff you've read, sensory input) as run through/managed by some neural network?
What makes you think the rest isn't just emergent properties?
And what makes you think you can't hook up the LLM with some algorithms or layers that handle some of the rest behavior of what your brain does?
and the issue of lost uncertainty is inherrent to this, yes.
to fix this, a new type of llm would have to be invented. this particular branch of development may very well be a dead end.
Perhaps it's because I'm ignorant about the inner workings, but calling the problem "unfixable" so early in the evolution of LLMs seems like foolish certainty.
We don’t teach babies by throwing lots of data at them, instead we teach them by giving them useful data.
The loop I see is: - train on a lot of existing data - run out of useful data - people use ai and give feedback (we’re here) - perform reinforcement learning on the data collected
Loop over the last two steps.
There is already more than enough data available, it’s just not nicely labeled to say if it’s high quality or should be discarded. Those last two steps will implicitly do the labeling.
156 million hours of YouTube videos
9,000 words/hour
6 characters/word (including space)
First, let's find out the total number of characters:
9,000 words/hour \* 6 characters/word = 54,000 characters/hour
Now, let's calculate the total number of characters for 156 million hours of YouTube videos:
54,000 characters/hour \* 156,000,000 hours = 8,424,000,000,000 characters
Since 1 character is typically 1 byte, we can convert this to gigabytes:
8,424,000,000,000 bytes / (1024 \* 1024 \* 1024) ≈ 7,842.11 GB
With a vision transformer each token may be around 16x16 pixels. I found an example where they use images of resolution 224x224 for training a vision transformer so if we go with that that 256 pixels per token and 50176 pixels per image, so 196 tokens per frame, 24 frames per second, that's 4704 tokens per second or 16934400 token / hour. In total we're at 2.6x10^15 tokens.
GPT-3 was trained on 5x10^11 tokens, so YouTube done this way would be around four orders of magnitude more tokens that GPT-3 was trained on.
GPT-3 was undertrained by 1-2 orders of magnitude, so the compute required to trained a model on YouTube would then be around 6 orders of magnitude higher than what was used to train GPT-3, so about one million times more.
I did a linear regression on the training costs from cerebras(1) and came up with the formula (1901.67366*X)-197902.72715 where X is number of tokens in billions.
Plugging in 5x10^15 tokens we get a training cost of 5 billion dollars. I guess a lot of optimizations could be done that would decrease the cost, so maybe its doable in a few years.