The same was said 10 years ago. It's astonishing what can be done, but you can already see fundamental limits. I think it will raise productivity for some tasks, but not fundamentally change society.
Popular sentiment is a pretty meaningless metric for predicting the future.
I‘m arguing public sentiment will be right 50% of the time (in difficult binary predictions), not 0%.
The trouble with many of those construction examples is that they're point in time. Installer fucks up how the windows are installed re: flashing and water-proofing, but then puts the siding over it... the error is now completely hidden.
You could automated the inspection by AI photo analysis of every single window install on the project, say - but we could already do that for the inspection vs sending someone out, and send the photos to the expert instead, and we don't. Whether that's for lack of incentive to improve, or for ability to go deeper out of distrust for builders? I don't know.
Clearly this cannot continue, as the training costs will exceed all the compute capacity in existence.
The other limit is training data, eventually you run out of cheap sources.
There could be huge efficiency savings within the implementations but when training costs are already this high it seems naive to think the low-hanging fruit is still there.
I was not familiar with Edge AI, interesting concept. I feel like improving the efficiency of very large models is much more likely.
The recent successes will lead to an even larger influx of $ in the short term --- this is an existential threat to Google, after all. We will see where things go!
I don't know why people just keep repeating this complete fantasy as if it were true. Where does it originate from, I wonder? I suspect someone said something like that on social media, their post went viral, and now all of the internet is reverberating with this thing. It's a meme, yes?
I don't have social media outside of HN.
It comes from a few observations:
1) Large models are still improving with increased parameter counts (we do not know where the ceiling is yet; it could be low but it could also be high).
2) Most current architectures train by using all model parameters to produce an output, which is vastly inefficient. While it is not clear how to improve on this in the general case yet, in the simpler problem of NERFs, sidestepping this issue has led to a ~100x improvement in training time.
3) https://mingukkang.github.io/GigaGAN/ very recently increased the parameter count of StyleGAN by selecting parameters dynamically at runtime. They improved on previous results by a very, very large margin, at somewhat comparable training times.
I stand by my claim: "neural network architectures are still in their infancy, it is very likely that more efficient approaches exist". I am not claiming that AI will become sentient or anything crazy and do not understand why you are associating my point of view with other people. I just said that it is likely that a novel technology will continue to improve (has this it ever NOT been the case for any new technology?).
Well, if you want to know whether neural nets are in their "infancy" you shouldn't make "observations", you should read the literature. It goes back many years. Go to the primary sources, why try to guess and risk guessing wrong, as here?
>> I stand by my claim: "neural network architectures are still in their infancy, it is very likely that more efficient approaches exist".
Half of your "claim" is incorrect. Don't just double down on it! There's really nothing to "claim" here, the "infancy" or not of neural nets is not a matter of claiming or guessing. Either you know what it is, or you don't.
"if you want to know whether neural nets are in their 'infancy' you shouldn't make 'observations'" -> I think you have understood this as "neural networks did not exist before" when what I meant is "neural networks will still change a lot in the future"
I was interested in discussing, specifically: will NNs continue scaling up in size in the near/long term. In answer to the statement "the training costs will exceed all the compute capacity in existence", I added "it is very likely that more efficient approaches exist.". You replied that NNs have existed for a long time, and that because of this, it is a complete fantasy that more efficient approaches exist. Do you feel like this is a fair assessment or no?
Well, my mistake then for taking you seriously. Many apologies. You can rest assured it won't happen again.
You are right that neural networks are not in their infancy. Best wishes.
LLMs of this type will just start making up shit when they dont know something. because they simply generate the most probable next token based on previous x tokens. this is not fixable.
this alone makes these LLMs practically unusable in vast majority of real-world applications where you would otherwise imagine this tech to be used.
for instance, your comment confidently states this is unfixable - presumably based on the frequency you've seen similar text on the internet. why should anyone believe the veracity of your statement? These things didn't have any of these emergent capabilities one year ago, why are you so sure you understand their nature one year from now?
no its because GPT is based on transformers.
Aren't you just a function of your input and memories (stuff you've read, sensory input) as run through/managed by some neural network?
What makes you think the rest isn't just emergent properties?
And what makes you think you can't hook up the LLM with some algorithms or layers that handle some of the rest behavior of what your brain does?
and the issue of lost uncertainty is inherrent to this, yes.
to fix this, a new type of llm would have to be invented. this particular branch of development may very well be a dead end.
Perhaps it's because I'm ignorant about the inner workings, but calling the problem "unfixable" so early in the evolution of LLMs seems like foolish certainty.
We don’t teach babies by throwing lots of data at them, instead we teach them by giving them useful data.
The loop I see is: - train on a lot of existing data - run out of useful data - people use ai and give feedback (we’re here) - perform reinforcement learning on the data collected
Loop over the last two steps.
There is already more than enough data available, it’s just not nicely labeled to say if it’s high quality or should be discarded. Those last two steps will implicitly do the labeling.
156 million hours of YouTube videos
9,000 words/hour
6 characters/word (including space)
First, let's find out the total number of characters:
9,000 words/hour \* 6 characters/word = 54,000 characters/hour
Now, let's calculate the total number of characters for 156 million hours of YouTube videos:
54,000 characters/hour \* 156,000,000 hours = 8,424,000,000,000 characters
Since 1 character is typically 1 byte, we can convert this to gigabytes:
8,424,000,000,000 bytes / (1024 \* 1024 \* 1024) ≈ 7,842.11 GB
With a vision transformer each token may be around 16x16 pixels. I found an example where they use images of resolution 224x224 for training a vision transformer so if we go with that that 256 pixels per token and 50176 pixels per image, so 196 tokens per frame, 24 frames per second, that's 4704 tokens per second or 16934400 token / hour. In total we're at 2.6x10^15 tokens.
GPT-3 was trained on 5x10^11 tokens, so YouTube done this way would be around four orders of magnitude more tokens that GPT-3 was trained on.
GPT-3 was undertrained by 1-2 orders of magnitude, so the compute required to trained a model on YouTube would then be around 6 orders of magnitude higher than what was used to train GPT-3, so about one million times more.
I did a linear regression on the training costs from cerebras(1) and came up with the formula (1901.67366*X)-197902.72715 where X is number of tokens in billions.
Plugging in 5x10^15 tokens we get a training cost of 5 billion dollars. I guess a lot of optimizations could be done that would decrease the cost, so maybe its doable in a few years.
What you're seeing here with LLMs is sledge hammer number 2.
It's understandable how most people don't notice the sledge hammer. The decade prior to 2010 there was another sledge hammer had no smart phones. We were hit with a smart phone hammer AND an AI sledge hammer and the integration was so seamless we didn't even notice.
Much of the same will happen with LLMs. In 5 years it's so normal, nobody cares and likely we will forget what life was like before LLMs.
And still almost all of these applications are not really impactful or that important compared to actually society and life changing developments like the steam engine, electricity, electromagnetic transmission, cars, computers themselves and the internet.
Just more of the same, with added spice.
In the sense that we could revert to 10 years ago, and nothing would be much different or missed. Whereas going back without electricity or cars would be a totally different thing!
I think LLMs can be far more impactful than anything else hyped in machine learning of the past 10 years...
Oh, browser, you know me so well.
The AI sledge hammer is as good as we are going to get in paradigm shifts for a while given how we are at the top of an s-curve for development. Additionally we're only really seeing the first decade. When the computer was discovered was it ubiquitous in a decade? No. In fact it was largely a sort of useless academic research project for the longest time.
“What work has gone on for supply chain improvement in our European operations this year?” - this is the kind of question that is easy to ask in natural language but might take someone a week of searching, messaging, etc to assemble. An LLM with access to all of the code, documents, chats, etc could just give an answer with citations. We are not betting $1B on the answer that it gives, but it has saved us 1 week of work and allows us to move on to the next step in the project.
There are plenty of tasks like this which are highly valuable yet don’t require high trust. The one-shot “what is the airspeed of a coconut-laden swallow” type questions are actually fairly rare.
Trust is only a bound for certain areas, and this bound is eroding.