OpenAI’s CEO says the age of giant AI models is already over | Hacker News Reader

OpenAI’s CEO says the age of giant AI models is already over

wired.com

405 pointslabrador439 comments

The point is that now we're at the point of diminishing return for increasing model size, unless we find a better modeling architecture than Transformer.

I think this is likely true; while all the other companies underestimated the capability of transformer (including Google itself!), OpenAI made a fairly accurate bet on the transformer based on the scaling law, put all the efforts to squeeze it until the last drop and took all the rewards.

It's likely that GPT-4 is on the optimal spot between cost and performance and there won't be significant improvements on performance in a near future. I guess the next task would be more on efficiency, which has a significant implication on its productionization.

> (including Google itself!)

Bet Google won’t make that mistake again, i.e. it won’t publish as much and will be much more careful about what it publishes, least they give a competitor a useful tool and get nothing in return - when the competitor (in this case very ironically named) goes full commercial and close source everything they can.

Open collaboration in AI, at least when it comes to corporations, might have come to an end.

The scale of the damage Open AI has done to the trust ecosystem with soliciting not just the work, but also massive fundraising and then privatizing the profits is almost unprecedented and permanent.

Sun Tzu.

When you are weak pretend to be strong.

When you are strong, pretend to be weak.

Doesn't Google have any patents on the transformer architecture? I assume large enterprises tend to patent everything that comes out of their research.

Seems like a wild claim to make without any examples of gpt models which are bigger and no demonstrably better.

I suspect you are right. We may be stuck at the gpt4 sizes for a bit just because of hardware costs though. As they get bigger it costs too much to run them until our hardware becomes more optimal for these large models at 4 bits or so.

I think the YouTube videos is going to be the next big training set. A transformer trained on all text and all of YouTube will be killer amazing at so much. I bet it can understand locomotion and balance and body control from YouTube.

I wonder if TPUs, like Google's Tensor chip, will beat out GPUs when it comes to image/video based training?

or like a curve of model complexity versus results or whatever showing it asymptotically approaches whatever.

actually there was a great paper from microsoft research from like 2001 on spam filtering where they demonstrated that model complexity necessary for spam filtering went down as the size of the data set went up. That paper, which i can't seem to find now, had a big impact on me as a researcher because it so clearly demonstrated that small data is usually bad data and sophisticated models are sometimes solving problems will small data sets instead of problems with data.

of course this paper came out the year friedman published his gradient boosting paper, i think random forest also was only recently published then as well (i think there is a paper from 1996 about RF and briemans two cultures paper came out this year where he discusses RF i believe), and this is a decade before gpu based neural networks. So times are different now. But actually i think the big difference is these days i probably ask chatgpt to write the boiler plate code for a gradient boosted model that takes data out of a relational database instead of writing it myself.

Better on which axis? Do you want an AI that takes one hour to respond to? Some would for certain fields, but getting something fast and cheap is going to be hard now that Moore’s law is over.

Perhaps (a) there do exist bigger models that weren't better or (b) this model isn't better than somewhat smaller ones. Perhaps the CEO has seen diminishing returns.

https://twitter.com/SmokeAwayyy/status/1646670920214536193

Sam explicitly said that there won't be GPT-5 in the near future, which is pretty clear evidence unless he's blatantly lying in public speaking.

It's not a wild claim when you have empirically well-validated scaling laws which make this very prediction.

If OpenAI's CEO is making this claim, don't you think he has internal data backing up the claim?

OpenAI may have those internally though.

Both ChatGPT 3.5 and 4 literally fail the question:

"What is the third letter in the third word of this sentence"

When you've spent 100 million on training the thing and it fails on 1st grade ordinality I think it's fair to say you may not be on the right path

Further improvements in efficiency need not come from alternative architectures. They'll likely also come from novel training objectives, optimizers, data augmentations, etc.

There's only so much first order derivatives can do for you.

My bet is on sparsity, lottery tickets and symmetries.

Does this mean we’ve reached the next AI winter? This is as good as it gets for quite a long time? Honest question :) perhaps this will postpone everyone’s fears about the singularity…

The current stage is now productionizing what we have; finding product fits for it, and making it cheaper. Even GPT-4 isn't necessary to push forward what is possible with AI; if you think about something dumb like "load all of my emails into a language model in real time, give me digests, automatically write responses for ones which classify with characteristics X/Y/Z, allow me to query the model to answer questions, etc": This does not really exist yet, this would be really valuable, and this does not need GPT-4.

Another good example is in the coding landscape, which feels closer to existing. Ingest all of a company's code into a model like this, then start thinking about what you can do with it. A chatbot is one thing, the most obvious thing, but there's higher order product use-cases that could be interesting (e.g. you get an error in Sentry, stack trace points Sentry to where the error happened, language model automatically PRs a fix, stuff like that).

This shit excites me WAY WAY more than GPT-5. We've unlocked like 0.002% of the value that GPT-3/llama/etc could be capable of delivering. Given the context of broad concern about cost of training, accidentally inventing an AGI, intentionally inventing an AGI; If I were the BDFL of the world, I think we've got at least a decade of latent value just to capture out of GPT-3/4 (and other models). Let's hit pause. Let's actually build on these things. Let's find a level of efficiency that is still valuable without spending $5B in a dick measuring contest [1] to suss out another 50 points on the SAT. Let's work on making edge/local inference more possible. Most of all, let's work on safety, education, and privacy.

[1] https://techcrunch.com/2023/04/06/anthropics-5b-4-year-plan-...

Many years ago, there was an image that floated around with Craigslist and all the websites that replaced small parts of it—personals, for sale ads, etc. It turned out the way to beat Craigslist wasn’t to build Yet Another Monolithic Craigslist, but to chunk it off in pieces and be the best at that piece.

This is analogous to what’s happening with AI models. Sam Altman is saying we have reached the point where spending $100M+ trying to “beat” GPT-4 at everything isn’t the future. The next step is to chunk off a piece of it and turn it into something a particular industry would pay for. We already see small sprouts of those being launched. I think we will see some truly large companies form with this model in the next 5-10 years.

To answer your question, yes, this may be as good as it gets now for monolithic language models. But it is just the beginning of what these models can achieve.

No. Winter means people have lost interest in the research.

If anything successes in ChatGPT etc will be motivation for continued efforts.

I think the next step is multimodality, GPT-4 can "see" probably using a method similar to miniGPT-4, so the embeddings are aligned using Q-former (or something similar), the next step would be to actually predict image tokens using the LM loss, this way it would be able to use the knowledge gained by "seeing" on other tasks like: making actual good ASCII art, making SVG that makes sense, and on a less superficial level having a better world model.

If you use GPT4 as a "routing" service to send down to other models the journey is only beginning and most of the industry hasn't understood we're well beyond a doubling at the moment.

Transformers were known that they kept scaling up with more parameters and more training data so if the Open AI hit the limits of this scaling that would be a very important milestone in AI.

I think performance will keep scaling, the main problem is cost. If you want to feed longer strings of tokens in, the price is going to keep going up.

> while all the other companies underestimated the capability of transformer (including Google itself!), OpenAI made a fairly accurate bet on the transformer based on the scaling law, put all the efforts to squeeze it until the last drop and took all the rewards.

You hit the nail

Seems like a gigantic increase in the context window would be a great next step. GPT-4 is so damned good I tend to only become disappointed when it is forced to forget history.

I think there's a single point that we need to keep an eye out for;

A model that got to the point where it's possible for it to propose new architectures, improve optimization & efficiency.

In other words, the extremely massive model that could do this doesn't need to be available to the public. It's sole purpose should be to be used internally by a company to improve itself.

This is actually the point at which many say could lead to the singularity.

The main question here is how soon will the model architecture improvements come. I fathom it won't be that long till OpenAI or Google come up with something better. The tricky piece here is that these companies will now keep their models more proprietary to make profits. Can a competitor company figure out an on par or better model architecture in time?

> It's likely that GPT-4 is on the optimal spot between cost and performance and there won't be significant improvements on performance in a near future.

Given the amount of resources being thrown at AI right now, i consider this to be very unlikely indeed.

I think GP's comment is probably apt if understood together with the qualification of "unless we find a better modeling architecture than Transformer".

I really want to see a gpt-4-turbo model that's almost as inexpensive as gpt-3.5-turbo

> 640K ought to be enough for anybody

Related reading: https://dynomight.net/scaling/

In short it seems like virtually all of the improvement in future AI models will come from better algorithms, with bigger and better data a distant second, and more parameters a distant third.

Of course, this claim is itself internally inconsistent in that it assumes that new algorithms won't alter the returns to scale from more data or parameters. Maybe a more precise set of claims would be (1) we're relatively close to the fundamental limits of transformers, i.e., we won't see another GPT-2-to-GPT-4-level jump with current algorithms; (2) almost all of the incremental improvements to transformers will require bigger or better-quality data (but won't necessarily require more parameters); and (3) all of this is specific to current models and goes out the window as soon as a non-transformer-based generative model approaches GPT-4 performance using a similar or lesser amount of compute.

Good thing he got a bunch of companies to pony up the dough for LLM before he announced they where already over.

I don't think LLMs are over [0]. I think we're relatively close to a local optimum in terms of what can be achieved with current algorithms. But I think OpenAI is at least as likely as any other player to create the next paradigm, and that it's at least as likely as likely as any other player to develop the leading models within the next paradigm regardless of who actually publishes the research.

Separately, I think OpenAI's current investors have a >10% chance to hit the 100x cap on their returns. Their current models are already good enough to address lots of real-world problems that people will pay money to solve. So far they've been much more model-focused than product-focused, and by turning that dial toward the product side (as they did with ChatGPT) I think they could generate a lot of revenue relatively quickly.

[0] Except maybe in the sense that future models will be predominantly multimodal and therefore not strictly LLMs. I don't think that's what you're suggesting though.

Better data is still critical, even if bigger data isn't. The linked article emphasizes this.

I'd bet on a 2030 model trained on the same dataset as GPT-4 over GPT-4 trained with perfect-quality data, hands down. If data quality were that critical, practitioners could ignore the Internet and just train on books and scientific papers and only sacrifice <1 order of magnitude of data volume. Granted, that's not a negligible amount of training data to give up, but it places a relatively tight upper bound on the potential gain from improving data quality.

So true. There are still plenty of areas where we lack sufficient data to even approach applying this sort of model. How are we going to make similar advances in something like medical informatics where we not only have less data readily available but its much more difficult to acquire more data

Improvements will not come from collecting more and more samples for current large models, but will come from improvements to algorithms, that also may focus on improving the quality and use of input data.

I don't think there is such a clear separation between algorithms and data as your comment suggests.

All the LC grinding may come in handy after all! /s

What algorithms specifically show the most results upon improvement? Going into this I thought the jump of improvements were really related more advanced automated tuning and result correction, in which it could be done at scale as it were allowing a small team of data scientists to tweak the models until desired results were being achieved.

Are you saying instead, that concrete predictive algorithms need improvement or are we lumping the tuning into this?

I think it's unlikely that the first model to be widely considered AGI will be a transformer. Recent improvements to computational efficiency for attention mechanisms [0] seem to improve results a lot, as does RLHF, but neither is a paradigm shift like the introduction of transformers was. That's not to downplay their significance - that class of incremental improvements has driven a massive acceleration in AI capabilities in the last year - but I don't think it's ultimately how we'll get to AGI.

[0] https://hazyresearch.stanford.edu/blog/2023-03-27-long-learn...

Traditional CS may have something to do with slightly improving the performance by allowing more training for the same compute, but it won't be an order of magnitude or more. The improvements to be gained will be found more in statistics than CS per se.

We need more data efficient neural network architectures. Transformers work exceptionally well because they allow us to just dump more data into it, but ultimately we want to learn advanced behavior without having to feed it Shakespeare

bubble sort /s

I strongly believe the next generation of models will be based upon spiking neural concepts wherein action potentials are lazily-evaluated throughout the network (i.e. event-driven). There are a few neuron models that can be modified (at some expense to fidelity) in order to tolerate arbitrary delays between simulation ticks. Using actual latency between neurons as a means of encoding information seems absolutely essential if we are trying to emulate biology in any meaningful way.

Spiking networks also lend themselves nicely to some elegant learning rules, such as STDP. Being able to perform unsupervised learning at the grain of each action potential is really important in my mind. This gives you all kinds of ridiculous capabilities, most notably being the ability to train the model while it's live in production (learning & use are effectively the same thing).

These networks also provide a sort of deterministic, event-over-time tracing that is absent in the models we see today. In my prototypes, the action potentials are serialized through a ring buffer, and then logged off to a database in order to perfectly replay any given session. This information can be used to bootstrap the model (offline training) by "rewinding" things very precisely and otherwise branching time to your advantage.

The #1 reason I've been thinking about this path is that low-latency, serialized, real-time signal processing is somewhat antagonistic to GPU acceleration. I fear there is an appreciable % of AI research predicated on some notion that you need at least 1 beefy GPU to start doing your work. Looking at fintech, we are able to discover some very interesting pieces of technology which can service streams of events at unbelievable rates and scales - and they only depend on a handful of CPU cores in order to achieve this.

Right now, I think A Time Domain Is All You Need. I was inspired to go outside of the box by this paper: https://arxiv.org/abs/2304.06035. Part 11 got me thinking.

I'm constantly telling this to people who don't know what I'm talking about and don't know why I brought it up at a wedding, but I think it's inevitable.

Meh, people have been trying to ape biological networks for decades and they consistently produce poor functioning versions of the leading edge... Why wouldn't we have already seen spiking networks contribute something by now?

> Why wouldn't we have already seen spiking networks contribute something by now?

This presupposes we've explored this space thoroughly, and we haven't. When everything you do with NNs improves results (how it mostly is now), that means not enough people are trying out ideas and new things.

I don't think you can invoke EMH-like reasoning quite yet. Give us a nice long winter like physics has had, and then we can use this heuristic.

I think people said the same thing about NNs in general before we hit a scale where they started performing magic.

There could be exponential or quadratic scaling laws with any of these black boxes that makes one approach suddenly extremely viable or even dominant.

I'm bullish on SNNs too. This Chinese research group is doing something quite comprehensive with them:

https://news.ycombinator.com/item?id=35037605

I know what it looks like in my head but I can’t quite figure the algorithm out. The spiking is basically reinforcement learning at the neuron level. Get it right and it’s basically all you need. You don’t even need training data because it will just automagically learn from the data it sees.

This is like trying to build a more sophisticated bird-costume for flight instead of a steel flying machine. Human intelligence has a lot of features that are not necessary for artificial intelligence, because it doesn't have to build itself from ground-up using biology.

I don’t know how much the biological inspiration is a useful abstraction anymore. I’ve gone from thinking of matrix operations as an efficient way to implement neural networks to thinking of neural networks as a complicated sort of allegory for chained matrix ops and it feels much more natural to me.

Do we want to train the model during inference? I would think we want to be very particular about the data used to train - notably after OpenAI's notes on the subject.

> This gives you all kinds of ridiculous capabilities, most notably being the ability to train the model while it's live in production (learning & use are effectively the same thing).

Bad, BAD idea. Remember the Tay chatbot, which 4chan managed to turn into a raging Nazi in the matter of a few hours?

>"the company’s CEO, Sam Altman, says further progress will not come from making models bigger. “I think we're at the end of the era where it's going to be these, like, giant, giant models,” he told an audience at an event held at MIT late last week. “We'll make them better in other ways.”

So to reiterate, he is not saying that the age of giant AI models is over. Current top-of-the-line AI models are giant and likely will continue to be. However, there's not point in training models you can't actually run economically. Inference costs need to stay grounded which means practical model sizes have a limit. More effort is going to go into making models efficient to run even if it comes at the expense of making them less efficient to train.

Quality over quantity. Just building a model with a gazillion parameters isn't indicative of quality, you could easily have garbage parameters with tons of overfitting. It's like megapixel counts in cameras: you might have 2000 gigapixels in your sensor, but that doesn't mean you're going to get great photos out of it if there are other shortcomings in the system.

What overfitting? If anything, LLMs suffer from underfitting, not overfitting. Normally, overfitting is characterized by increasing validation loss while training loss is decreasing, and solved by early stopping (stopping before that happens). Effectively, all LLMs are stopped early, so they don't suffer from overfitting at all.

Is cost really that much of a burden?

Intelligence is the single most expensive resource on the planet. Hundreds of individuals have to be born, nurtured, and educated before you might get an exceptional 135+ IQ individual. Every intelligent person is produced at a great societal cost.

If you can reduce the cost of replicating a 135 IQ, or heck, even a 115 IQ person to a few thousand dollars, you're beating biology by a massive margin.

But we're still nowhere near that, or even near surpassing the skill of an average person at a moderately complex information task, and GPT-4 supposedly took hundreds of millions to train. It also costs a decent amount more to run inference on it vs. 3.5. It probably makes sense to prove the concept that generative AI can be used for lots of real work before scaling that up by another order of magnitude for potentially marginal improvements.

Also, just in terms of where to put your effort, if you think another direction (for example, fine-tuning the model to use digital tools, or researching how to predict confidence intervals) is going to have a better chance of success, why focus on scaling more?

The reason we put everyone through school is we believe that it’s in society’s best interest to educate everyone to the peak of their abilities. It’s good for many different reasons.

It would be much easier to identify gifted kids and only educate them, but I happen to agree that universal education is better.

Are intelligent people that valuable? There's lots of them at every university working for peanuts. They don't seem to be that valued by society, honestly.

This only makes sense if you use "IQ" ignoring the actual definition of "IQ", in which case it's silly to use numbers in your post to make it look technical.

IQ 1. can't be compared against generations of IQ tests 2. supposedly doesn't test education (of course, it actually does) 3. isn't real.

Since IQ is just a normal distribution on a population it is a bit misleading to talk about it like that.

Even if we don't expend any cost on education the number of people with IQ 135 stays the same.

My IQ is 140 and I'm far from exceptional.

115 IQ isn’t all that high- that’s basically every Indian American or a healthy percentage of the Chinese population.

Edit: I don’t understand the downvotes. I don’t mean this in any disparaging way, just that an AGI is probably going to be a lot higher than that.

Yes, but it also tells us that if Altman is honest here, then he doesn’t believe GPT-like models can scale to near level human performances (because even if the cost of compute was 10x or even 100x it would still be economically sound).

No it doesn't.

For one thing they're already at human performance.

For another, i don't think you realize how expensive inference can get. Microsoft with no scant amount of available compute is struggling to run gpt-4 such that they're rationing it between subsidiaries while they try to jack up compute.

So saying, it would be economically sound if it cost x10 or x100 what it costs now is a joke.

I've been training large 65b models on "rent for N hours" systems for less than 1k per customized model. Then fine tuning those to be whatever I want for even cheaper.

2 months since gpt 4.

This ride has only just started, fasten your whatevers.

Finetuning cost are nowhere near representative of the cost to pre-train those models.

Trying to replicate the quality of GPT-3 from scratch, using all the tricks and training optimizations in the books that are available now but weren't used during GPT-3 actual training, will still cost you north of $500K, and that's being extremly optimistic.

GPT-4 level model would be at least 10x this using the same optimism (meaning you are managing to train it for much cheaper than OpenAI). And That's just pure hardware cost, the team you need to actually makes this happen is going to be very expensive as well.

edit: To quantify how "extremely optimistic" that is, the very model you are finetuning, which I assume is Llama 65B, would cost around ~$18M to train on google cloud assuming you get a 50% discount on their listed GPU prices (2048 A100 GPUs for 5 months). And that's not even GPT-4 level.

You are confusing training with fine-tuning which is a different beast.

"The great supercomputer Deep Thought is the most powerful computer ever built, with one exception. It was designed by hyper-intelligent pan-dimensional beings, who wanted to know the answer to Life, the Universe, and Everything.

Its creation annoyed a fair few philosophers, who felt that it was taking over their turf.

After seven and a half million years of serious cogitation, Deep Thought spoke the answer. However, it was so inexplicable that Deep Thought then had to go on and design the most powerful computer ever built (with no exceptions) to work out what the question was."

Feels very much like we are entering the Happy Vertical People Transporters era.

How long till the air con goes on strike for miserable working conditions?

"Brain the size of a planet and they ask me write a lesson plan in the style of a pirate" - chatgpt5, probably...

And that was earth, so are we asking the right questions?

and the answer to question of life is 42.

I'd bet that what he, and the competition, is realizing is that the bigger models are too expensive to run.

Pretty sure Microsoft swapped out Bing for something a lot smaller in the last couple of weeks; Google hasn't even tried to implement a publicly available large model. And OpenAI still has usage caps on their GPT-4.

I'd bet that they can still see improvement in performance with GPT-5, but that when they look at the usage ratio of GPT3.5 turbo, gpt3.5 legacy, and GPT4, they realized that there is a decreasing rate of return for increasingly smart models - most people don't need a brilliantly intelligent assistant, they just need a not-dumb assistant.

Obviously some practitioners of some niche disciplines (like ours here) would like a hyperintelligent AI to do all our work for us. But even a lot of us are on the free tier of ChatGPT 3.5; I'm one of the few paying $20/mo for GPT4; and idk if even I'd pay e.g. $200/mo for GPT5.

> I'd bet that what he, and the competition, is realizing is that the bigger models are too expensive to run.

I think it's likely that they're out of training data to collect. So adding more parameters is no longer effective.

> most people don't need a brilliantly intelligent assistant, they just need a not-dumb assistant.

I tend to agree, and I think their pathway toward this will all come from continuing advances in fine tuning. Instruction tuning, RLHF, etc seem to be paying off much more than scaling. I bet that's where their investment is going to be turning.

I think Sam is referring to transition from "Deep" to "Long" learning [1]. What new emergent properties, if any, will 1 billion tokens unlock?

[1] https://hazyresearch.stanford.edu/blog/2023-03-27-long-learn...

I don't quite understand this. The returns to scale have always been sublinear (ie "diminishing"), but the scale-maximalists didn't worry about that before. Also I heard Sam tout on the Lex Friedman podcast how amazing it is that the scaling laws worked so well for GPT-4. So I wonder what changed?

Sure yeah the cost numbers are getting very large and we can't keep scaling forever. But Google could easily 10x the training cost of GPT-4 if they thought it would protect their search business. I'm still skeptical that scaling is enough to reach the thresholds we want, but I'm surprised that it's being claimed right now when there's a huge rush of new money into the space. I wonder if this is some sort of misdirection by Sam

> So I wonder what changed?

Maybe cost & latency for both training and inference it getting too high. If costs doubled for every 5% better performance, would it be worth it? NVIDIA is making a small fortune from this.

> But Google could easily 10x the training cost of GPT-4 if they thought it would protect their search business

Google makes /\$0\.[0+]\d/ per search query. If the inference cost of the model exceeds that, they go from making money to losing money. It is not clear if the Bing integration is a money maker or a lost leader.

Once you've trained on the internet and most published books (and more...) what else is there to do? You can't scale up massively anymore.

You could have it start talking to itself in the way that AlphaGO learns to get better at Go. All that needs to be done is find some fitness function that indicates that useful knowledge has been produced. In Go and Chess this is easy.

It can start posting synthesized ideas on social media and see how many likes it gets. Coupled with a metric containing dissimilarity to current information, this could be a useful way to progress to superhuman insights.

Ilya Sutskever (OpenAI Chief Scientist): "Yeah, I would say the data situation is still quite good. There's still lots to go" - https://youtu.be/Yf1o0TQzry8?t=685

There was a rumor that they were going to use Whisper to transcribe YouTube videos and use that for training. Since it's multimodal, incorporating video frames alongside the transcriptions could significantly enhance its performance.

If you were devious enough, you could be listening in on billions of phone conversations and messages and adding that to your data set.

This also makes me doubt that NSA hasn't already cracked this problem. Or that China won't eventually beat current western models since it will likely have way more data collected from its citizenry.

You can transcribe all spoken words everywhere and keep the model up to date? Keep indexing new data from chat messages, news articles, new academic work etc.

The data is not finite.

Video content (I don't know why someone flagged Jason for saying such, he is totally right)

Right. They've already sucked in most of the good general sources of information. Adding vast amounts of low-quality content probably won't help much and might degrade the quality of the trained model.

Videos - all of youtube, all the movies, everything that's ever been captured on film. Transcribe the audio, automatically describe the images and try to predict the next one.

Youtube. This is where Google have huge advantage having largest collection of user generated video

They didn't train it on the entire internet tho, only a small amount (in comparison to entire internet). Still plenty they could do.

Real answer? Buy proprietary data from social media companies, credit card companies, retail companies and train the model on that data.

Video.

> YouTubers upload about 720,000 hours of fresh video content per day. Over 500 hours of video were uploaded to YouTube per minute in 2020, which equals 30,000 new video uploads per hour. Between 2014 and 2020, the number of video hours uploaded grew by about 40%.

I doubt they have trained on 0.1% of the tokens that are 'easily' available (that is, available with licencing deals that are affordable to OpenAI/MSFT).

They might have trained on a lot of the 'high quality' tokens, however.

people seem to have forgotten about the multi-modal GPT-4

There's a ton of potential left on the table. The question is if transformers have hit their limit with GPT-4 or not.

It's a pretty simple equation when you think about it this way and why Sam would say they have hit their limit. Sam is basically Microsoft and they want to retain their lead. Once Google learns to put their data to use correctly, it's almost guaranteed game over for OpenAI if they want it to be.

You can generate textual examples that teach logic, multi-dimensional understanding and so on. Similar to the ones that are in math books, but in a massive scale.

> Once you've trained on the internet and most published books (and more...) what else is there to do? You can't scale up massively anymore.

Dataset size is not relevant to predicting the loss threshold of LLMs. You can keep pushing loss down by using the same sized dataset, but increasingly larger models.

Or augment the dataset using RLHF, which provides an "infinite" dataset to train LLMs on. Limited by the capabilities of the scoring model which, of course, you can scale the scoring model infinitely so again the limit isn't dataset size but training compute.

In other words, "this approach is getting to expensive for us and we need to control the narrative for our model sizes going down".

Most increases in Imagenet etc scores came from bigger models. "The Unreasonable Effectiveness of Data" has aged very well. It seems very convenient for OpenAI that this trend should be over a few months after their "eye-wateringly expensive" comment.

SamA strikes me as overly disingenuous - as if most of his interactions are marketing pieces / warfare. The entire lex + sama podcast seemed this way to me.

I’m incredibly biased though and feel slighted (as a part of humanity) by openAI’s actions, so perhaps I’m looking at his actions through a hateful lens.

I feel the same. His narrative seem very calculated. It doesn't help that OpenAI is very much the most closed Saas thing to be released in recent years. I am a paying customer for sometime, have ChatGPT Plus as well. I am still in waiting list for GPT-4 API.

Not only that, but their business model is completely unclear, which is the scariest part for me, as an individual developer.

Let's go one step forward, they use our texts online including this very comment perhaps. They have a software that can analyse all our texts at scale. But we don't get even the API access to this thing.

It is the most self-serving software ever released, consuming the human creativity both the content and the future relevance of it.

They could have it open source and people like Gerganov could rewrite the code in C/C++, but god forbid they actual are open

https://archive.is/s4V9e

He did not say what kind of research strategies or techniques might take its place. In the paper describing GPT-4, OpenAI says its estimates suggest diminishing returns on scaling up model size. Altman said there are also physical limits to how many data centers the company can build and how quickly it can build them.

> In the paper describing GPT-4, OpenAI says its estimates suggest diminishing returns on scaling up model size.

I read the two papers (gpt 4 tech report, and sparks of agi) and in my opinion they don't support this conclusion. They don't even say how big GPT-4 is, because "Given both the competitive landscape and the safety implications of large-scale models like GPT-4, this report contains no further details about the architecture (including model size), hardware, training compute, dataset construction, training method, or similar."

> Altman said there are also physical limits to how many data centers the company can build and how quickly it can build them.

OK so his argument is like "the giant robots won't be powerful, but we won't show how big our robots are, and besides, there are physical limits to how giant of a robot we can build and how quickly we can build it." I feel like this argument is sus.

The Age of Giants is over... The Age of Behemoths has begun!

but sir, that means the same thing

Throw this heretic into the pit of terror.

The pit of terror is full.

Fine, to the outhouse of madness then.

Before I get nuked from orbit for daring to entertain humor, if someone is running ahead of me in a marathon, and running so far ahead, yet still broadcasting things to the back for the slow people (like myself), then eventually we catch up to them, and they suddenly say, you know what guys, we should stop running in this direction, there's nothing to see here right before anyone else is able to verify the veracity of their statement, perhaps it would still be in the public interest for at least one person to verify what they are saying. Given how skeptical the internet at large has been of Musk's acquisition of a company, it's interesting that the skepticism is suddenly put on hold when looking at this part of his work...

Eh.

Altman has a financial incentive to lie and obfuscate about what it takes to train a model like GPT-4 and beyond, so his word is basically worthless.

First of all, if Altman continually makes misleading statements about AI he will quickly lose credibility, and that short term gain in whatever 'financial incentive' that birthed the lie would be eroded in short order by a lack of trust of the head of one of the most visible AI companies in the world.

Secondly, all the competitors of OpenAI can plainly assess the truth or validity of Altman's statements. There are many companies working in tandem on things at the OpenAI scale of models, and they can independently assess the usefulness of continually growing models. They aren't going to take this statement at face value and change their strategy based on a single statement by OpenAI's CEO.

Thirdly, I think people aren't really reading what Altman actually said very closely. He doesn't say that larger models aren't useful at all, but that the next sea change in AI won't be models which are orders of magnitude bigger, but rather a different approach to existing problem sets. Which is an entirely reasonable prediction to make, even if it doesn't turn out to be true.

All in all, "his word is basically worthless" seems much to harsh an assessment here.

I've seen Altman say in an interview that training GPT-4 took "hundreds of little things".

I don't find this implausible, but it folds slightly to Ockham's razor when you consider that this is the exact type of statement that would be employed to obfuscate a major breakthrough.

It just makes me crook my eyebrow and look to more credible sources.

It is possible that GP meant that Altman’s word is basically worthless to them, in which case that’s not something that can be argued about. It’s a factually true statement that that is their opinion of that man.

I personally can see why someone could arrive at that position. As you’ve pointed out, taking Sam Altman at face value can involve suppositions about how much he values his credibility, how much stock OpenAI competitors put in his public statements, and the mindsets people in general have when reading what he writes.

Elon Musk has been constantly doing this and thriving.

Anyone with the expertise to have insightful takes in AI also has a financial incentive to steer the conversation in particular directions. This is also the case for many, many other fields! You do not become an expert by quarantining your livelihood away from your expertise!

The correct response is not to dismiss every statement from someone with a conflict of interest as "basically worthless", but to talk to lots of people and to be reasonably skeptical.

OpenAI has gone from open-sourcing its work, to publishing papers only, to publishing papers that omit important information, to GPT-4 being straight-up closed. And Sam Altman doesn't exactly have a track record of being overly concerned about the truth of his statements.

I had a fun conversation (more like argument) with ChatGPT about the hypocrisy of OpenAI. It would explicitly contradict itself and then began starting every reply with “I can see why someone might think…” and then just regurgitating fluff about democratizing AI. I finally was able to have it define democratization of technology and then recognize the absurdity of using that label to describe a pivot to gating models and being for-profit. Then it basically told me “well it’s for safety and protecting society”.

An AI, when presented with facts counter to what it thought it should say, agreed and basically went: “Won’t someone PLEASE think of the children!”

Love it.

This trend has happened in the small for their APIs as well. They've been dropping options - the embeddings aren't the internal embeddings any more, and you don't have access to log probabilities. It's all closing up at every level.

It's incredible that people are so eager to eat up these unsupported claims.

This is the second [1] OpenAI claim in the span of a few days that conveys a sense of "GPT-4 represents a plateau of accomplishment. Competitors, you've got time to catch up!".

And it's not just a financial incentive, it's a survival incentive as well. Given a sufficiently sized (unknowable ahead of time) lead, the first actor that achieves AGI and plays their cards right, can permanently suppress all other ongoing research efforts should they wish to.

Even if OpenAI's intentions are completely good, failure to be first could result in never being able to reach the finish line. It's absolutely in OpenAI's interest to conceal critical information, and mislead competing actors into thinking they don't have to move as quickly as they can.

[1] https://news.ycombinator.com/item?id=35570690

In this case I think it's Wired that's lying. Altman didn't say large models have no value, or that there will be no more large models, or that people shouldn't invest in large models.

He said that we are at the end of the era where capability improvements come primarily from making models bigger. Which stands to reason... I don't think anyone expect us to hit 100T parameters or anything.

"640KB of memory ought to be enough for anybody!"

- Abraham Lincoln

What about 1T though, seems silly to stop here.

It could also be argued that there's financial incentive to just saying "giving us more money to train bigger models = better AI" forever

I don't think these comments are driven from financial incentives. It's a distraction and only a fool would believe Altman here. What this likely means is they are prioritizing adding more features to their current models while they train the next version. Their competitors scramble to build an LLM with some sort of intelligence parity, when that happens no one will care because ChatGPT has the ecosystem and plugins and all the advanced features....and by the time their competitors reach feature parity in that area, OpenAI pulls its Ace card and drops GPT5. Rinse and repeat.

That's my theory and if I was a tech CEO in any of the companies competing in this space, that is what I would plan for.

Training an LLM will be the easy part going forward. It's building an ecosystem around it and hooking it up to everything that will matter. OpenAI will focus on this, while not-so-secretly training their next iterations.

Does he even have any background in machine learning? I always found it bizarre that he was chosen to be OpenAI's CEO...

On the Lex Fridman podcast, he pretty much admitted he's not an AI (per se) and isn't the most excited about the tech (as he could be).

He has background in CEO (smooth-talking charmer in the VC crowd). That's why he's CEO.

Do you think GPT-4 was trained and then immediately released to the public? Training finished Aug 2022. They spent the next 6 months improving it in other ways (eg human feedback). What he is saying is already evident therefore.

People can be honest even when money is involved. His word is worthless because it's Altman

IIRC Altman has no financial stake in the success or failure of OpenAI to prevent these sorts of conflicts of interests between OpenAI and society as a whole

https://www.cnbc.com/2023/03/24/openai-ceo-sam-altman-didnt-... (https://news.ycombinator.com/item?id=35289044 - 24 days ago; 158 points, 209 comments)

> OpenAI’s ChatGPT unleashed an arms race among Silicon Valley companies and investors, sparking an A.I. investment craze that proved to be a boon for OpenAI’s investors and shareholding employees.

> But CEO and co-founder Sam Altman may not notch the kind of outsize payday that Silicon Valley founders have enjoyed in years past. Altman didn’t take an equity stake in the company when it added the for-profit OpenAI LP entity in 2019, Semafor reported Friday.

Right. All the evidence points to more potential being left on the table for emergent abilities. It would make no sense that the model would develop all of these complex skills for better predicting the next token, then just stop.

It's a massive bet for a company to push compute into the billion dollar range - if saying something like this has the potential to help ward off those decisions, I don't see what's stopping them from saying it.

I basically see Microsoft talking when Sam talks.

Citation needed. What are his financial incentives?

We will need a combination of technologies we have in order to really achieve emergent intelligence.

Humans are comprised of various "subnets" modelling aspects which, in unison, produce self-conciousness and real intelligence. What is missing in the current line of approaches is that we only rely on auto-alignment of subnetworks by machine learning, which scales only up to a point.

If we would produce a model which has

* something akin a LLM as we know it today, which is able to

* store or fetch facts to a short- ("context") or longterm ("memory") storage

* if not in the current "context", query the longterm context ("memory") by keywords for associations, which are one-by-one inserted into the current "context"

* repeat as required until fulfilling some self-defined condition ("thinking")

To me, this is mostly mechanical plumbing work and lots of money.

Also, if we get rid of the "word-boundedness" of LLMs - which we already did to some degree, as shown by the multi-language capabilities - LLMs would be free to roam in the domain of thoughts /s :)

This approach could be further improved by meta-LLMs governing the longterm memory access, providing an "intuition" which longterm memory suits the provided context best. Apply recursion as needed to improve results (paying by exponential training time, but this meta-NN will quite probably be independent of actual training, as real life / brain organization shows).

The other elements that may be required could be some version of the continuous sensory input that to us creates the sensation of "living" and, this one is a bit more philosophical, the sensation of suffering and a baseline establishment that the goal of the entity is to take actions that help it avoid suffering.

Check out David Shapiro's cognitive architecture stuff.

Saying “hey don’t go down the path we are on, where we are making money and considered the best in the world.. it’s a dead end” rings pretty hollow.. like “don’t take our lunch please?” Might be a similar statement it feels..

Nah - GPT-4 is crazy expensive, paying 20$/mo only get's you 25messages/3hours and it's crazy slow. The api is rather expensive too.

I'm pretty sure that GPT-4 is ~1T-2T parameters, and they're struggling to run it(at reasonable performance and profit). So far their strategy has been to 10x the parameter count every GPT generation, and the problem is that there's diminishing returns everytime they do that. AFAIK they've now resorted to chunking GPT through the GPUs because of the 2 to 4 terabytes of VRAM required (at 16bit).

So now they've reached the edge of what they can reasonably run, and even if they do 10x it the expected gains are less. On top of this, models like LLaMa have shown that it's possible to cut the parameter count substantially and still get decent results (albiet the opensource stuff still hasn't caught up).

On top of all of this, keep in mind that at 8bit resolution 175B parameters (GBPT3.5) requires over 175GB of VRAM. This is crazy expensive and would never fit on consumer devices. Even if you use quantization and use 4bit, you still need over 80GB of VRAM.

This definitely is not a "throw them off the trail" tactic - in order for this to actually scale the way everyone envisions both in performance and running on consumer devices - research HAS to be on improving the parameter count. And again there's lots of research showing its very possible to do.

tl;dr: smaller = cheaper+faster+more accessible+same performance

It's a pretty sus argument for sure when they're scared to release even parameter size.

although the title is a bit misleading on what he was actually saying. still, there's a lot left to go in terms of scale. Even if it isn't parameter size(and there's still lots of room here too, it just won't be economical), contrary to popular belief, there's lots of data left to mine

Everyone hoping to compete with OpenAI should have an "Always do the opposite of what Sam says" sign on the wall.

Ok cool, so release the weights and your research.

Ilya Sutskever from OpenAI saying that the data situation is good and there's more data to train on - https://youtu.be/Yf1o0TQzry8?t=657

Maybe more data to train on while keeping model size the same

Sam Altman and OpenAI must be pretty nervous. They have first mover advantage but they hold no hook or moat.

Unless they can somehow keep their improvements ahead of the rest of the industry then they'll be lost among a crowd.

Yeah. Sadly I believe they may lose their spot due to their concern for safety. They got here by pushing the envelope. But now with everyone freaking out, he's not putting off the training of a new model due to 'safety issues' (with plugins i suspect)

On the other hand, if Microsoft can change enough people's habits of using Google to search, they'll be happy as a clam.

Is anyone aware of techniques to prune a model from useless knowledge to leave more space for the reasoning capabilities?

It really shouldn't matter that it can give the exact birthdate of Steve Wozniac, as long as it can properly make a query to fetch it and deal with the result.

I follow your design, couldn't you also solve hallucinations with a "fact checking" LLM (connected to search) that corrects the output of the core LLM? You would take the output of the core LLM, send it to the fact checker with a prompt like "evaluate this output for any potential false statements, and perform an internet search to validate and correct them"

You can't trust what sama says. Period. He's shown his willingness to deceive for gain. He deleted the codex model because it was better than the text model and only way to explain it is that by training on code instead of just text of human interactions the model develops stronger reasoning abilities (because of higher clarity around logic and higher signal to noise overall in the training data.) So he removed that so researchers don't catch on. Also, why 1 Trillion params is the limit? Why not 2 Trillion. He's been begging for regulations and trying to pull up the ladder behind him. Just really "street smart" thinking but I have not seen any elevated humanity serving thinking coming form him. He would not have taken $100M for a non-profit to do open source research only to bait-and-switch into for profit closed source sold to the highest bidder. Give me a break. Don't believe anything he says (well, use your brain, as he may mix truths and deceptions with the goal of deceiving his competitors... he's not out there to do a service for humanity... if he was, he would have kept it open and non-profit, not basically steal the funds of the non-profit to pocket $29B from it.) It's ridiclous.

The role of a CEO is more to benefit the company than the public. Only time will tell.

I am curious though how something like Moore's Law relates to this. Yes, model architectures will deal with complexity better and the amount of data helps as well. There must be a relation between technology innovation and cost which alludes to effectiveness. Innovation in computation, model architecture, quality of data, etc.

The 2017 Transformers paper has ~71,000 papers citing it. The sheer magnitude of human mental effort globally that is chasing the forefront of machine learning is unprecedented and amazing.

Local minimum. LeCun is right.

"When we set the upper limit of PC-DOS at 640K, we thought nobody would ever need that much memory."

Bill Gates

Gates has refuted saying this. Are you implying by analogy that Altman hasn't said/will disclaim saying that "the age of giant AI models is almost over"?

That was in context of a how much RAM would be needed for software of that era, not the entire future of computing.

Interesting how this contradicts "The Bitter Lesson": http://incompleteideas.net/IncIdeas/BitterLesson.html.

I don't think there is a contradiction at all. Altman is essentially saying they are running out of compute and therefore can't meaningfully scale further. Not that scaling further would be a worse plan longterm than coming up with new algorithms.

There's plenty of room for models to continue to grow once efficiency is improved. The basic premise of the Google ML pathways project is sound, you don't have to use all the model all the time. By moving to sparse activations or sparse architectures you can do a lot more with the same compute. The effective model size might be 10x or 100x GPT-4 (speculated at 1T params) but require comparable or less compute.

While not a perfect analogy it's useful to remember that the human brain has far more "parameters", requires several orders of magnitude less energy to train and run, is highly sparse, and does a decent job at thinking.

The efficiency thing is what worries me. I think Nvidia has rough ideas for increasing efficiency 100 - 1000 times without changing the fundamental paradigm (i.e. memristors or crossbar arrays or something). If it doesn't go to 1000, I assume there is a lot of investment ramping up for realizing the new fully compute-in-memory systems.

If the software can leverage these efficiency gains effectively, then the concerns about runaway AI will be very relevant. Especially since people seem to think that they need to emulate all animal (like human) characteristics to get "real" general intelligence. Despite the fact that GPT is clearly general purpose. And people make no real differentiation between the most dangerous types of characteristics like self-preservation or full autonomy.

GPT shows that we can have something like a Star Trek computer without creating Data. People should really stop rushing their plans to create an army of Datas and then enslave them. Totally unnecessary and stupid.

I believe Altman, but the title is misleading.

Have we exhausted the value of larger models on current architecture? Probably yes. I trust OpenAI would throw more $ at it if there was anything left on the table.

Have we been here before? Also yes. I recall hearing similar things about LSTMs when they were in vogue.

Will the next game changing architecture require a huge model? Probably. Don’t see any sign these things are scaling _worse_ with more data/compute.

The age of huge models with current architecture could be over, but that started what, 5 years ago? Who cares?

Also scaling doesn't address some of the challenges for ai that chatGPT doesn't meet, like:

- learning to learn, aka continual learning - internalised memory

bringing it closer to actual human capabilities.

The age of CEOs that recently got washed to the top saying dumbish things is just starting, though.

We arrived at the top of the tree in our journey to the moon.

No we haven't. the title is misleading. there's plenty of scale room left. part of it might just not be economical (parameter sie) but there's data. If you take this to mean, "we're at a dead end" you'd be very wrong

"You can't get to the moon by climbing successively taller trees"

It's probably economically difficult to rationalize building a GPT5. They're already having a hard time getting everyone GPT4. Better to spend the resources optimizing to get everyone onto a cheap(er) GPT4-comparable model than build a marginally better model that is even more expensive to scale. Like most tech there is a sigmoid curve where you start to see diminishing returns.

LLM's will be treated much like occupations for people going forward. Just like our own education, there will be a basic set of skills all LLM's need to be trained on to provide a minimal useable product. With GPT4 I suspect we are at this point now if not slightly beyond the optimal basic training stage. However, now it is time to send our LLM's off to university so to speak, to specialise them in certain tasks and knowledge base. The current approach of attempting to just build everything in to one does not work for a human mind, so why would it work for a vector based replication?

I suspect in the coming months we will hear more about tiny models trained on much smaller datasets and then specialised using a mix of adaptors and LoRA modifications to excel at specific tasks like code generation, translation, and conversation. Then multiple models will be implemented in one application chain to best leverage each of the respective strengths.

What age? Like, 3 years?

On the other hand though, Chinchilla and multimodal approaches already showed how later AIs can be improved beyond throwing petabytes of data at them.

It is all about variety and quality from now on I think. You can teach a person all about the color zyra but without actually ever seeing it, they will never fully understand that color.

It does seem, though, that using chinchilla like techniques does not create a copy with the same quality as the original. It's pretty good for some definition of the phrase, but it isn't equivalent, it's a lossy technique.

I agree on the lossy. There is a tradeoff between efficiency and comprehensiveness, kind of. It would be pretty funny if in the end, the most optimal method turns out to be the brain we already have. Extremely efficient, hardware optimized, but slow as hell and misunderstand stuff all the time unless prompted with specific phrases.

Something kind of funny (but mostly annoying), about this announcement is the people arguing that OpenAI is, in fact, working on GPT-5 in secret.

To my knowledge, NFT/crypto hype never got so bad that conspiracy theories began to circulate (though I’m sure there were some if you looked hard enough).

Can’t wait for an AIAnon community to emerge.

Isn’t it obvious? Q is definitely an LLM, trained on trillions of words exfiltrated from our nation’s secure systems. This explains why it’s always wrong in its predictions: it’s hallucinating!

Yeah I think what's happened to OpenAI is the same happening to all tech companies. They were spending a lot of money building these LLM's and now they've been told to cut back costs they no longer have the funding to build the next "double the size of the last one" model.

Interestingly what's makes ChatGPT work is the size of the model so I think they've found their dead stop.

Lot's of the bullish comments have been talking about how ChatGPT is a bit shit right now but will exponentially get better and I think the answer is now the progress will be much slower and more linear. That is if they can stay funded which is a very big if. As an org they are bleeding money.

"Altman said there are also physical limits to how many data centers the company can build and how quickly it can build them."

An H100 has 80GB of VRAM. The Highest end system I can find is 8xH100. Is a 640GB model is the biggest model you can run on a single system? Already GPT-4 is throttled and has a waiting list and they haven't even released the image processing or integrations to a wide audience. Maybe they are just unable to do inference in a cost-efficient manner and at an acceptable speed on anything bigger than GPT-4?

You might not see it available via self serve interfaces, but I think the h100 cards can do 256x to a single host.

You're right! You can nvlink 32 systems with 8 h100s! That's 20 terabytes of VRAM total!

>Maybe they are just unable to do inference in a cost-efficient manner and at an acceptable speed on anything bigger than GPT-4?

Yes, that's the whole thing. As others have pointed out, GPT-4 seems like an optimum point that balances cost, ROI, etc... it won't improve much just by throwing more data at it.

Sorry, but this sounds a lot like 640KB is all the memory you will ever need. What about “Socratic model” for video? There should me many applications that would benefit from a bigger model

This is just push back from elon and crews fake article about the dangers of AI, they specifically state the next versions will be ultra deadly.

Sam is now saying there will be no future model that will be as good.

This is all positioning to get regulators off the track because none of these control freaks in government actually understand a whit of this.

All said and done, this all just to try to disempower the OSS community. But they can't, we're blowing past their barriers like the 90s did with the definition of slippery slope.

He isn't saying that. Only that different approaches are needed to produce better results. If he's confident to say this I think he must have something up his sleeve.

Bruv has to pay for the data he's been using or soon there won't be any to nick on. Groupies claiming their ai is "intelligent", and not just a data ingesting beast, will soon learn a heard lesson. Take your blogs offline, stop contributing content for free and stop pushing code or else chavs like this one will continue monetising your hard work. As did bezos and many others that now want you to be out of a job.

I'm no expert but doesn't the architecture of minigpt4 that's on the front page right now give some indication of what the future might look like?

eh, I haven't personally found a usecase for LLMs yet given the fact that you can't trust the output and it needs to be verified by a human (which might as well be just as time consuming/expensive as actually doing the task yourself)

I’d reconsider the “might as well just be as time consuming” thing. I see this argument about Copilot a lot, and it’s really wrong there, so it might be wrong here too.

Like, for most of the time I’m using it, Copilot saves me 30 seconds here and there and it takes me about a second to look at the line or two of code and go “yeah, that’s right”. It adds up, especially when I’m working with an unfamiliar language and forget which Collection type I’m going to need or something.

Thing is, you can't trust what you find on stack overflow or other sources either. And searching, reading documentation and so on takes a lot of time too.

I've personally been using it to explore using different libraries to produce charts. I managed to try out about 5 different libraries in a day with fairly advanced options for each using chatGPT.

I might have spent a day in the past just trying one and not to the same level of functionality.

So while it still took me a day, my final code was much better fitted to my problem with increased functionality. Not a time saver then for me but a quality enhancer and I learned a lot more too.

They're good for tasks where generation is hard but verification is easy. Things like "here I gesture at a vague concept that I don't know the name of, please tell me what the industry-standard term for this thing is" where figuring out the term is hard but looking up a term to see what it means is easy. "Create an accurate summary of this article" is another example - reading the article and the summary and verifying that they match may be easier than writing the summary yourself.

the way i see it, the expensive part should be to train the models via simulated architectures in GPUs or TPUs or whatever.

but once they are trained, is there a way to encode the base models into hardware where inference costs are basically negligible? hopefully somebody is seeing if this is possible, using structurally encoded hardware to make inference costs basically nil/constant.

I don't understand why size is an issue in the way that is being claimed here.

Intelligence isn't like processor speed. If I have a model that has (excuse the attempt at a comparison) 200 IQ, why would it matter that it runs more slowly than a human?

I don't think that, for example, Feynman at half speed would have had substantially fewer insights.

We're not going to get a 200 IQ model by simply scaling up the current model, even with all the datacenters in the world running 24/7

I didn't think this article was very good. Sam Altman actually implied that GPT-5 will be developed when he spoke at MIT. And if Sam said that scaling is over (I doubt he said this but I could be wrong) the interesting part would be the reasoning he provided for this statement - no mention of that in the article.

All warfare is based on deception -- Sun Zu

Wait till some one pulls a Cerberus for MEMORY. And than we connect a wafer scale processor with wafer scale memory and BOOM. New Mega size works emerge.

As for current claim, it might have to do with the amount of time spent taming the wildness of answer by raw GPT-4. So focus is shifting from increasing the model size.

"...for the current cycle, in our specific public-facing market."

As most here well know "over" is one of those words like "never" which particularly in this space should pretty much always be understood as implicitly accompanied by a footnote backtracking to include near-term scope.

An amusing thought I've had recently is whether LLMs are in the same league as the millions of monkeys at the keyboard, struggling to reproduce one of the complete works of William Shakespeare.

But I think not, since monkeys probably don't "improve" noticeably with time or input.

But I think not, since monkeys probably don't "improve" noticeably with time or input.

Maybe once tons of bananas are introduced...

He's finally catching up with me. I wrote an article about that a few months ago.

https://twitter.com/igorcosta/status/1612704076315389952

I wonder how much the scarcity and cost of Nvidia GPUs is driving this message?

Nvidia is in a perfect "Arms Dealer" situation right now.

Wouldn't be surprised to see the next exponential leap in AI models trained on in-house proprietary GPU hardware architectures.

Google has been using TPUs for years and continuously improving the designs.

dudes. if none of you see it, it's because you're turning a blind eye, is the best way I can say it. the future of the 50's is about to show up, if, and only if, we're not so dumb as to ignore it. flying cars, killer robots, we can help decide, if we're only aware enough to do a bit of research, and say anything. your voice does matter :D

watch this: https://www.youtube.com/watch?v=VcVfceTsD0A&ab_channel=LexFr...

then, think, and then, think again, and then comment be optimistic, but be forewarned, and speak your voice

Maybe they're now training teams of AIs to work together? This way the hardware limitations are less of a barrier in terms of maximum memory per AI. They can run in parallel in separate hardware and communicate to reach higher goals.

Someone described me a emerging rule called “Penelope‘s rule” where you scale by having the largest and most powerful model delegate to the less powerful models and then have the larger model improve upon the results. (Eg have GPT-4 delegate the more mundane tasks to GPT-3.5).

Hm, all right, I'm guessing that huge models as a business maybe are over until economics are figured out, but huge models as experts for knowledge distillation seems reasonable. And if you pay a super premium can you use huge model.

GPT4 already obsolete? Geez, we got the wrong bot, silly heads: https://www.youtube.com/watch?v=EdRMVhlNP5I

It’s a good things there bots don’t have feelings yet because there’s going to be an lot obsolete bots once we’re done and getting made obsolete isn’t very fun.

I call bullshit. There will be bigger and better models. The question is not whether big companies will invest in training them (they will), but whether they’ll be made available to the public.

Those stochastic parrots are done for, finally! I have been telling people AI is a lie and it’s all a scam and now you can see for yourself. Buy nice warm jackets guys, winter is coming.

Sounds like we will be on gpt4 for a while, while we wait out a new “AI winter”, mind you there is still a lot to squeeze from gpt4 and local/edge llm models

it's over, billions of parameters must be released

Soon, the EU will pass a law that forces any AI company to provide you with a printout of the entire model upon request.

Hmmmmmm I know nothing about this but… guy who spends a bunch of money on thing to sell says others shouldn’t do it.

Sounds sus

Sorry I know this adds little to the conversation

There will still be giant AI models, they will just be improved by other means than making them bigger. Subtle but important distinction.

small AI model != cheap AI model.

It costs the same to train as these giant models. You merely spend they money on training it for longer instead of larger.

The most comforting AI news I have read this year.

If you are worried about AI, this shouldn't make you feel a ton better. GPT4 is just trained to predict the next word, a very simple but crude approach and look what it can do!

Imagine when a dozen models are wired together and giving each other feedback with more clever training and algorithms on future faster hardware.

It is still going to get wild

I agree, probably for a different reason. IMO the fact that Sam is saying this likely means that the LLMs are close to the upper knee of the S-Curve and after GPT5 they likely dont have many new fundamental ideas (additional SCurves) to throw at it. since they are ahead of the rest, it likely means we might be headed for an AI improvement pause for couple of years after GPT5.

Still good enough to upset the balance in search/ad market. Interesting times.

Title is misleading lol. Plenty of scale room left.

To those in the know, what are the potential successors to the Transformer architecture?

Good ideas here: https://youtu.be/mBjPyte2ZZo

"640K of memory is all that anybody with a computer would ever need"

No one will ever need more than 64k moment or whatever it was

train the entire model for a single question.

Now we need another letter

dude someone lied their way into being president of the united states all while people fact checked him basically immediately after each lie. lying doesnt make a difference.

"Eschew flamebait. Avoid generic tangents."

https://news.ycombinator.com/newsguidelines.html

We detached this subthread from https://news.ycombinator.com/item?id=35606550.

He’s not presenting false evidence here, he’s presenting a hunch. It’s a guess. No one is going to gain anything from this one way or another.

These kind of comments have me update my priors so I downgrade them in relevance. Two days ago we were talking about AGI and Altman was letting himself be cuddled, knowing full well it was mindless hype. Now he's trolling the other way. Ok. My weights have been updated.

Reply on news.ycombinator.com