Hacker News Reader

Comment by bubblelicious | Hacker News Reader

Where does this view come from? I’m not aware of any real evidence for this. Also consider our data center buildouts in 26 and 27 will be absolutely extraordinary, and scaling is only at the beginning. You have a growing flywheel and plenty of synthetic data to break the data wall

View on HN

Let me put it this way: when ChatGPT tells me I've hit the "Free plan limit for GPT-5", I don't even notice a difference when it goes away or when it comes back. There's no incentive for me to pay them for access to 5 if the downgraded models are just as good. That's a huge problem for them.

It is a problem easily solved with advertising.

No, because as the history of hardware scaling shows us, things that run on supercomputers today will run on smartphones tomorrow. Current models already run fairly well on beefy desktop systems. Eventually models the quality of ChatGPT 4 will be open sourced and running on commodity systems. Then what? There's no moat.

10-20 years of your data in the form of chat history

Billions of users allowing them to continually refund their models

Hell by then your phone might be the OpenAI 1. The world's first AI powered phone (tm)

> The world's first AI powered phone

Do you remember the Facebook phone? Not many people do, because it was a failed project, and that was back when Android was way more open. Every couple of years, a tech company with billions has the brilliant idea: "Why don't we have a mobile platform that we control?", followed by failure. Amazon is the only qualified success in this area.

I agree that a slight twist on android doesn't make sense. A phone with a in integrated LLM with apps that are essentially prompts to the LLM might be different enough to gain market share.

HP, Microsoft, and Samsung all had a go with non-Android OSes.

This based on any non anecdotal evidence by chance?

Of course not but explain how I am ever going to pay OpenAI, a for-profit company any dollars? Sam Altman gets explosive angry when he's asked about how he's going to collect revenue, and that is why. He knows when push comes to shove, his product isn't worth to people what it costs him to operate it. It's Homejoy at trillion dollar scale, the man has learned nothing. He can't make money off this thing which is why he's trying to get the government to back it. First through some crazy "Universal Basic Compute" scheme, now I guess through cosigning loans? I dunno, I just don't buy that this thing has any legs as a viable business.

Sell at a loss and make it up in volume.

It's been tried before, it generally ends in a crater.

I think you’re welcome to that opinion and are far from alone but (1) I am very happy to pay for Claude, even $200/mo is worth it and (2) idk if people just sort of lose track or what of how far things have come in the span of literally a single year, with the knowledge that training infra is growing insanely and people are solving on fundamental problem after another.

We live in a time when you can't even work for an hour and afford to eat a hamburger. You having the liquid cash to spend $200 a month on a digital assistant is the height of privilege, and that's the whole problem the AI industry has.

The pool of people willing to pay for these premium services for their own sake is not big. You've got your power users and your institutional users like universities, but that's it. No one else is willing to shell out that kind of cash for what it is. You keep pointing to how far it's come but that's not really the problem, and in fact that makes everything worse for OpenAI et al. Because, as they don't have a moat, they don't have customer lock-in, and they also soon will not have technological barriers either. The models are not getting good enough to be what they promise, but they are getting good enough to put themselves out of business. Once this version of ChatGPT gets small enough to fit on commodity hardware, OpenAI et al will have a very hard time offering a value proposition.

Basically, if OpenAI can't achieve AGI before ChatGPT4-type LLM can fit on desktop hardware, they are toast. I don't like those odds for them.

Ditto for Gemini Pro and Flash, which I have on my phone.

I've been traveling in a country where I don't speak the language and or know the customs, and I found LLMs useful.

But I see almost zero difference between paid and unpaid plans, and I doubt I'd pay much or often for this privilege.

We need a fundamental paradigm shift beyond transformers. Throwing more compute or data at it isn't pushing the needle.

And you don’t think that’s already happening? Also where is your evidence for this?

> Also where is your evidence for this?

The fact that "scaling laws" didn't scale? Go open your favorite LLM in a hex editor, oftentimes half the larger tensors are just null bytes.

Show me a paper, this makes no sense of course scaling laws are scaling

Just to point, but there's no more data.

LLMs would always bottleneck on one of those two, as computing demand grows crazy quickly with the data amount, and data is necessarily limited. Turns out people threw crazy amounts of compute into it, so the we got the other limit.

Synthetic data works.

There's a limit to that according to: https://www.nature.com/articles/s41586-024-07566-y . Basically, if you use an LLM to augment a training dataset it will become "dumber" every subsequent generation and I am not sure how you can generate synthetic data for a language model without using a language model

Synthetic data doesn't have to come from an LLM. And that paper only showed that if you train on a random sample from an LLM, the resulting second LLM is a worse model of the distribution that the first LLM was trained on. When people construct synthetic data with LLMs, they typically do not just sample at random, but carefully shape the generation process to match the target task better than the original training distribution.

Yeah I’m constantly reminded of a quote about this- you can’t make another internet. LLMs already digested the one we have.

Epoch has a pretty good analysis of bottlenecks here:

https://epoch.ai/blog/can-ai-scaling-continue-through-2030

There is plenty of data left, we don’t just train with crawled text data. Power constraints may turn out to be the real bottleneck but we’re like 4 orders of magnitude away

There is zero evidence that synthetic data will provide any real benefit. All common sense says it can only reinforce and amplify the existing problems with LLMs and other generative “AI”.

Sounds like someone has no knowledge of the literature, synthetic data isn’t like asking ChatGPT to give you a bunch of fake internet data.

Reply on news.ycombinator.com