famouswaffles

There's just not much value to a 'Every technology hit a brick wall eventually' take. Anyone can spout that. Why even bring 'credible AI researchers' if that's all it amounts to. What is relevant is the foresight to predict when and at what capabilities. It doesn't change much of anything if LLMs 'plateau' right after they can replace any human at any job.

They were not. That was OpenAI with o1

>Something like 99% of the people who have interacted with an LLM chatbot lose interest after a month.

This is not remotely true. It literally couldn't be true, considering Open AI have 1.2 Billion weekly active ChatGPT users.

>OpenAI said they sat on it for 8 months

They sat in it. We had concrete details ('rumors' that turned out surprisingly accurate) months before the public release.

>Yes, they do. Nothing in this sentence implies that what they have access to is "light years ahead of the public models".

"Light years" is subjective, but the accomplishments of those internal models we're priven to should be enough to give you some indication. You can fire up as many Astra agents as you like you're not going to get the performance of Open AI's internal model in math. Just as you can fire as many GPT-4 instances and you're not going to get anything resembling Opus 5.5.

OpenAI sat on GPT-4 for 8 months before actually releasing it and that was a major stride over 3. It's kind of silly not to realize they do indeed have access to models months ahead of the public at minimum. Nothing you have access to is the true frontier. It only takes some common sense to acknowledge that. It's not unbelieveable, you just don't want to believe it.

>That is different from spending 10s of millions to do a brute-force search of lots of different proofs which can be verified by LEAN automatically.

That is not remotely how the vast majority of the proofs have worked. You don't know what you're talking about, but that seems par the course for these discussions.

We can't rule out different architecures, but absent some architecure or hardware or quantisization breakthrough, frontier models will simply not be fitting on phones.

Right? Occasionally, I would think about how the concept of the singularity or recursive self improvement was a bit incongruous today. It was made when people expected generally intelligent machines to be these logic automatons that could 'alter their own code'. It seemed to me that this path would be closed off with the advent of Giant Neural Networks that take weeks/months to train. That GPT-X would 'improve itself' by designing a better/more efficient NN architecture. I didn't write it off completely but that just seemed a LOT more likely than altering weights directly for much the same reasons you describe. Maybe not lol.

This is pretty massive, and a result I (and I think most?) would have never expected. Holy Shit. How?? Did OpenAI figure out a way to train this? Are they even aware?

For A, a common misconception is that models write out lean proofs from the start. They write/develop the Natural Language proofs, and then translate (sometimes not even the same model) to lean afterwards.

Being racist comes easy for you huh?

Today with Opus, Sol, etc there's basically no reason to visit the site, but when chatGPT launched it was powered by GPT 3.5, which would realistically only be able to answer a (minor) chunk of all possible StackOverflow questions.

Software Development is far from the only white collar work frontier agents are very useful for. Most people just have no idea and marketing around them hasn't really struck. If you think it's just code or math that LLMs have gotten rapidly better at then you'd be wrong.

>For Python libraries they would be written in Python, eventually ctypes for syscalls.

So if you can't call/use the 'library' from the 'native' language, it's still a library? Whatever floats your boat. Lol.

I know what bindings are. A front-end to C++ for pytorch was only released from 1.0, literally 2 years later. The relationship is very clear here. There was no native C++ library. It was an afterthought. Even in instances where there was, like Tensorflow, functionally it was still an afterthought.

They are python libraries because people reached for those languages so they could continue work in python. They are there to serve python and nothing else. It's the same with Rust. It's by no means a C++ thing.

EDIT: Not saying this is always the case. Qt>Pyside is an example where it's both technically and functionally bindings, but the vast majority of the time they are serving python. Some don't even have a native library to speak of.

>the problem is that the Python community at large rather prefers their C bindings or nowadays Rust, sprinkle a bit of Python FFI on top, and call them "Python" libraries.

They are Python libraries. Don't make the mistake of screwing up the cause and effect here. C, Rust etc - those are the replaceable parts of the chain. There is an official C++ pytorch library and virtually nobody uses it for ML Research. People want to code in python and build those libraries so they can.

Results come from an internal model that has not been publicly released. And there are lesser problems and conjectures ordinary folks have solved with Astra/Sol.

Because I might look elsewhere if they starting charging several orders of magnitudes more then I'm not sticky. Does that remotely make any sense to you ?

I'm not rich and you don't have to be rich.

Most people buy their apples from the same grocery store every week. They're not shopping around every month to find who has the cheapest apples or 'best' apples. They go where they're accustomed to going, and as long as price and quality are satisfactory, they have little reason to change. Walmart, price-sensitive as they are, have relationships with fruit suppliers they've maintained for well on a decade. Even in commoditized markets, people are sticky.

Google, the titan mega-corp with more money than they know what to do with and a billion+ android devices to ship gemini on and it's made no dent for ChatGPT's consumer growth and adoption. Yet some plucky open source chinese models will?. It takes more than that.

Yes it is. Tiktok's content and content recommendation algorithms are much better than Instagram. Users spend more time on Tiktok than they do Instagram and it's not close. Tiktok's engagement rates are several times better than Instagram. Reels did nothing to dent it (yet another example in the book). Again, nobody will care about Chinese models if they're just more of the same. They can take a piece of the pie, but this idea that they'll spell doom for the frontier labs is nonsense.

Given we are comparing two different substrates, there's no reason to assume this means anything.

Yeah. People have very strange ideas about competition and choice and moat. If all you needed to unseat the entrenched was a competitor then Bing would be going toe to toe with Google, and Edge with Chrome. They have a billion+ devices they can stick those options as defaults and still nothing. There are many examples.

People are naturally sticky. If they're satisfied with a product, they'll stick around unless something comes along that is so much better it gets people talking. Open AI's 1B+ customers won't give a shit about Chinese models. Enterprise customers won't move to Chinese models, especially just to save money. They are a non issue, especially if the M.O is just going to be playing catch-up.

Difussion Gemma performed a lot worse than its equivalent autoregressive counterpart (it was particularly bad because rather than a sort of constant-ish gap you might see between 2 model classes, the gap seemed to worsen the harder the task/benchmark) and the speed benefits got negated at scale. Not really surprising we haven't heard much.

No it's already explicitly stated that translation to lean is a second step that happens after NL proof is done. That's why some of 400 paper dump is without lean proofs.

All this is saying is that there's a lot of stuff benchamrks aren't covering. It's not saying models can't do those things well or don't get used to do those things.

A blip for NLP? What bespoke NLP research is about anything other than LLMs these days? How many companies are handcrafting bespoke NLP models instead of sending a LLM request? Really confused here.

Open AI have the best math models for sure, but it was Claude Fable that found the Jacobian Conjecture counter example. All the frontier models are much better mathematicians than even a year ago.

If you say so

Being in the loop doesn't make LLMs anymore of a tool. How many software engineers can just take the vague description, bang something out, push to prod without talking, clarifying or discussuing with anyone in the meantime. Are they tools?