MadxX79

I get what you're saying, but I remember watching teletubbies back in the days with my nephew, and all questions of the form:

Have ____ surpassed teletubbies?

Can always be answered in the affirmative.

I love the total lack of humility on that site. "What if the METR study turns out not to capture anything relevant? We just add a constant gap to be conservative!". But I guess these guys aren't really scientist, so it's probably a lot to ask that they relate critically to what they are doing and be honest about the limitations of their methods.

What if it turns out that the more you scale the more your LLM resembles a lobotomized human. It looks like it goes really well in the beginning, but you are just never going to get to Einstein. How does that affect everything?

What if it turned out that those AI companies were maybe having a whole bunch of humans solving the problems that are currently just below the 50% reliability threshold they set, and do fine tuning with those solutions. That will make their models perform better on the benchmark, but it's just training for the test... will the constant gap be a good approximation then?

It really dispelled the illusion for me, but it's not that easy to find those examples, but the combinatorics of possible number of guesses is untractable enough that it can't learn a good set of clues for all possible guesses.

Isn't it just that he left way before gpt-5, then? At that point a sufficiently naive person could have believed that scaling was going to lead to AGI, but that sort of optimism died after he was already an outsider.

I'm guessing they have a lot of shares in the AI companies they work(ed) for, and they would like to pump their value so they can buy an even nicer carribean island than they can already afford?

I enjoyed playing mastermind with LLMs where they pick the code and I have to guess it.

It's not aware that it doesn't know what the code is (it isn't in the context because it's supposed to be secret), but it just keeps giving clues. Initially it works, because most clues are possible in the beginning, but very quickly it starts to give inconsistent clues and eventually has to give up.

At no point does it "realise" that it doesn't even know what the secret code is itself. It makes it very clear that the AI isn't playing mastermind with you, it's trying to predict what a mastermind player in it's training set would say, and that doesn't include "wait a second, I'm an AI, I don't know the secret code because I didn't really pick one!" so it just merilly goes on predicting tokens, without any sort of awareness what it's saying or what it is.

It works if you allow it to output the code so it's in context, but probably just because there is enough data in the training set to match two 4 letter strings and know how many of them matches (there's not that many possibilities).

The guy founded two AI companies and exited one, is a researcher who is an expert on how actual human babies learns things and who has done research comparing how LLMs learn and represent language compared to humans. How much more expertise does he need to have an opinion?

I rushed so much that I didn't have time to do it right. It could have been the AST tree of my PIN number validation algorithm for ATM machines. :-P

Can it store my PIN numbers and my map of ATM machines also?

What's your argument here? He's not allowed to discuss crony capitalism because you imagine that he thinks LLMs suddenly became reliable.

Brook's law anno 2026:

"Adding manpower to a late software project makes it later -- unless that manpower is AI, then you're golden!"

From your link:

"We excluded reasoning models from our analysis of per-token prices. Reasoning models tend to generate a much larger number of tokens than other models, making these models cost more in total to evaluate on a benchmark. This makes it misleading to compare reasoning models to other models on price per token, at a given performance level."

It's just price per token. Token usage is exploding.

source?

Is that the cost per token or the actual cost of the user having a conversation, reasoning and all?

If you don't give me a discount on my salesforce subscription I'll shoot myself in the face with this AI enabled gun?

Absolutely, the belief in scientific circles is that the way forward to develop cures (or at least treatment that slows down the progression) is to treat it early. When you get to the point where you start showing clear symptoms, your brain is already mush. If you have a potential treatment that attacks the root cause, you would have to catch the very early, pre-clinical, stages of the disease, but without good diagnostics there is no way to do that (short of giving the disease to a wide swath of the population, like a vaccine... but that gets expensive very quickly, and side effects become a bigger worry.

There's really two cases here.

You live in a democracy?

YES) the violation you describe is verifiable to a journalist. You publish story, and you keep the government accountable.

NO) Why are you even discussing if age verification is a good idea or not, you freak. It's not really up to you anyway. Go fix your country first.

If I worked in a European intelligence agency, and considering how the the official US security policy revolves around bringing about regime change in Europe in support of far right extremist parties, and how supportive the tech company leadership seems to be of those goals, I would probably think that locking that very real existential threat to their democracies out would be a worthwhile tradeoff.

What I really want is to be able to search through the training dataset to see the n closest hits (cosine distance or something). I think the illusion would very quickly be dispelled that way.

Yeah, the interpolation works if there are enough close examples around it. Problem is that the dimensionality of the space you are trying to interpolate in is so incomprehensibly big that even training on all of the internet, you are always going to have stuff that just doesn't have samples close by.

I don't understand peoples problem with this! Now everyone is going to discuss this on the internet, it will be scraped by the AI company web crawlers, and the replies goes into training the next model... and it will never make this _particular_ problem again, solving the problem ONCE AND FOR ALL!

"but..." you say?

ONCE AND FOR ALL!

Isn't a mathematician arguing that Galileo was a bad scientist because he wasn't as good at deriving the area under some function as Archimedes, a bit like a fish arguing that Galileo was a bad scientist because he couldn't swim as fast as a tuna?