No, no, you don't get it. See, it's just like a text search, if a text search could return text that never existed before, and that solves original math problems, and answer your emails for you, ... and ...
Start with clean llm, no external previous ideas of humans inserted into it, and let it generate some wisdom on its own and then lets talk. (btw thats how I would expect we could get closer to AGI with these statistical models, but thats just my opinion)
No, it did not.
Start with clean llm, no external previous ideas of humans inserted into it, and let it generate some wisdom on its own and then lets talk.
An LLM is about as likely to do that as you are. The ability to generate "wisdom" ab initio cannot possibly be a criterion for intelligent reasoning. The ability to arrive at novel mathematics proofs, on the other hand, is good enough for me.
Intelligence means making the most of the resources and information available, not the ability to speedrun the Big Bang. LLMs are certainly smarter than humans who dismiss them as "text searchers."
It did. In the form of a pattern that humans are too incapable of recongizing. LLMs identify and repeat the pattern. That is all.
For example, if A -> B and B -> C, logic dictates A -> C. But LLMs will be able to state that A -> C, without actually using logic, if there is sufficient statements in its training data that says A -> B and B -> C and A -> C. So now if you say P -> Q and Q -> R, it will say that P -> R, when there is no explict P -> R in the training data and NOT using logic. For you, it looks like a new discovery inferred using logic when it is not. But that is how that happens..
It is just pattern recognition masquarading as logic, x, y, or z.
You're not feeling even the faintest stirrings of cognitive dissonance when a "text search" solves math problems you couldn't solve yourself?
This "super intelligent" and "capable" thing cannot even understand that your ssh keys are private and should not be sent to randos. It can solve complex math, but does not understand basic security/privacy.
What does that say to you?
When somebody posts their private keys to Github, it's usually a human. Enough said.
(And if you had ever used Claude Code, you'd know that it nags you endlessly about key hygiene.)
Oh wait, I thought these things were super smart. I didn't expect "social engineering" to work on them.
> models nowadays are way sharper than they were even a year ago.
You are missing the point. If the thing can solve complex math problems and at the same time be so dumb as to fall for "social engineering", then that means that it is not "smartness" or "reasoning" that is helping it to solve those problems. Just some form of advanced, but yet dumb, search algorithm.
No they are not. Social engineering won't work on a human security expert who knows and understands the implications of the information they are giving away. Your analogy is pointless.
Social engineering, like prompt injection, is a context attack — easy to spot if you're ready for it, but harder in different circumstances (rushed, panicked, tired, having a bad day, etc.).
Troy Hunt (security consultant, creator of HaveIBeenPwned) and Cory Doctorow have both been successfully phished [0][1]. They're both tech- and security-savvy people who "should have known better" but it happened to them anyway. But maybe you're different... you'd never fall for an online scam, right? [2]
[0] https://www.troyhunt.com/a-sneaky-phish-just-grabbed-my-mail...
[1] https://doctorow.medium.com/https-pluralistic-net-2025-04-05...
[2] https://news.harvard.edu/gazette/story/2024/09/youd-never-fa...
Are you serious? LLMs, being a computer program, should always "be ready"?
Unless you want to also claim that LLMs can be rushed, panicked, tired or can have a "bad day"!
Jesus! The mental gymnastics people will go through to justify LLMs is just absurd!
Largely agree with you otherwise, not sure why you read my comment as mental gymnastics to justify LLMs. I don't think that they have an internal emotional state that can feel rushed, panicked, so on. They do — superficially — "respond" to such cues in language, which is why they can be "threatened" [0] and "flattered" [1]. But without an internal theory of mind, LLMs do this sycophantically without any internal model of the world (hence the quotes above, to avoid fully anthropomorphizing their behavior).
The only parallel I'm drawing is that both humans and LLMs can be coerced into unintended behavior via language ("social engineering" and "prompt jailbreaking" respectively), and those attacks are more effective if an attacker is allowed to control more "context", even if the underlying mechanism of why those attacks work is completely different.
[0] https://arxiv.org/pdf/2507.21133
[1] https://arstechnica.com/science/2025/09/these-psychological-...
I'm not just trying to be snarky, but I have no idea how to read this without taking the implication that humans are advanced, yet dumb, search algorithms.
An LLM will happily say X and behaves in contradiction to X. Because it does not reason. Its behavior is not derived from things that it claim (or appears) to know.
That is literally not true and why we talk about "stated vs revealed preference" and such.