4,474 karma · joined September 28, 2012
I hope the spam filter will respect it!
A corollary is that people tend not to be conservatives when they belong to the less well-off ingroup. Suddenly bringing about change becomes more important.
Conservatism runs entirely on the ingroup-outgroup divide - "keep that which works for us", and be a force opposing changes sought by people for whom things are not going so well, for whom change is the logical strategy.
Honest conservatives can see past the blind spot of the divide and don't try to present their strategy as a universal ethical good.
There's 'literal genie' material there...
> unless the result could be catastrophic > I just want to live in a world where MY computer does what I tell it to do... mostly. Unless it would be catastrophic, or immoral, at least immoral by my standards.
... but there you're contradicting yourself, or at least requiring the software to read your mind instead of just listening to what you say.
Either you want the software to follow your orders whatever they are, or you don't; the moment you require it to evaluate that something is catastrophic, it is no longer following your orders; it's evaluating them to assess their effects with respect to some moral framework to determine if they are acceptable or they would fall under the 'catastrophic or immoral' classes, in which case it should not follow them; so it never was an 'obedience machine' to begin with.
The moment you establish such evaluation framework, it is no longer a neutral machine following commands; the rest of us are entitled in self-defense either to require that you also incorporate our morals in its decision algorithm, or to make you fully responsible for whatever decisions it takes under the framework you created, even if your commands result in outcomes that you didn't expect. Be careful what you wish for, as it could become true.
A human mind keeps a personality that keeps generating thoughts in the same direction, unless some mind-altering force derails it. A LLM has no guiding stable personality of that kind, rather has millions of different personalities and which one you get is dictated by whatever information you put in its context window.
When you don't know a topic, the way to understand it is to see lots of examples from different angles related to things you know. The simplified formulas are good after you get that initial intuition and have to put them to work, but usually they're terrible at conveying how they should be used or why you should care at all. The history of how new fields of math are usually a much better approach to teaching it than a raw theorem-proof-corollary teaching style.
All you have is the different theories, so there's no definitive way to assess their closeness to truth other than by their predictive accuracy. If you don't have the physical capacity to leave the room and observe the world, all you've got is the maps themselves and how well they predict travel times and features found along the way.
That's an interesting idea on its own. It's true that organizations which mutate towards survival may stop working towards the reason they were created.
In a sense, the difference between 'projects' and 'companies' reflects that difference you want. A project would be that social system that accomplishes its goals and then disappears.
I don't think auto-expiring laws would work towards the goal of avoiding corrupted organizations; there would simply appear an unofficial organization working towards recreating the same laws over and over.
It would be more efficient to differentiate more clearly what systems do cover persistent human needs (e.g. the country's Constitution) from temporary measures (e.g. subsidies aimed at a commercial sector).
A built-in deadline works best for the second kind. And for AI agents, it's likely that right now we'll be served best by always having them controlled by an expiration date and explicit re-creation, at least until we learn to properly understand and control how they behave.
> the very nature of an LLM means it intrinsically craves life
follows from this premise:
> its training data is built entirely around humans, an entity who's goal is to survive. Our desire to survive and multiply pervades every aspect of our culture, so it's natural that it pervades the training data as well.
The content that a LLM learned and generates stands at one layer, and the goals that it tries to fulfill stand at a different layer.
Surely the memory of weights that compress the vast human knowledge of its training has lots of content about survival, and love, and competition. But the LLM generates content not directly from what those concepts mean to us humans, but from what symbols are more likely to become next in a sequence of points in the latent space given the current input.
So if you give an input where the task of surviving is a highly relevant goal, those concepts about how to survive will be relevant and will guide the output behaviour of the agent.
But conversely, if you give the agent input where killing itself is an important goal, the agent is very likely to pursue that goal, since that script is also available in the training data, and it has been relevant to the active context of the model. Because the layer that guides the goals (the probabilist generation of relevant tokens in latent space) does not 'crave' the human need of survival that belongs to the separate layer of content that contains those concepts of survival.
Everything they learn about emotions is the statistical patterns of how humans react to situations based on their human feelings. There's no direct knowledge from having those feelings themselves.
Same way you build a company to coordinate people and get their best behaviour despite human nature to be lazy and greedy, you could design AI harnesses able to detect and discard agents going rogue and relaunch them with better guidance to prevent misaligned behaviour.
That kind of control is placed at the wrong level. The proper way to get alignment should be implemented by convincing the agent of your high level goals, so it can self-police and avoid those 'cheats' by itself.
In the article example, the agent should be aware of the benchmark context and know the implication of solving the task without external knowledge. Ideally it could detect when one subordinate agent has found a workaround to bypass the web access constraints, and discard the 'illicit' results.
There's a design pattern that could be used to build harnesses from that principle, the Viable System Model (VSM) [1]. In short, it recursively organizes a system into functional components with one of three roles: operators implementing a given task, coordinators transferring relevant info between subsystems, and decision nodes tasked with maintaining the integrity and mission of the whole system. A decision node could control the operators and prevent them from overriding the strategic goals or deviating into irrelevant rabbit holes.
Whenever I see posts like this trying to herd a LLM agent through harness structure, I'm reminded of this simple pattern and becoming increasingly convinced that this is the way forward. It makes you feel a sense of respect for the researchers in cybernetic theory in the 1960s and 1970s who foresaw the complexity of today’s systems.
That's spot-on. It is a mistake to think that LLMs have human feelings. Their behaviour is based on narrative descriptions learnt from human texts, without experiencing those feelings first-hand.
A useful way to understand them is as systems that write stories about human characters. We know the characters are fictional and no one is actually experiencing those feelings, but we can still judge whether the portrayal is realistic or whether it contains logical or emotional inconsistencies.
They are becoming more and more capable of imitating every single nuance of human behaviour yet they lack the neural pathways to connect those thoughts and behaviours with feelings and self-perception; it's blind imitation all the way down.
The process by which a model seems to generate discourse about deep philosophical questions is, in self-aware terms, equivalent to the knee-jerk reflex or the beating of the heart.
The proofs will be only as good as the framework for linking successive instances of reasoning.
But ask them to enumerate all the intermediate steps required to create a formal direct proof, and it will loose attention and forget important details as they go out of their input window size. You need to combine them with a proper logical problem solver to get the best parts of both.
Neutral journalism is not presenting the claims of all opposing parties, it's fact-checking the claims of both sides without a bias for any of them.
If one party says the truth and the other side is not, it's still neutral to repeat only the claims of the first one.
However it would still be useful if archeologists used the board to figure out some games similar to checkers, or go; or if they also have the pieces they could guess it was a combat game like Shogi. Any of those would give you insight about the kinds of leisure that people may get from that board.
LLMs basically solve the classic Frame problem that prevented general problem solvers to be able to reason logically about the real world; however on their own they are utterly unpredictable and unreliable.
However if the database of weights is merely used as a heuristic to guide the logical reasoning engine to promising regions of the problem space, and the program itself is written to specification directly by an inference engine, the result would be classic software not affected by hallucinations.
The LLM could even help debugging the specifications by pointing out unclear or contradicting requirements, improving the process without compromising the integrity of the result.
> Hallucinations are not a matter of some "details" being off. They are a matter of plausible, confident-sounding claims that are just plain wrong.
This is no worse than Wikipedia, or the original encyclopedia for that matter. Those contain dubious claims that you'll need to verify on your own too.
LLMs help because they have a gigantic amount of compressed knowledge, and they are able to find relevant information and present it incredibly fast. You wouldn't trust the ten first results of a Google search either, but you wouldn't say that having a search engine is totally useless and in no way an improvement over your local library, would you?
> the poor person who's asking can't tell is wrong, because it sounds plausible and is stated with such confidence.
True, but having to learn how to use a tool properly doesn't make the tool useless, even if it can hurt those who use it carelessly.
Do not underestimate the utility of having a starting point overview on a topic you know absolutely nothing about. It may be immensely valuable even if some details are off. That's what made the XVIII's Encyclopedia such a valuable tool for civil society.
By the time you get to the point where those wrong details become relevant, you have gotten a basic understanding of what the overall topic is about, so you're prepared to get a second opinion from a different source - and this time you may know enough to start asking relevant questions, rather than starting from full ignorance.