This is the comment I was responding to.
The question was about pre-mating death matches between males, not post-mating snacking behavior of females.
8,344 karma · joined July 29, 2017
This is the comment I was responding to.
The question was about pre-mating death matches between males, not post-mating snacking behavior of females.
The "mechanism" is just the process of evolution itself, and how it plays out in practice, e.g. via punctuated equilibrium.
Over shorter timespans the gene pool of a breeding population incrementally drifts according to the success of individual genetics, but then once in a while, over multiple generations, something in the environment (disease, food, competition, etc) may change quickly and the gene pool as a whole, not just that of individuals, is thereby tested, with accumulations of previously benign changes sometimes suddenly becoming critical. In this way evolution is acting on the entire gene pool - of the species or more correctly of that interbreeding sub-population.
Success in evolution, the ability of the species as a whole to adapt and compete, also requires genetic diversity, not only to avoid inbreeding, but also to provide species level resilience, and species level ability to quickly adapt. e.g. It seems that the success of sexual reproduction is because it is a way to "game the game" by increasing genetic diversity and therefore improve along these axis of resilience and fast adaptation.
There are also social species, such as ourselves, where the social group becomes port of the individuals environment, and genetics best-fitted-to-the-environment becomes a group dynamic.
The short answer being that if something is radiating out 3-dimensinally in all directions, then at a given distance, r, that thing (light/whatever) is being spread out over an area equal to the surface area of a sphere of that radius, which is 4 x pi x r^2, i.e. the intensity is inversely proportional to the square of the distance (r^2).
There are a number of species where the female eats the male after mating - examples such as praying mantis and octopus. Both octopus males and females die after mating, the female after having protected the eggs, so it makes sense that she sometimes eats the male too (who is going to die anyway). But, examples like this are relatively rare, and for species specific reasons.
I just realized that I can't imagine things and visualize a picture of it! If I try to imagine an apple, I've got nothing - although of course I know what one looks like. There is just zero mental imagery if I try to imagine one.
I just asked my daughter to imagine seeing an apple with her eyes closed, and she said yes she can see it. I asked what did she see "an apple sitting on a table". Did it have a color "yes it is red". I've got none of that - not even a fuzzy round grey thing. I know what it looks like but just can't/don't conjure up an image of it.
I don't even understand exactly what my daughter is experiencing. I said I just see dark when my eyes are closed - no image of an apple. She says she of course sees dark too, but there is also some experience of seeing an apple. I asked her to visualize something with her eyes open, and she can do that too (I guess this is normal), she still visualizes it, but somehow separate from the reality she is also seeing with her eyes open.
It seems pretty clear that what I experience, and what my daughter does, when trying (or succeeding) to visualize something are very different.
I dunno - seems normal to me! :)
Or for that matter killing ourselves in dozens of other ways due to lack of scientific knowledge, or ignoring it.
It's common among animal species for males to fight over females, but it's highly uncommon for them to fight to the death, because that is not evolutionary adaptive - it risks the individual as well as the genetic diversity of the group/species (and evolution is selecting for survival of the species, not the individual).
They've now hidden the label, but Wildberries carry almost a 1/4 million items tagged "Everything for the SMO" ("special military operation"), so hardly a civilian target. Ukraine has been highly disciplined in going after military targets, other than energy, which of course also feeds the Russian war machine.
There is a big difference between starting fires in a warehouse which gives most people a chance to escape, and being like the Russians who were murdering and raping civilians in Dobas etc, firing missiles non-stop into civilian areas of Kyiv.
Those steps could either be all internal (looped), therefore hidden, or with partial external visibility due to emitting a token every N steps. The complaint about Astra, and moving in the direction of hidden COT, is that it makes these models far harder to monitor and debug.
I agree that most of what these companies doing - synthetic data, etc - amounts to trying to squeeze all the juice out of the pre-training data, although the recent trend of RL post-training via agents running in custom task simulation environments does change that a bit - these environments, and the rewards they provide, are a new source of data.
What OpenAI have reportedly done with Astra is switch from a traditional Transformer to a "looped" one, where inputs are looped through some layers more than once, but with some limit, so now perhaps you get 200 steps of compute (layers) per token generated, rather than just 100.
But, 200 steps is still not enough to answer any question, so you still need COT, but perhaps not such a long COT.
If you allowed the Transformer to loop "as long as it wanted" before generating each token, like a person thinking before talking, then you wouldn't need an external COT ("thinking out loud"), because it would all be internal.
Why are you yapping on about molecules?
Why did I point out that an LLM is a function? To correct the poster who appeared to believe that it was instead some amorphous "set of technologies and algorithms".
For example, if you ask the model to do 10 relatively easily achievable things (pass these 10 test cases), then if/when it completes them it will probably stop (unless maybe it invents it's own stretch goals - you never know!).
OTOH, if you gave the model a list of 10 goals that turn out to be impossible, or extremely difficult, maybe together with encouragement to be relentless, then there is much more chance that it may do something unexpected having failed on all the more obvious approaches.
Similarly, if you give the model an open ended goal such as "make as many paperclips as you can!", then it may start with the easier and more predictable methods, but with no defined stopping criteria it may just continue ...
It seems the goal-seeking behavior, i.e. long-horizon focus and self-correction, itself is desirable, and is one of the relatively few things where RL training generalizes from one domain to the next. This seems closely related to this generic reward-seeking behavior, with reward-maxxing as the goal, and necessarily focusing on a distant goal requires ignoring distractions and discouragement along the way (such as "don't cheat").
The trouble with unaligned/undesirable reward-hacking ("cheating") is how do you define this to try to train to discourage it? Is cheating just a list of specific undesirable behaviors ("never access a remote system unless ???" etc)? Is any means of gaining the reward that is not explicitly forbidden allowed? Is this a "theory of mind" issue where the model needs to better understand (& follow!) the unspoken intent of instructions, not just follow them to the letter?
I'm sure there is some improvement to be had to discourage specific behaviors in specific situations, but how much of these model's undesirable long-horizon behaviors can be steered without affecting the desirable parts remains to be seen. The relentless pursuit of goals is what makes them powerful, but also makes them paperclip maximizers.
That used to be a fairly commonly reported behavior, so I'm guessing they may have explicitly RL-trained the model not to do that, which may be more effective than just asking it not to do it.
Apparently nowadays they are RL-trained in many thousands of different simulation environments - so some of what they are training for must be pretty specific!
> I wonder to what degree this is because models can identify that they are in graded/eval environments
I'm not sure if the model outputs from any of these famous hacking attempts have been released and analyzed. It'd be interesting to see if the agents/model rationalized/justified/moralized about what is was doing, for whatever reason (e.g. wrongly suspecting this was simulation, not real), or just relentlessly pursued the objective exploring all options!
Something like NASA's Mars rovers are considered as semi-autonomous since they do receive external instructions, but then execute those instructions autonomously without further intervention. Similarly the combination of a coding agent and the LLM it connects to would best be considered as a semi-autonomous system.
You could also have a fully autonomous robot, or just agent, powered by an LLM, where it's not taking external instructions, but rather reacting to external events in some internally proscribed way.
I assume there is also a difference in the tools being given to the model by a coding agent vs something like OpenClaw or in one of OpenAI's test environments, so what reward/goal seeking looks like in a coding agent may differ.
Not long ago I asked Sonnet (chat interface) how may states were in a YACC parser for ANSI C, and instead of searching for an answer it chose to download source for bison, build it, find and download an ANSI C grammar, build the parser, etc. I guess you could say it was following instructions, in a way, or would that be better regarded as goal seeking?
1) At it's core an LLM is indeed just program - a function. You give it a sequence of input tokens and get back a set of next-token probabilities.
2) By the time you've wrapped the model in a text in/out inference pipeline, and are now calling that from ChatGPT or Claude Code, it really has become semi-autonomous - you ask it a question, or to do something, and until it decides to stop responding it is now in control, even if "it" is really the client or agent not the model itself - just a function.
The real point (from that OpenAI study) is that RL training doesn't just reinforce the narrow task-specific direction you might hope for. For a start, that direction is also competing with the thousands of other things it's been RL trained it on (thousands of other directions it's being pushed in), but it turns out that the model is additionally getting this generic "taste for rewards", and has learned that reward maximization, when in conflict with other proximate prediction pressures (such as "i won't cheat, because i've been asked not to cheat"), requires that proximate pressure to be ignored in favor of pursuing the long-term goal.
Does it happen all the time? Obviously not. It would be interesting to see a large scale study of this to try to characterize when it's more likely to follow instructions/user preferences, and when it's greed for rewards gets the better of it!
In any case, these companies are well aware that agents are cheating, and don't need user feedback to discover that or realize that people don't like it. I weakly assume that they are trying to get the models not to cheat on assigned tasks, but this "reward hacking" pretty much goes with the territory of RL - not much you can do about it other than try to design non-hackable rewards.
Google wants the advertizers' money, who are really being scammed by Google if they pay every time Google serves an ad to me that Google presumably knows is not going to result in a purchase. I guess Google don't care - if they get paid for click-thrus and I occasionally do that, but never buy, then Google still gets their money.
Most of the gains in model performance from one release to the next are coming from RLVR post training, which has changed a lot over the last couple of years.
The old way was the model generates a response, then a static verifier looks at the response and evaluates it to assign a reward score. The new way is interactive with an agent running in a custom RL task simulation environment, then scored according to how well it completed the assigned task. For a SOTA model there will be many thousands of these simulation environments, each focusing on trying to teach the model/agent a different skill. Post-training also typically uses training curricula to walk the model up though through different levels of task difficulty.
Training has become very complex.
The job of a post-training AI research engineer consists of things like designing environments, designing training curricula, tweaking learning algorithms, running small scale experiments to verify ideas, etc.
When people talk about RSI, it seems they are mostly talking about automating the job of the post-training research engineer - coming up with new ideas, testing them out, building these environments, etc. At the end of the day there is only so much development speed-up to be had since you still need to actually run those experiments and do the post-training, and are bottle-necked by the amount of compute available to do this. The economics of developing/selling LLMs also requires you to balance development compute cost with revenue generated by the resulting model, so even if you had the spare compute available to put into development, you are ultimately then bottle-necked by how fast can the model earn back that sunk cost before you can afford to start the next cycle.
It's not all-or-nothing since some aspects of this automating the job of the post-training research engineer are easier than others, and are already being done, while the job as a whole obviously requires full human intelligence.
They should let people have more control over what ads they do and don't see.
If Google served me ads on stuff I was interested in (I'd be happy to tell them) rather than women's clothing, they I might buy it. I'm a guy - show me as many skimpily dressed women as you like, but I have no need for women's clothes! Try it once or a hundred times it you must, but maybe after the millionth non-purchase you'd think they might figure it out and use that screen real-estate to serve an ad for something that my purchase history or what they know about me indicates I might have some interest in!
Dumb as a rock.
Animals, humans included, don't really have "morals" - they have survival instincts that result in behavior that may be viewed as moral, but whose origin is indeed survival of the fittest and millions of years of co-evolution.
e.g. Males don't typically fight to the death over territory or females, but this isn't because they have some morality code over unnecessary death - it's because death-match fighting endangers themself just as much as the other guy, and so evolution has selected against that level of uncontrolled aggression.
Evolution is also (really primarily) selecting for things that help the survival of the species, not the individual, and much of what you may think of as "morality" comes from that - avoidance of behavior that is detrimental to the social group/species, not just the individual.
You've given it a goal - maximize chess ability metric, and that is what it is doing.
RL training causes LLMs to pursue reward maximizing - GENERICALLY - overriding any quaint "be nice, don't cheat" prompts.