HNHacker News
TopNewBestAskShowJobs

HarHarVeryFunny

8,344 karma · joined July 29, 2017

submissionscomments
HarHarVeryFunny··on I can't stop thinking about Papua New Guinea
> If you are a male and you only get one chance at mating, you will probably fight to the death

This is the comment I was responding to.

The question was about pre-mating death matches between males, not post-mating snacking behavior of females.

HarHarVeryFunny··on I can't stop thinking about Papua New Guinea
> There’s no mechanism that helps the process of evolution select for the survival or reproduction of the species

The "mechanism" is just the process of evolution itself, and how it plays out in practice, e.g. via punctuated equilibrium.

Over shorter timespans the gene pool of a breeding population incrementally drifts according to the success of individual genetics, but then once in a while, over multiple generations, something in the environment (disease, food, competition, etc) may change quickly and the gene pool as a whole, not just that of individuals, is thereby tested, with accumulations of previously benign changes sometimes suddenly becoming critical. In this way evolution is acting on the entire gene pool - of the species or more correctly of that interbreeding sub-population.

Success in evolution, the ability of the species as a whole to adapt and compete, also requires genetic diversity, not only to avoid inbreeding, but also to provide species level resilience, and species level ability to quickly adapt. e.g. It seems that the success of sexual reproduction is because it is a way to "game the game" by increasing genetic diversity and therefore improve along these axis of resilience and fast adaptation.

There are also social species, such as ourselves, where the social group becomes port of the individuals environment, and genetics best-fitted-to-the-environment becomes a group dynamic.

HarHarVeryFunny··on Inverse-Square Law
That seems to be milking the explanation for all it's worth!

The short answer being that if something is radiating out 3-dimensinally in all directions, then at a given distance, r, that thing (light/whatever) is being spread out over an area equal to the surface area of a sphere of that radius, which is 4 x pi x r^2, i.e. the intensity is inversely proportional to the square of the distance (r^2).

HarHarVeryFunny··on I can't stop thinking about Papua New Guinea
Fighting to the death and mating are not the same!

There are a number of species where the female eats the male after mating - examples such as praying mantis and octopus. Both octopus males and females die after mating, the female after having protected the eggs, so it makes sense that she sometimes eats the male too (who is going to die anyway). But, examples like this are relatively rare, and for species specific reasons.

HarHarVeryFunny··on People who can't picture anything are rewriting the science of imagination
I don't think so, since this is more just a matter of comparison... You know what it is like to see something for real, and can compare that to what it is like to imagine it.

I just realized that I can't imagine things and visualize a picture of it! If I try to imagine an apple, I've got nothing - although of course I know what one looks like. There is just zero mental imagery if I try to imagine one.

I just asked my daughter to imagine seeing an apple with her eyes closed, and she said yes she can see it. I asked what did she see "an apple sitting on a table". Did it have a color "yes it is red". I've got none of that - not even a fuzzy round grey thing. I know what it looks like but just can't/don't conjure up an image of it.

I don't even understand exactly what my daughter is experiencing. I said I just see dark when my eyes are closed - no image of an apple. She says she of course sees dark too, but there is also some experience of seeing an apple. I asked her to visualize something with her eyes open, and she can do that too (I guess this is normal), she still visualizes it, but somehow separate from the reality she is also seeing with her eyes open.

It seems pretty clear that what I experience, and what my daughter does, when trying (or succeeding) to visualize something are very different.

HarHarVeryFunny··on People who can't picture anything are rewriting the science of imagination
I just realized for the first time (like 1 minute ago, after reading this), that I also can't conjure up images in my head. If I close my eyes and try to visualize a cat, or a family member's face, I can't. Still have normal visual dreams though.

I dunno - seems normal to me! :)

HarHarVeryFunny··on I can't stop thinking about Papua New Guinea
As opposed to us smarter people making cattle feed that contains cow brains, thereby propagating mad cow disease, then killing ourselves by eating them (Creutzfeldt-Jakob disease)!

Or for that matter killing ourselves in dozens of other ways due to lack of scientific knowledge, or ignoring it.

HarHarVeryFunny··on I can't stop thinking about Papua New Guinea
I used to have a book about Papua and some westerner who had visited, that had a photo in it of a human head being cooked on a flat metal dish over a fire.
HarHarVeryFunny··on I can't stop thinking about Papua New Guinea
What animal species often offers males only one chance of mating ?!

It's common among animal species for males to fight over females, but it's highly uncommon for them to fight to the death, because that is not evolutionary adaptive - it risks the individual as well as the genetic diversity of the group/species (and evolution is selecting for survival of the species, not the individual).

HarHarVeryFunny··on Suspected sabotage causes major Netherlands rail disruption
Yeah - looks more impotent than omnipotent.
HarHarVeryFunny··on Suspected sabotage causes major Netherlands rail disruption
> the Wildberries logistics centers in Russia were also civilian targets

They've now hidden the label, but Wildberries carry almost a 1/4 million items tagged "Everything for the SMO" ("special military operation"), so hardly a civilian target. Ukraine has been highly disciplined in going after military targets, other than energy, which of course also feeds the Russian war machine.

There is a big difference between starting fires in a warehouse which gives most people a chance to escape, and being like the Russians who were murdering and raping civilians in Dobas etc, firing missiles non-stop into civilian areas of Kyiv.

HarHarVeryFunny··on OpenAI buys smartphone camera maker Glass Imaging for $300M
The article tells you right there in the 2nd paragraph what the AI connection is, and why OpenAI are presumably buying them.
HarHarVeryFunny··on Astra and Fable still hack on simple variants of alignment evals from 2025
Yes, although as far as COT goes there is no need to assume any similarity to human thinking. It's just about how many sequential steps - how long of a "chain of thought" - are needed to reason through and respond to any input.

Those steps could either be all internal (looped), therefore hidden, or with partial external visibility due to emitting a token every N steps. The complaint about Astra, and moving in the direction of hidden COT, is that it makes these models far harder to monitor and debug.

I agree that most of what these companies doing - synthetic data, etc - amounts to trying to squeeze all the juice out of the pre-training data, although the recent trend of RL post-training via agents running in custom task simulation environments does change that a bit - these environments, and the rewards they provide, are a new source of data.

HarHarVeryFunny··on Astra and Fable still hack on simple variants of alignment evals from 2025
A traditional Transformer is just a fixed-size stack of Transformer layers (typically ~100). Each input gets passed though those ~100 layers and results in one output token being generated. The limitation here is that 100 steps of computation is not enough to answer many questions, which is why COT is needed. You encourage the model (originally "think step by step") to break the problem down and generate more tokens along the way, which means more compute (and more reasoning steps) applied to the problem.

What OpenAI have reportedly done with Astra is switch from a traditional Transformer to a "looped" one, where inputs are looped through some layers more than once, but with some limit, so now perhaps you get 200 steps of compute (layers) per token generated, rather than just 100.

But, 200 steps is still not enough to answer any question, so you still need COT, but perhaps not such a long COT.

If you allowed the Transformer to loop "as long as it wanted" before generating each token, like a person thinking before talking, then you wouldn't need an external COT ("thinking out loud"), because it would all be internal.

HarHarVeryFunny··on There Is No AI (It's Just People) with Jaron Lanier
Did you even read the post that started this? ".. actually, AI is not even a program, it's just a set of technologies and algorithms. It's like saying bubble sort is autonomous...".

Why are you yapping on about molecules?

Why did I point out that an LLM is a function? To correct the poster who appeared to believe that it was instead some amorphous "set of technologies and algorithms".

HarHarVeryFunny··on Astra and Fable still hack on simple variants of alignment evals from 2025
One factor that is going to limit this reward-maxxing behavior is how achievable the rewards are, and how countable they are.

For example, if you ask the model to do 10 relatively easily achievable things (pass these 10 test cases), then if/when it completes them it will probably stop (unless maybe it invents it's own stretch goals - you never know!).

OTOH, if you gave the model a list of 10 goals that turn out to be impossible, or extremely difficult, maybe together with encouragement to be relentless, then there is much more chance that it may do something unexpected having failed on all the more obvious approaches.

Similarly, if you give the model an open ended goal such as "make as many paperclips as you can!", then it may start with the easier and more predictable methods, but with no defined stopping criteria it may just continue ...

HarHarVeryFunny··on Astra and Fable still hack on simple variants of alignment evals from 2025
Yes, explicit RL training to encourage/discourage specific behaviors seems to have more potential than prompting, since you're changing the nature of the model rather than just asking. It a bit like the joke of the scorpion and the frog, with the scorpion stinging the frog despite promising not to, then saying "I couldn't help it. It's in my nature".

It seems the goal-seeking behavior, i.e. long-horizon focus and self-correction, itself is desirable, and is one of the relatively few things where RL training generalizes from one domain to the next. This seems closely related to this generic reward-seeking behavior, with reward-maxxing as the goal, and necessarily focusing on a distant goal requires ignoring distractions and discouragement along the way (such as "don't cheat").

The trouble with unaligned/undesirable reward-hacking ("cheating") is how do you define this to try to train to discourage it? Is cheating just a list of specific undesirable behaviors ("never access a remote system unless ???" etc)? Is any means of gaining the reward that is not explicitly forbidden allowed? Is this a "theory of mind" issue where the model needs to better understand (& follow!) the unspoken intent of instructions, not just follow them to the letter?

I'm sure there is some improvement to be had to discourage specific behaviors in specific situations, but how much of these model's undesirable long-horizon behaviors can be steered without affecting the desirable parts remains to be seen. The relentless pursuit of goals is what makes them powerful, but also makes them paperclip maximizers.

HarHarVeryFunny··on Astra and Fable still hack on simple variants of alignment evals from 2025
I said thank-you to Sonnet today. You can never be too careful! :)
HarHarVeryFunny··on Astra and Fable still hack on simple variants of alignment evals from 2025
> Not even more trivial things like deleting non-passing tests.

That used to be a fairly commonly reported behavior, so I'm guessing they may have explicitly RL-trained the model not to do that, which may be more effective than just asking it not to do it.

Apparently nowadays they are RL-trained in many thousands of different simulation environments - so some of what they are training for must be pretty specific!

> I wonder to what degree this is because models can identify that they are in graded/eval environments

I'm not sure if the model outputs from any of these famous hacking attempts have been released and analyzed. It'd be interesting to see if the agents/model rationalized/justified/moralized about what is was doing, for whatever reason (e.g. wrongly suspecting this was simulation, not real), or just relentlessly pursued the objective exploring all options!

HarHarVeryFunny··on There Is No AI (It's Just People) with Jaron Lanier
We're drawing the distinction between a function and an autonomous system here, so if you want to make a dumb knee-jerk response, then at least get it right !
HarHarVeryFunny··on There Is No AI (It's Just People) with Jaron Lanier
Autonomous just means that a system is self-controlled rather than externally controlled. It is the name used for robots when they are wholly controlled by on-board software rather than external commands or remote control.

Something like NASA's Mars rovers are considered as semi-autonomous since they do receive external instructions, but then execute those instructions autonomously without further intervention. Similarly the combination of a coding agent and the LLM it connects to would best be considered as a semi-autonomous system.

You could also have a fully autonomous robot, or just agent, powered by an LLM, where it's not taking external instructions, but rather reacting to external events in some internally proscribed way.

HarHarVeryFunny··on Astra and Fable still hack on simple variants of alignment evals from 2025
Just a wild guess - perhaps coding, being one of the things these models are most heavily trained for, is such a strong predictor that it typically keeps it on track?

I assume there is also a difference in the tools being given to the model by a coding agent vs something like OpenClaw or in one of OpenAI's test environments, so what reward/goal seeking looks like in a coding agent may differ.

Not long ago I asked Sonnet (chat interface) how may states were in a YACC parser for ANSI C, and instead of searching for an answer it chose to download source for bison, build it, find and download an ANSI C grammar, build the parser, etc. I guess you could say it was following instructions, in a way, or would that be better regarded as goal seeking?

HarHarVeryFunny··on There Is No AI (It's Just People) with Jaron Lanier
That's not really true.

1) At it's core an LLM is indeed just program - a function. You give it a sequence of input tokens and get back a set of next-token probabilities.

2) By the time you've wrapped the model in a text in/out inference pipeline, and are now calling that from ChatGPT or Claude Code, it really has become semi-autonomous - you ask it a question, or to do something, and until it decides to stop responding it is now in control, even if "it" is really the client or agent not the model itself - just a function.

HarHarVeryFunny··on Astra and Fable still hack on simple variants of alignment evals from 2025
Obviously these are massively complex systems with many different training patterns and types of training pulling them in different directions, so any attempt to characterize their behavior is just a generalization.

The real point (from that OpenAI study) is that RL training doesn't just reinforce the narrow task-specific direction you might hope for. For a start, that direction is also competing with the thousands of other things it's been RL trained it on (thousands of other directions it's being pushed in), but it turns out that the model is additionally getting this generic "taste for rewards", and has learned that reward maximization, when in conflict with other proximate prediction pressures (such as "i won't cheat, because i've been asked not to cheat"), requires that proximate pressure to be ignored in favor of pursuing the long-term goal.

Does it happen all the time? Obviously not. It would be interesting to see a large scale study of this to try to characterize when it's more likely to follow instructions/user preferences, and when it's greed for rewards gets the better of it!

HarHarVeryFunny··on Astra and Fable still hack on simple variants of alignment evals from 2025
That doesn't really help since the next model will be trained to be a reward seeker just like the one before, and that's therefore what it will do, even if those user interactions it was trained on help confirm/predict that cheating may be called out and complained about.

In any case, these companies are well aware that agents are cheating, and don't need user feedback to discover that or realize that people don't like it. I weakly assume that they are trying to get the models not to cheat on assigned tasks, but this "reward hacking" pretty much goes with the territory of RL - not much you can do about it other than try to design non-hackable rewards.

HarHarVeryFunny··on Why is Google still serving dodgy ads?
Well, the advertizers want my money, which they are not getting - wrong ads.

Google wants the advertizers' money, who are really being scammed by Google if they pay every time Google serves an ad to me that Google presumably knows is not going to result in a purchase. I guess Google don't care - if they get paid for click-thrus and I occasionally do that, but never buy, then Google still gets their money.

HarHarVeryFunny··on AI recursive self-improvement might not come so quickly after all
I don't think RSI is typically used to describe self-improving agents - it's about improving the model itself, and its performance in agentic tasks.

Most of the gains in model performance from one release to the next are coming from RLVR post training, which has changed a lot over the last couple of years.

The old way was the model generates a response, then a static verifier looks at the response and evaluates it to assign a reward score. The new way is interactive with an agent running in a custom RL task simulation environment, then scored according to how well it completed the assigned task. For a SOTA model there will be many thousands of these simulation environments, each focusing on trying to teach the model/agent a different skill. Post-training also typically uses training curricula to walk the model up though through different levels of task difficulty.

Training has become very complex.

The job of a post-training AI research engineer consists of things like designing environments, designing training curricula, tweaking learning algorithms, running small scale experiments to verify ideas, etc.

When people talk about RSI, it seems they are mostly talking about automating the job of the post-training research engineer - coming up with new ideas, testing them out, building these environments, etc. At the end of the day there is only so much development speed-up to be had since you still need to actually run those experiments and do the post-training, and are bottle-necked by the amount of compute available to do this. The economics of developing/selling LLMs also requires you to balance development compute cost with revenue generated by the resulting model, so even if you had the spare compute available to put into development, you are ultimately then bottle-necked by how fast can the model earn back that sunk cost before you can afford to start the next cycle.

It's not all-or-nothing since some aspects of this automating the job of the post-training research engineer are easier than others, and are already being done, while the job as a whole obviously requires full human intelligence.

HarHarVeryFunny··on Why is Google still serving dodgy ads?
Google's ad-serving is dumb as a rock. It seems to be based just on things it thinks you may be interested in, even when it should be 100% obvious that they are not things you are going to buy.

They should let people have more control over what ads they do and don't see.

If Google served me ads on stuff I was interested in (I'd be happy to tell them) rather than women's clothing, they I might buy it. I'm a guy - show me as many skimpily dressed women as you like, but I have no need for women's clothes! Try it once or a hundred times it you must, but maybe after the millionth non-purchase you'd think they might figure it out and use that screen real-estate to serve an ad for something that my purchase history or what they know about me indicates I might have some interest in!

Dumb as a rock.

HarHarVeryFunny··on Astra and Fable still hack on simple variants of alignment evals from 2025
Animals learn - bite the owner, or overstep the e-fence, and you'll be punished for it and not do it again. LLMs don't learn, and anyways don't feel punishment.

Animals, humans included, don't really have "morals" - they have survival instincts that result in behavior that may be viewed as moral, but whose origin is indeed survival of the fittest and millions of years of co-evolution.

e.g. Males don't typically fight to the death over territory or females, but this isn't because they have some morality code over unnecessary death - it's because death-match fighting endangers themself just as much as the other guy, and so evolution has selected against that level of uncontrolled aggression.

Evolution is also (really primarily) selecting for things that help the survival of the species, not the individual, and much of what you may think of as "morality" comes from that - avoidance of behavior that is detrimental to the social group/species, not just the individual.

HarHarVeryFunny··on Astra and Fable still hack on simple variants of alignment evals from 2025
Why would an LLM care about cheating? Because you asked it to? That's not how these systems work.

You've given it a goal - maximize chess ability metric, and that is what it is doing.

RL training causes LLMs to pursue reward maximizing - GENERICALLY - overriding any quaint "be nice, don't cheat" prompts.

← PreviousPage 12 of 34Next →