Hacker News Reader

Comment by saagarjha | Hacker News Reader

> At that frontier, even human experts may struggle to assess results and guide further progress, particularly on non-verifiable tasks…This motivates homogeneous recursive self-improvement (RSI), in which the model being improved (optimizee) also serves as its own optimizer and evaluator.

Good thing we know how to keep models focused on goals of our choosing and not ones that they naturally evolve to have and must be steered away from by humans

View on HN

But if the machines capability is beyond the ability of humans to evaluate, how would it make sense that humans are choosing the goals? The machine is more intelligent than you and thus logically able to solve problems and evaluate your needs better that you yourself are. The fact that this sounds incredibly scary should probably stop people from building said machine, but it won't. Some people, perhaps more than you would think, want control to pass out of humanity's hands, no matter the consequences. The Nash equilibrium of the situation kind of shows one outcome,that this will be built providing it's technically possible, we just need to kind of pray the outcome is utopia rather than oblivion. Not sure the odds honestly.

No, they believe that if they're rich enough, the machines will obey them instead.

> The machine is more intelligent than you and thus logically able to solve problems and evaluate your needs better that you yourself are.

Not a logical conclusion; Einstein was famously smart, less famously married his cousin, so you'd probably not seek his advice on relationships: https://en.wikipedia.org/wiki/Elsa_Einstein

How did Einstein marrying his cousin personally harm him? It looks to me that it was a successful application of intelligence to advance his personal goals. The fact that it conflicts with other people's goals should tell you something important about AI.

The biggest problem right now is that this is better modeled as a 3-player game, not 2.

Individual success until now has been a function of personal qualities (skill, attitudes, intangibles), luck, and anti-qualities (cheating, corruption, duping). In many areas (e.g. investing) human qualities are losing their edge thanks to AI and other automation. Luck averages itself out. That might leave anti-qualities as the putative determinant of success going forward.

So at a minimum you have a 3-player game of good humans, bad humans, and AI. Maybe AI turns into Daleks. Maybe good humans end up leveraging AI to castigate bad ones. Or maybe bad humans use AI to put the good ones out of their misery. Then there are the more complex multi-stage games based on the dynamics you mention.

If you wanted a fire to not burn down your house, despite it's destructive capability and detailed mechanics being unknown to us, what sense would it make... Sometimes I suspect there's some common sense self preservation gene that's been disabled in the Valley

Reply on news.ycombinator.com