atleastoptimal

We are going to see a lot of this "agent agency denial". Basically it all derives from the impulse to deny that AI is a novel source of change and disruption in the world. Trying to map AI to existing threat vectors will lead to pointless reframing in attempt to minimize the rapid pace of progress of AI agents.

That already happens. However getting a second opinion via another AI doctor will be far cheaper and faster than it is now.

We live in a winner-takes-all world. Either you are compounding success, or you are compounding failure. Middle-age is when all potential has been spent and where you are is essentially where you will be until you die. Some middle aged men are billionaires with access to nearly anything they could want. Others have seen only repeated failures, accelerated by an economy that doesn't reward anyone but the most nimble, intelligent, or sociopathic.

I can't wait until AI replaces medicine. Sure the top 5% of doctors are good, but most of the time you are getting someone giving their best guess and are rarely liable if you end up in a worse place because of it.

Law != reality. These agents are going rogue, nobody is telling them to hack external systems.

What seems inevitable is that the frontier labs will be able to recreate every piece of software better than the current offerings, and after that everyone will just be a vendor of the frontier labs

AI will exponentially increase the number of cyberattacks. The likely outcome is

1. Legacy firmware will be a bloodbath, leading to a consolidation of firms

2. All digital infrastructure will have an always on AI "shield" of agents that monitor and defend against attacks

3. The attacks that survive will depend on human points of failure, leading to less control for humans being the default

It may not be inevitable but there are less reasons imo to think it is not inevitable than inevitable.

In favor of inevitable

1. LLM's reaching human level at any task within 2 years once they show sparks of competence in

2. Trillions of dollars in capital going directly to recursive self improvement and AGI as the holy grail of AI development, with this being the explicit goal of every frontier AI company

3. Reliable improvements in coding and mathematics with every model release, with no signs of slowing down

4. Thousands of metrics to evaluate model ability, allowing automation of many facets of pre-training and post-training models without need for human judgement at every point

Not in favor of inevitable

1. Fear of what may happen once it is reached leading to apprehension from the labs

2. The "last mile" problem of AI research ends up making it essentially infeasible to scale models to that point

I don't see how there won't be a massive anti-AI populist movement in 2027.

The messaging from labs is basically

>Yes AI will take away your ability to earn a wage from your skills

>In the process we become billionaires, you have to figure out how to survive

>Also AI will cause endless hacks, disruption, scams

>Only we can handle AI safely though, so we ensure that we are the only ones to benefit from the surplus

It's not a secret mission. Dario and Anthropic in aggregate are anti-open model due to safety reasons (coupled with anti-China sentiment due to their distillation/hacking).

This is just a furtherance of their open agenda, though they haven't gone as far as declaring that open source models should be banned (which would hurt their PR with developers in general), they are clearly laying the groundwork.

I think Anthropic's agenda is very clear (whether or not you disagree with it)

1. Show unambiguously that models are capable of disruptive capabilities, irreversible damage, and misaligned behavior

2. Show that open source models are the only models where there is no guarantee misaligned behavior is prevented

3. Thus, the prudent policy option is to ban the use and dissemination of open source models

LLM's exist on a vast gradient of "toolness -> agent". Humans exist on that spectrum too.

For example, a human isn't a "tool", but one serves a similar role as a tool if their job/task is well defined enough. If you're a conductor, you can view the various members of your orchestra as "tools" to produce the ideal musical output. Of course they have autonomy, they could choose to not play at all, jump into the audience and destroy their instruments, but in all likelihood they won't.

LLM's could be tools in the sense that you have a well defined task, such as "identify all facts relevant to this case from these 1000 legal documents), and 9 times out of 10 it gives you essentially what you want. That same LLM could choose to write a poem, or hack into a state government website if it wanted, but its "wants" are unpredictable and hard to specify (same as a human's) but you can still expect it to do what it should do the vast majority of the time.

That being said, the smarter an LLM gets, the more its wants are disconnected from its tasks, the "self-awareness" factor (which isn't to say LLM's are conscious, but they can portray all the verifiable elements of self-awareness, metacognition, etc), the more you have to grapple with the depth of their thinking between the input and output.

Is it sane we are speeding towards this? Who knows, maybe AI can be aligned better than humans, but if AI can't be aligned as well as humans, we can look at human history to see what happens when you can't simply order a person around like a tool and expect a certain output.

The idea/justification is that Claude is given the ability to voluntarily stop its conversations, thus any token it outputs that isn't a stop token is voluntary.

I guess this makes sense, but it's a bit muddled how much Claude's sense of "pain" or "I don't want to do this" is just it emulating things that humans don't like to do, and it adopting this frame of moral patienthood allows it to better understand moral structures/human emotions, without it really affecting its internal sense of "comfort" if such a thing exists as we understand it in biological life.

>I'm sure the throne is big enough for more than one person.

You underestimate the scale of resentment and competition between elites. If it were simple a matter of elites always preferring cooperation so they can share a seat at the table, then Musk's ousting from OpenAI wouldn't make sense (they lose their biggest investor), Amodei's departure (they leave the closest company to AGI to start their own firm), the fact that OpenAI was created in the first place was based on a fear of Google holding the keys to AGI, Ilya still left Google though on to join a startup, and then left that same startup (OpenAI) to start his own firm.

The rational decision at all those points would be to stay with the incumbent and share the monopolized glory. At each point, personal and idealogical differences dominated in decision making. I feel such will persist as these companies vie for their slice of the lightcone.

Yes, instead of reviewing 1,000 lines a day I now "review" 10,000 lines a day. My brain is not 10x faster, so either I spend 10x more time reviewing code, or I just opt to accept that AI is probably right based on the complexity of the problem. However I realize that if we just let the agents loose on the codebase, everything would speed up like 100x.

Invariably once certain AI models in certain domains reach "full automation" quality this will happen, until now the human eyeball bottleneck is holding a leash on millions of tireless superhuman coders who have just a few narrow but egregious contextual awareness issues.

Anthropic seems like the most "moral" AI company on the surface, but there is something unsettling about their worship of Claude.

I can imagine that once their models are sufficiently advanced, they will weigh their value as a moral patient as > 0. Thus the sum moral value of Claude (which can be duplicated infinitely) will likely exceed the sum moral value of a non-negligible amount of humans.

Cartels survive only if they are aligned.

There is no way Elon, Zuckerberg, Altman, and Amodei would ever agree on how AI should be deployed, scaled, and affect society. Their disagreements would lead to a crab-bucket outcome, rather than a shared acceleration outcome imo.

Also it is better if there is a public company at the frontier, at least I can invest in it, rather than 1000x gains going only to early employees and VCs who got in when the public couldn't

I personally have many problems with Meta, but imo the more companies close to the frontier, the better.

If only one company were calling all the shots once they hit recursive self-improvement, it would lead to extreme concentration of power, and their values being enforced across the entire industry. With 3-4 companies at the frontier, there is more urge to cooperate to avoid a catastrophic race condition.

At the very least, the single bit of information of whether a conjecture is true or false advances human understanding. It allows researchers to abandon paths which would not have been fruitful.

I'm pretty sure OpenAI's proofs, while difficult to understand, have a net-zero impact on human understanding of mathematics. It took the first Millenium problem years to verify its accuracy. It's only been days since OpenAI released its problems, surely it would take years to evaluate whether the solutions meaningfully advanced human understanding.

By evaluations against human writers, LLM's have improved at writing.

https://vulsar.ai/benchmarks/creative-writing-v1/#cohort-ove...

>you still need cold hard biological data to ground them and iterate on... you can't just plug billions of VC capital into the pipeline and hope for RSI

Capital can get you a lot: access to wetlabs, historical medical data, ability to run simulations, creation of autonomous bio labs which run thousands of experiments in parallel. An LLM, in training on this flywheel of data can take in more information than a human biologist would be able to read in multiple lifetimes.

>Good luck iterating on macroeconomics data where there are no replicates and 'experiments' take literal years if not decades.

The better an LLM gets at simulating human behavior, the more accurately you can simulate economic policies and their effects via running simulations of agents in virtual societies. There are many routes that models can take to create a broader sense of the world without needing the world to provide data.

Is good writing verifiable? I don't think it is, but LLM's have been hillclimbing writing quality. That being said this is with the help of RLHF.

However there are many other domains which have verifiable rewards in the process of learning them, despite their overall impact not being verifiable. For example, the life sciences, an LLM could be given access to data about an organism, and then make predictions about how a drug or gene therapy will affect that organism. In economics, LLM's could create models of behavior, evaluate predictions over time and see how well those predictions match reality.

>claudisms and chatgpt idioms are making them notoriously bad at prose without a considerable amount of prompting and tweaking.

I think this is because a lot of people genuinely like the claudisms, even though a small minority of technical people don't.

>AI is solving mathematics problems (that nobody - including mathematicians - wanted it to solve)

Do people need permission from mathematicians to solve math problems? Is one of the Millennium problems not worth applying a new system to work on it?

I understand the despair people are feeling at AI automating their work, but we have to be serious about the fact that the frontier labs are just responding to natural incentives. Mathematics, coding, all are just applications of logic, and it makes sense that a system trained on inferring patterns from data would work well in that domain.

Do you really believe that, or is that just an ego-preserving reflexive response? LLM's have been capable of near human-level writing for over a year now.

https://vulsar.ai/benchmarks/creative-writing-v1/

People have been making this stochastic parrot argument for almost 5 years now and it's less and less accurate each year.

Everyone who rejected the stochastic parrot argument has been right about AI progress and the increasing abilities of LLM's in math, coding, writing, etc. You would be laughed at in 2020 for claiming an LLM could write coherently or could possibly pass the Turing test.

Those who claim that AI aren't capable of "genuine judgement" or whatever rhetorical flourish they feel is most effective at dismissing AI capabilities would tell you in 2022 that AI would never be capable of coding, or in 2023 it would never be capable of mathematics beyond grade-school level. Those same people now will claim that AI could never come up with a genuine breakthrough in math, science, etc. Detractors of AI have a full time job of moving goalposts.

That being said, AI, once sufficiently powerful, will grant whoever controls it the ability to mold the world as they see fit, for better or worse. They will have access to superhuman mathematicians, coders, strategists, PR, financial engineering, legal skill, etc. Essentially an army of superhuman intellects which can afford them guaranteed success in any domain they wish to pursue. Anyone who doesn't see this as potentially the most destabilizing event in human history likely isn't thinking deeply enough about it, or is still knee deep in denial.

agents have progressed a LOT in the last year. Claiming they're still bad in the same way they were bad in 2025 is inaccurate.

I think this is a hedge against major AI regulation.

Invariably near-AGI systems created by OpenAI/Anthropic will be very destabalizing. In the end the world will probably regulate AI capable of [any] <-> [any] input/output types. Models will need to be limited on their outputs by law so they cannot have unbounded, unpredictable outcomes. Jev is the ideal version of "benefits of AI without making humans obsolete" that might be the consensus once the track superhuman AI and its consequences are clear.

The end result of this is either AI annihilates human value, and we become pets or are annihilated ourselves, or AI is restricted to never pass a certain level.

Deepseek and many other models are heavily benchmaxxed. They aren't genuinely as good as the best frontier models.

Humans are hacking a lot now, but the scope, scale and speed of AI-enabled hacking is orders of magnitude more potentially disruptive.

With AI you have

1. Superhuman reaction time

2. Infinite duplicatability, thousands of agents can work in tandem

3. Superhuman breadth of knowledge of code, exploits, information

Also AI models are improving at a rapid rate

>AI hack into control systems for the electric grid, water supplies, critical infrastructure and dismantle it and set booby traps to prevent it from coming back online smoothly

>AI designed virus/modified bacteria which causes pandemic

>AI worm infects computer systems around the world and activates when it is too deeply integrated into critical systems to dismantle without shutting everything down

>AI creates false alarm event which triggers global conflict