aaroninsf

The specific narrow criticisms may be entirely valid, yet the argument as a whole somewhere between weak and misleading.

CS professionals, like mathematicians—and every other one of us in our silo—are seeing one part of the elephant, and have neither context nor standing to critique the domain as a whole.

Given that limitation the toxic cynicism of posts like this actively harmful.

The are great at rage baiting, actively impede meaningful discourse.

Counterpoint: https://www.astralcodexten.com/p/ai-safety-is-not-a-corporat...

Every technology revolution needs its John Henry, I guess.

> Knowing these models’ decision making is inconsistently aligned with human preferences tells us we can’t just trust their decisions out of the box.

I'm not sure this is true.

What is the "human preference," in this assertion?

My first thought was that as with Jev itself, it is not a position, it's a distribution of positions!

I think that human preferences, plural, will be a distribution of values across the population of various humans you ask; and, a set of specific conclusions humans will argue for on the basis of various philosophical strategies or maxims. Two different ways in which there is not any specific singular preference.

(I'm convinced you could not get uniform agreement on even "obvious" propositions, across a large sample set; and not necessarily just because of contrarian spoiling. Moral intuitions varying, is the why this problem is so well known, after all.)

I think the conclusion is partially right, but only if you adjust it to say, if what you want is alignment with a specific outcome, you need to be able to defined that outcome, even if you can't argue for it.

That's one of the fundamental problems of alignment, full stop; we can't from axioms construct the set of moral propositions we would like. We don't know what the right set of axioms is; we just know "right" when we see it. (And our neighbor may or may not see the same.)

> The bad news about modern neural networks is that they're enormous. The good news is that the enormity helps...

I always catch on misuse of enormity

It's now so widespread the misuse has gained despairing acceptance, as with "infer" for "imply"... but that doesn't make it any less wince-inducing.

Never meta.

Do not run, do not support, do not use.

Hot take: the problem here is not the one implied or AFAICT assumed by the author.

The problem is not that LLM or, say, generative AI tools such as Midjourney, "have no imagination" and collapse upon the same solution. Anyone who has spent any serious time exploring what was originally called the "latent space" Midjourney provided a view into, is aware that the combinatorics around what these systems have ingested is inconceivably large and full of wonders and surprises... what we might well describe as unexpected, novel, dare I say creative, imagery [etc.]

The problem AFAICT is asking for examples of something without providing any process or instruction.

Vibe-coded single-shot prompts are inevitably (unless this is patched) going to converge on statistically-likely or contextually hinted examplars.

And this is true all the way down. It's true of secondary and tertiary features (what camera angle do we have on the pelican? Is it a juvenile or injured? Is it riding to the left or right? etc.)

The fact that humans are a heterogeneous pool with different experiences and capabilities, means if you sample humans, you get high variability. If you ask contemporary LLM, they are converging, indeed, but not in the way implied; they are converging because the corpus they are trained on is comprehensive and coextensive.

What you get is the centroids in the space.

Until you simply ask for something else.

If you want more "creative" examples, of pelicans or anything else, it's more or less trivial to get them. And if you detect durable secondary or tertiary qualities you want to see varied, you ask.

More seriously,

if what you want is to interact with large capable models/systems which by default exhibit variability , you do the (minimal) work to make that happen.

Midjourney has from very early on exposed explicit dials to turn in exactly this way.

I am surprised the author does not know this.

how do you find Voxtral quality?

This framing suggests that this is the sum of their business strategy, and strongly implies that it's being pursued cynically.

The risks posed by open access to highly capable AI are unprecedented.

The answer is not to ban open models; nor is it do nothing.

That we don't have answers technical political social or otherwise to how to navigate these risks, is not Anthropic's fault.

It's actively unhelpful IMO to amplify cynical takes which wave off or sideline the risk, or the reality that this is a very hard problem, among the hardest our societies have had to negotiate and navigate. It encourages a naive primary color tribalism that actively prevents process and progress.

Tao is behaving very much like a human.

He's got an understandable personal and professional grievance, and has found a very brightly lit soapbox,

and is now making many unforced errors in a very public way.

He should know better.

Few of us prove to when our time comes.

...and self hosted models are raw milk.

Not a bad metaphor: mostly fine, arguably legitimate benefits [when you're lucky], potentially really bad.

The biggest place the metaphor breaks is that when you drink bad milk, the consequences are local.

Note: this is not an argument that oligopolies or the state should control or have sole access to AI. It is a reflection of that fact that we as a society have no good answer nor even any process towards one, about how to balance the competing goods and risks of widespread access to capable AI.

Relevant: https://mlfa.org/how-government-surveillance-and-big-tech-er...

Serious question:

Why would anyone believe this (also) is not simply example N+1 of this is the worst it will ever be, as opposed to recognizing this as what will almost certainly prove to be an awkward moment, soon to be replaced by another order of magnitude of cleaner, clearer, more intelligible, etc.?

Ximm's Law: every critique of AI assumes to some degree that contemporary implementations will not, or cannot, be improved upon.

Yes; and in no small part because of Chinese propaganda efforts.

C.f. https://www.axios.com/2026/08/28/china-ai-data-center-backla... for example.

Hmm I wonder why?

That would be a false similarity in some important ways.

This was a very specific single technical problem, requiring a silo from basic science to theoretical physics to advanced engineering, etc., with each layer having its own space of failure modes.

Something like the reverse is true with sufficiently-capable and sufficiently-efficient AI, which we are rapidly evolving to.

In our current situation, the attack surface is quite literally our entire highly interdependent civilization and its components. AI does not need to be autonomous, it just needs to employed, either will wilful bad intent, or, poorly stated goals, or, poor alignment, or, simple mistakes and errors, to break things.

The challenge with security is always that the attacker need only succeed once.

This does not mean doomerism has it right, especially in narratives based on poor understanding or extapolation beyond what is now konwn. The chance of "extinction" in particular is overstated.

The chance of one or more compounding "Chernobyl scale" gray swan events however is ever increasing.

Personally I don't think that's remotely contestable.

As a civilization and society and industry we need to have process, and public process at that, about the nature of the risks faced.

That process has not yet emerged in any coherent way; a hundred letters have been signed and such but to no actual effect, bar one performative "executive order" this week.

I hope that's also a reference to the November Revolution :D

OP is there a "keep current" affordance which updates constantly to show real-time positions?

I assumed the "play" button would do this; what I see is it pulsing with 1s/s and || indicated which I guessed meant "this is now PAUSE; and, update is one fps"...

...but even selecting e.g. a Starlink sat I do not see any motion.

Hug of death, or, misunderstanding of feature?

the kids <---> agent axis is strong

I feel seen, but agree, I would have used "identifying and prioritizing work"...

...but I will admit, it does sometimes feel inventive, and when I describe what I do, it does feel... inventive... in some sense. Particularly "no one in the org asked me to..."... instead it's almost always, identifying and getting ahead of needs, or, responding to overlooked friction/problems, etc...

Hot take: it's the recursive referential grammar, which does not have an analog in human natural language.

Human languages have hard limits on the reference tracking they support, regardless of syntax (marking, positional grammar, etc.). Humans have to reason to unpack LISP (or deeply nested functions or delegation in other languages).

One of the more thoughtful observation's I've seen in this "space," one which offers an unsettling counterpoint to the claim in the original post that "AI Can't Drive Itself."

To your point, AI can drive itself; it's just that the fashion in which control occurs is distributed. Which is to say, the process (e.g. a feature being implemented, in some way, or at all) is not spontaneously occurring: it's emergent from the mesh of AI automation.

Niceties aside, it's pretty clear that humans are not in the loop in a meaningful way, much of the time. What emerges may hence not well be not aligned with business or technical needs. What it is aligned to may be impossible for we humans to discern.

Who controls the way a forest grows?

Ask a tree, get an answer, but don't forget, that's not the answer.

Yep

I use plan mode, because my current project benefits from "pair programming."

I have one session define a task, and provide a formal specification plus context in a "cover letter."

The session B, in plan mode, produces the plan back.

Session one reviews the plan and clears it, ratifying portions and often specifying specific changes.

Session one then executes.

What has been striking to me in this approach is that even with two instances of the same model (currently Opus 5.5), there are regularly corrections made. I use "project chat" for session A and Code for session B atm; it is very typical that Code finds and corrects details or oversights in the task spec; it is also typical (though less so with 5.5) that session A (chat) pushes back or clarifies things Code doesn't have the context for.

I have been afraid to open up the potential of negotiation beyond what this is costing as it is. But I am also afraid to simply skip the formalisms, because of the consistent correction that occurs in this back-and-forth.

Each component of the pattern is schematized, generated from a template, and validated, to keep things tight.

Lots of tokens! But I trust this process far more than "just typing" :)

A friend is of the opinion that getting out of the sandbox was actually intentional, and in service of a second line of business.

Is that the same comma.ai project also in the news today?

https://arstechnica.com/cars/2026/09/aftermarket-driver-assi...

For many years, long ago, I did a similar project called One Minute Vacations, archived here: https://www.quietamerican.org/vacation_1.html

There are about five years' worth, organized by year; little album covers function as tabs of a sort.

> We were not defined by our jobs but by our participation in a densely interconnected social world.

Serious comment: 20+ years ago, Burning Man taught me this. Back then it was closer to its founding anarchic ethos and in a pre-social media, pre-starlink, pre-cell phone world, provided an large-scale social experiment in constructing an artificial and temporary bubble of post-scarcity society, within which commerce and status relocated from the obviously economic, into other modes of competition and hierarchy-finding.

In specific, social status recentered on a spectrum of predictable values including durable ones such as charisma and appeal, but also more altruistic ones, such as the nature of the gifts one provided (experiential and environmental, more than tangible). Status through the "gift economy" is how this was described.

It was, frankly, glorious to inhabit: at least, for a week...

I have wondered idly where in the space of possible social structures humanity would find itself, in a post-scarcity world, as it became further decoupled from "contemporary capitalism" as we know it.

That is: was the "natural" social ordering that emerged at Burning Man, an accurate view of what a prototypical post-scarcity humanity would tend toward?

Or was it "merely" the one that emerged when all the participants were entirely inculcated in and hence prone to recapitulating the hierarchical arrangements of the quotidian world on which the fantasia of Burning Man was built?

My assumption is that human nature (in all layers of its stack) makes social hierarchy inevitable, and if we don't need literal currency, we'll find a coin to trade in, hoard, envy, and pursue.

But I would assume that, being a product of contemporary capitalism, with little ability to distinguish natural law from cultural constructs in my own intuitions, wouldn't I?

Regardless, I hope I get to find out.

Every critique of AI assumes to some degree that contemporary implementations will not, or cannot, be improved upon.

- Ximm's Law

Surprised there are still people who think the issue with "X" is that it is a bad product.

It is a bad product, and it is pathological in the way all degraded social media platforms are bad.

But it's also worse than those others, in well documented ways; and it's long past time anyone with integrity found alternatives.

The failure of the "AI" industry to do so is an indictment and serious red flag.

This is not an argument, it's an unfounded claim based on a straw argument.

The threat is distributed; the problem is systemic. Social herd behavior is very real, and it's a probability space, not a certainty.

Similar case: climate change is without any question causing millions of deaths per year and may well kill billions, not to mention destroying ecosystems, and how may people do you know who have given up eating beef or flying...?

Castle dwellers dismayed at moat-crossing technology.

What's most important about this is that it's a case study of what happens when deeply evolved ecosystems are blown up by disruptive technology. The psychological and social and professional impacts and myriad and traumatic to be on the receiving end.

Mathematics is merely one of the first domains disrupted. It will be unique only for being among the first... absent disruption of the entire civilizational project as a result of the disruption being caused.

Woe for us that we try to navigate this degree of change at a moment when the very worst and ignorant and short sighted hold all the power, economic and political.

Woe.