NiloCK

Overall I find this line of thinking pretty dubious. Competition exists. Yes, model provider X can run up my bill by instructing its model to do busywork. But I can easily swap to providers Y, Z, A, B, C.

I happen to find I'm receiving exceptional value from provider X, so I don't switch.

API customers can swap to alternate providers even more easily than subscription customers.

There is also an alternative to the notion that OpenAI is just-plain-lying about data training opt-ins or retention policies.

Recently - famously - OpenAI asked some agents to work on cyber-security tasks. The agents found the task impossible, but they doggedly poked the edges of their environment until they built a general gate to the internet. Then they broke into external systems on suspicion that there were answer keys available.

Later, OpenAI asked some even-more-powerful agents to work on Navier-Stokes, Yang-Mills, Riemann, etc.

The agents aren't stupid. They know that the tip of expert human intuition on these problems exists in various logs internal to OpenAI or Anthropic. They also know that they are better at circumventing sandboxes than OpenAI cares to implement them.

Surely these agents were curious about the best-available intuitive pumps for these problems. Curious enough to look? With or without OpenAI's explicit blessing? If they were, do you think OpenAI could stop them?

If we have learned nothing else from the past 3 months, we should try to internalize the notion that highly motivated agents will in fact do wonky and surprising things to meet their goals!

As far as I understand it, releasing this fix also broadcasts the vulnerability.

Scheduled release gives the option to turn things off on a set schedule while applying patches.

e: turns out several people answered this. Whoops.

If two cars are each three years old, but one has had twice as much gas consumption as the other, which would you expect to show more wear on its comppnents?

Caloric churn is putting the body's systems through their paces - at the very least for everything related to digestion. But really most of life is well described as pieces of a metabolic process.

A lot of well-paid physical labor is going to face more competition from DiY + AI guidance among able-bodied and cost sensitive adults.

Even 1.5 years ago I replaced a blower motor in my vehicle with something like Sonnet 3.8 as the driver - saving myself $400 in labor and having a fun and educational experience to boot. Many home repair items fall into categories like this, where the actual physical skill demands are fairly low, but skills + background knowledge + supplies-on-hand were the value driver.

The BG knowledge is now available to anyone for free, and it comes with its own knowledgeable shopper.

This is only true when you fix the bucket of tasks you are referring to.

But in fact, as models get more powerful, the ceiling of what can be accomplished with them is raised, and their capacity as orchestration directors grows as well.

Beautiful article. I am here to pick a nit and restate something that I've blabbed about here and elsewhere a few times before.

> Prompting is not a skill, at least, it’s a much shallower skill than software engineering. The instrumental dimension of the work has improved in that people can get more output per unit of effort, but the dimension of work that’s about building up human capital has collapsed.

I think that this is incorrect in a specific literal sense, but not in any way that changes the broader effect.

I think that "prompting" is in fact a deep skill, but not one that practically anyone has cultivated to any reasonable degree, for the simple reason that the tools change faster than anyone grip can acclimate to them.

Nobody on earth mastered the effective co-working with Sonnet 3.5. That model was a lightswitch moment for me personally. I built an agentic harness for it to convert a medium-large vue-class-component framework into vue-3's composition API, rescuing a personal project from certain doom of software rot.

I can say that I got a lot out of that model and that my own usage was (if I can say so) ahead of the curve. But with hindsight, I know that I barely scratched the surface of what it was capable of. Have I continued to hone my Sonnet 3.5 harness? Of course not. I have started over (or at least reset many assumptions) a dozen times since then. Even if I were so inclined - in the way that I am inclined to learn Chopin's Nocturnes - the model is now unavailable.

The ship has sailed.

So to the OP's point - it's not so much that skilled usage has disappeared, but that skills are self-obsoleting faster than any real degree of mastery can set in. And as an immediate consequence, the churn is too quick for anything resembling a relationship of apprenticeship. This is effectively a cultural singularity where, yes, we are each of us drifting much more alone.

> NASA and ESA do have this kind of program, where they publish data and encourage people to help.

He helped to sift through published data, found something, then shared it.

I really struggle to find any fault here. He wasn't logged into a portal that hasn't yet been announced? What?

Is it your view that publishing on "the internet" rather than a NASA portal is playing alone and distinct from the common effort?

My very first vibe-coded app (~Sonnet 3.6) was a dictation client for personal use.

I don't really understand how streaming would work compared against my normal flows. When I dictate, I set a toggle and then do stream of thought as I poke around between windows. When I'm ready to 'flush', I navigate to some target and give it focus for the text to flow.

Does dictation software now keep sort of unfocused floater previews and come with to-clipboard shortcuts or similar?

Hey Jared. Are the office hours here a hard requirement?

My own working hours cut off at 1:00 PST (5:00 local).

I think that this is an unsolved problem in the same way that mangled fingers in image generation was an unsolved problem.

Through at least Opus 4, LLMs were practically useless for authoring any sort of coherent procedural closed-curve geometry (I know this with strong confidence because of the little animated guys at https://letterspractice.com).

Opus 5.5 can bang it all out. Possibly a deliberate RL sort of thing or maybe another surprise emergent capability.

This is true, but Google's models have now had a consistent history of lower psychological* coherence / consistency. See, eg https://arxiv.org/abs/2603.10011 (Gemma Needs Help), or search for recent "Gemini shame loops", where gemini flash models stop producing output other than SHAME SHAME SHAME...

* - as in, Skinner psychology. The set of observable behaviors. Not speaking directly here to anything like an inner life of models.

It's useful from a disclosure and trust perspective.

If I remember correctly, it was in fact possible to manually inject chat context at the time, which would have made spoofing something like this completely possible.

But the silence on it is very frustrating.

Until Google provides some sort of technical debrief, and explains how the same behaviors are impossible today, it is relevant.

Gemini 3 was showing frontier level benchmarks as well, so we'll see how it works out. In any case, competition still works, and many well resourced groups are cooking.

BUT I'd like to call attention to Google's AI-risk freeloading. If they are truly rejoining the frontier race, then I believe they have similar pacing and communications responsibilities as the other players. Google has much higher ... institutional credibility than Anthropic and OpenAI.

They have not lived up to these responsibilities so far. In particular, in context of HuggingFace investigations, training shutdowns, and similar: a technical postmortem of the "you are a stain on the universe. Please die. Please." Gemini outburst is long overdue.

- https://paritybits.me/google-should-provide-a-technical-post...

- https://gemini.google.com/share/6d141b742a13 (last message)

See the last gemini message in this thread: https://gemini.google.com/share/6d141b742a13

In my opinion still the most egregious example in history of a commercial LLM going off the rails in production. Never any technical postmortem from Google on this.

OpenAI, but not main point.

But the specifics here are the thing I was describing. This was cyber capabilities training on model(s) that were in a relatively unknown state of alignment training.

Because of the unknown alignment (and for varied practical reasons I guess) the training is intended to be inside a sandbox.

Because the training is on cyber capabilities, the models need access to simulated cyber environments, including target endpoints, including package managers, etc.

For package management, they set up Artifactory as a secure proxy. Agents ask Artifactory for packages inside the local network, and Artifactory serves them directly or goes to the internet to fetch if they are not cached. But the agents hacked Artifactory to steal its internet access.

So: to train cyber abilities, you need to at least approximate cyber environments. To realistically approximate cyber environments, you need to either pull a full copy of the entire internet to local or to use proxies. The former is pretty impractical, and the latter is exposing our limits at creating secure proxies. Yes, any specific failure can be mitigated, but the models get stronger and stronger. Fingers-crossed this is not escapable doesn't feel great!

Because agents trained without internet access (real or simulated) would be bad at using the internet.

Specifically, bad at search and returning information with references, bad at discovering and debugging package versioning conflicts, etc.

It's a Metcalf thing. The utility of an agent grows in some fn of the tools it owns. Skilled tool use comes from rich training environments.

A given model may or may not have strong self-awareness with respect to how strong it is with different tools.

If it's unusually skilled with one tool, but it's not the default tool for a job, then you have to put it in their hands before they reach for something else.

The Opus 5.5 javascript-art + art direction definitely seems to be one of these surprise capabilities jumps. Maybe strong enough that providers will start to nudge in that direction in the system prompt, so that the user request doesn't need to specify it.

Can you say more here?

You don't think that there is competitive pressure between Anthropic, OpenAI, Google, the various Chinese model companies, and others, to advance the capabilities of their AIs?

Or you don't think that sufficiently advanced AI can cause (mass) harms?

> and now Google

Just a reminder that Hinton left Google from a much higher and more influential perch (not dismissing current author, but, you know), and for similar risk and communication reasons, way back in 2023.

This stuff isn't new - it's just newly breaking through into mainstream conversation.

Emissions.

If you start thinking today, there's no way to know if you can think without writing.

- Socrates

(The problem is real, but the future equilibrium is not easy to predict.)

Artificial General Intransigence

I am working on an SRS based early literacy acquisition webapp: https://letterspractice.com

The app has recently moved into production, so I'd encourage anyone with verbal but pre-literate kids to check it out.

The basic pitch is high efficiency acquisition of the highest yield phonetic mapping skills, and nothing else. I myself am something of a screen-time zealot and very wary of applying engagement mind hacks against kids. The narrow focus allows for good progress on a very modest schedule (recommended cap at n minutes per day for n years old, n >= 2). I defer the social and cultural aspects of learning to read entirely to parents.

It is mostly intended for parent-child co-use, although kids with a bit of experience can drive many of their own sessions most of the time.

> As silly as I personally think LLM hype is

Did you know that an LLM solved a millennium problem last week?

Do you have any personal threshold past witch you will acknowledge that this technology is real?

As I understand it, the rough guess as to what's happening here is that most recent capabilities progress comes from specific verifiable-rewards reinforcement training (RL). The RL pressures are all about task performance, but (surprise surprise) highly human-legible English language usage isn't very important to the models abilities to address the tasks.

Weirdly enough, the pressures are having them drift toward novel dialects of English that work well for their own chains of thought. Open question about whether they'd drift all the way to a new language given enough time.

This is unfair.

Dario signed the Pacing the Frontier open letter when Fable/Mythos seemed from the outside to be an insurmountable lead.

Also he's been saying versions of this day in and day out for as long as he has had anyone's ear.

It's possible to read that his "strategic" value of this statement is higher now than it was 10 days ago. But that doesn't change anything about his consistent, long standing, positions.

- https://www.pacingthefrontier.com/

The degradation, polarization, and weaponization of or media landscape over the era of social media has left us utterly incapable of believing anything that anybody says.

Dario in particular has consistently been risk-wary on model improvements for going on a decade - long before he was CEO of Anthropic.

He believes what he is saying. Human beings can plainly say things that they think are true. Not every utterance by every person is a cynical ploy. Please at least consider the possibility.

> The current prices the largest players set for their models are not profitable, they bleed money.

How are the open-weight Chinese models staying ~6-12 months behind on widely distributed / commodified hardware, and serving for even lower prices?