nsingh2

Also with Qwen 3.8 being more token hungry than Luna, using around 2.3x tokens. Which hurts for local deployment.

One line of evolution seems to be toward some form of latent-space reasoning, as in [1]. Natural language seems like a relatively low-bandwidth channel for intermediate reasoning.

[1] https://github.com/sapientinc/HRM-Text

I've never found these discussions to be all that useful, because it's hard to define what conditions are sufficient to say something is "thinking" or "reasoning". It just ends up being circular and metaphysical arguments.

That being said, current generation LLMs do have issue, it's more productive to talk about those and their impact on real tasks (long term memory, continual learning, tokenization, context rot, reversal curse, etc)

You are making two different claims, and then treating them as one.

1. Mimicking a process may not prove that the system is truly preforming the same process. In this case, a "reasoning" process.

2. A system that simulates a process cannot, by definition, be doing the process.

The first claim can be defended, but the second doesn't follow from the first.

Calling it a "simulation" is assuming the conclusion. What makes it a simulation rather than some implementation of reasoning on a different substrate? I.e. transistors rather than neurons.

Work doesn't start with a PR description though. I'm assuming most people that are using LLMs start with some sort of document (plan, spec, intent, etc) which captures intent.

I guess you could also use all the session rollouts saved to disk that were related to that task, and distill them somehow.

Their models don't consume encrypted text, it would be absurd to train them to do so. Surely they decrypt the text before feeding it into the LLM, so the contents could get leaked out by asking it.

My own experience: much less context bloat, and hidden system prompts. Can tune it to your own workflow, instead of dealing with whatever hidden system prompt Claude Code forces on you.

To be fair, Pi only had to be useful. Claude Code had to solve the difficult unsolved problem of making a terminal print text, so naturally they built a small game engine [1]

[1] https://x.com/trq212/status/2014051501786931427

I think it might affect real work if part of it requires a lot of thinking, i.e. something similar in nature to a puzzle.

There seems to be something wrong with the "commentary" channel related intermediate updates, maybe the model gets confused about what's an intermediate update vs what's the final answer? [1]

[1] https://github.com/openai/codex/issues/30364#issuecomment-48...

This is preliminary, but it seems like it might somehow be related to the `## Intermediary updates` system prompt that's provided to the model. Seems like it forces the model to stop thinking and return early to provide updates. Removing that entirely makes all runs succeed [1].

I wonder if it's somehow getting confused between what's supposed to be an intermediate update vs the final result.

[1] https://github.com/openai/codex/issues/30364#issuecomment-48...

This is preliminary, but it seems like it might somehow be related to the `## Intermediary updates` system prompt that's provided to the model. Seems like it forces the model to stop thinking and return early to provide updates. Removing that entirely makes all runs succeed [1].

I wonder if it's somehow getting confused between what's supposed to be an intermediate update vs the final result.

[1] https://github.com/openai/codex/issues/30364#issuecomment-48...

From some of the numbers I'm seeing in the GitHub issue, the codex desktop app has the same 516 spikes. So most likely it is affected.

Oh this seems bad, and is fairly easy to reproduce using codex cli. You give it a puzzle prompt that it has to reason about and solve, occasionally it will seemingly short circuit and think for exactly 516 tokens, and return the wrong result. When it ends up using 6000-8000 thinking tokens it returns the correct result.

Maybe some issue with adaptive thinking? Another point for local models I guess, don't have to worry about silent server side changes.

Edit: To follow up, it seems to happen quite often. Out of 10 runs of the exact same prompt, 4/10 had this 516 thinking token issue, and every one of these had the wrong solution. So nearly half the time, 5.5 xhigh could be short circuiting and degrading performance. Granted the sample size is small.

Codex apparently added this too, I only noticed a few days ago: https://github.com/openai/codex/issues/28969

I'm usually at my desk, and get a notification when it stops to ask a question, so I've never accidentally had it timeout yet. Still an annoying change.

A highly detailed specification is not what I mean here. It's closer to plugging in a few sentence descriptions (or a totally cluttered brain dump) and having the model interview you to help pin down critical details before continuing.

In my own work, it's usually been a few critical assumptions the model made silently (and I never even though of initially) that end up being the difference between passable results the first try, and me having to go back and fix things. Occasionally some questions force me to rethink the problem entirely.

I basically always begin any long-running session with this kind of brainstorming. I don't find the existing plan modes in Claude Code/Codex to be critical enough.

Refusing to sufficiently specify a task and hoping the model guesses correctly is not being productive. Again, these models still don't really ask questions when they should. You have to explicitly tell them to.

Specifying the problem is not extra work separate from solving it. If you skip that step, the ambiguity gets pushed into the model’s assumptions. Then you get a plausible looking answer to the wrong problem and have to waste time backing out of it.

LLMs are not magic machines that can read your mind.

Why supply underspecified requirements in the first place? Both models are good at challenging assumptions/edge cases and asking questions to clarify, but seemingly only when explicitly asked (i.e. something like a "brainstorm" skill).

I don't think either harnesses do enough to encourage the model to challenge all assumptions and ask questions, maybe because users might find it annoying. That step is basically a requirement IMO.

I've found all of the GPT-5 models to be very nit-picky, useful for code review and mathematics (important for my work), but seemingly gets in the way of "aesthetic" code, e.g. overly defensive code to cover all edge cases, even if unlikely.

There is seemingly also a tradeoff between flexibility vs instruction following. In my experience Opus will sometimes ignore instructions but can "fill in the blanks" more, vs GPT-5.5 follows instructions better but perhaps at the cost of rigidity.

Cost per task is shockingly high. More expensive than Opus 4.8, second in place to Fable.

Cost per task data is only available for max effort though, might just be very inefficient at that effort level.

That seems to only be true for the "Agentic Search" benchmark. That benchmark in particular is a bit weird, because Sonnet 4.6 effort levels had a relatively small effect, so Sonnet 5 med is basically comparable to all effort levels of Sonnet 4.6.

> Don't use it to write code

One of the things I've started doing is to write the high level scaffold myself, create stubs (with comments) and have it fill out the details. I haven't done this for long, but I feel like my mental model of the code rots less, and I can review code faster.

I used to do this in the past, when these models were not all that capable, now there is a huge temptation to leave everything to the model. But long term I lose track of whats going on, and have a hard time even communicating with the model about the changes I want. Just a total decay of my mental model of the system, and it's really hard to recover from that state.

This is false. A process needs read permission on the relevant `/dev/input/` device, typically by running as root or as a user in a group like `input`. Normal desktop users generally should not be in the input group. Regular applications receive keyboard input through the compositor/windowing system.

Yea these plots are too noisy and dense. Especially that second one, lines all over the place.

From what my own experiences are, and what's on their checkout page, $100 is 5x base usage and $200 is 20x. If $100 was 10x, then I personally would drop down. They want people to go to the highest tier.

I'm really getting sick of reading about safeguards and what I'm not allowed to do on every model release.

To follow up on this, I had it solve a nasty ODE problem that I saw in the recent Mathematica 15 release post:

    Solve the following first-order ODE for f(x):

    ((-1 - 2*x)*f(x)*tan(1 + x - exp(-61 - 2*x)*f(x)/x)
    + exp(61 + 2*x)*x*(1 - x*tan(1 + x - exp(-61 - 2*x)*f(x)/x))
    + x*tan(1 + x - exp(-61 - 2*x)*f(x)/x)*f'(x)) = 0

    Find the general solution f(x).

And surprisingly it found a valid solution! Extra impressive because it runs 25 tok/s on my measly RTX 2070 super.

    f(x) = x*exp(61 + 2*x)*(1 + x - arccos(C/x))

    C is an arbitrary constant.

Apparently Mathematica 14.3 couldn't solve this ODE.

Lots of confusion about what this model is actually focused on.

It is a cheap specialist for closed-world, verifiable reasoning tasks like math, self-contained coding problems, and similar.

"Closed-world" means the needed information is already in the context. It is not a tool-using agent that can discover missing context. "Verifiable" means answers are hard to generate but easy to check.

So no open ended research, repo wide agent work, factual Q&A, or SVG generation. More of a compact reasoning module for bounded problems.

The lack of tool use will hinder it a lot I think, since bug hunting requires collecting context across a code base and stitching it together. It might be good in a more narrow sense, i.e "is there a bug in this block of code" and not considering how it interacts with the rest of the code base.

That's also more aligned to its leetcode style training data, the code under test is fully in the context window. It might be interesting to have a bigger tool use model go through the effort of collecting the context, and feeding it into this kind of model for analysis only. It becomes more of a thinking tool, instead of the orchestrator.

This model doesn't support tool calling, was not part of its training. It's focused on Python (and I think C++) competitive programming and mathematics tasks, i.e. tasks with verifiable rewards. So if you have a task that fits that description, the size-to-capability ratio is good.

These kinds of models might be more useful as tools to be used by larger orchestrator models, than being the orchestrators themselves.

No one really goes into an interview speaking about politics, or propaganda, or unionizing. So there isn't much signal in telling people to not do that, most already bend over backwards to look like an ideal candidate.

Telling people to have a good attitude is easy to say, hard to do when you've been applying for 6+ months with no response and bills to pay.

Unfortunately, the advice that's most likely to help is something everyone already knows. Networking, practice, and luck play a huge role in a environment like this. And cut expenses as much as possible, just in case you need a cash cushion.

How does this have anything to do with a sluggish job market? Do you think people are struggling to find jobs because they are too political? One can be a servile dog, and still struggle to land a role, it's not particularly correlated.