nsingh2

More paternalistic nonsense

I think big reasoning LLMs are just the wrong architecture for live games like this, latency is going to be crap given that every token requires an entire forward pass through the model.

Card games or turn based strategy are much better suited.

> Read my other comment in the thread. Why would I post a non-sincere comment on this anyway?

HN is social media, saner than most, but I still don't assume sincerity.

> Yet, saying this simple stance just as I did now, that both of them are guilty, will have people call you an "ultra" or call you a conspiracy theorist, that you are a fence sitter, that you don't know what you are talking about, and so and so on.

If you mean people online, then that's expected. It's what I meant by social media being unproductive. Hostility attracts engagement and makes it more visible.

National politics is heavily influenced by economic interests, including global capital. And as wealth concentrates so does political power. That doesn't mean ordinary citizens have no political agency though, don't underestimate the difference you can make in your local community.

> not resentful or passive nihilism

To be more precise, it was passive nihilism I was criticizing. That's the sentiment I sensed in the original comment.

I don't know how sincere this comment is, but if there are others thinking this way, it's a mistake. Nihilism is a dead end (or political cynicism/fatalism, whatever you want to call it).

Social media is a veil of manipulation, engagement maximization, and bots. Do not misconstrue it as any kind of real or productive politics. There is a signal there amongst the noise, but it's hard to make sense of if you do not have a coherent world model or belief system. You have to learn to cut through it if you want to keep your sanity.

Note that paper is saying that the lean proof and the natural language proof do not necessarily coincide. It is not saying that the lean proof is wrong, just that the lean proof does not necessarily mean the natural language proof is correct.

Oh yea, completely misread that. But still, it's been my experience that TPS is lower on Sol 6.1 than Opus 5.5. To the point that 6.1 takes forever to finish a task.

They did speed it up in the last few days [1], but I don't find it to be enough.

[1] https://x.com/sama/status/2105688354834756036

> The notable difference to me is tokens/sec are still much higher on 6.1 Sol

This is also the reason why I'm swapping over to Anthropic for a month. 6.1 Sol seems good, but is unbearably slow even with 6.1 being more token efficient than 5.5.

No public repo but it's not too difficult to point a LLM at the general idea. Since Codex is open source, you can even take a look at how they do it. Here is what the model is instructed to do in codex (look at "guidance_message"): https://github.com/openai/codex/blob/d91294c39edb93d204926b3...

One approximation of this is the experimental context management Codex has been moving towards (not released yet). Rather than relying on summary compaction, the model maintains notes as it works and as it approaches the context limit. A new session is just a fresh context with those notes attached, and a pointer back to the previous session.

Not exactly like what this paper is suggesting, but similar in the sense it lets the model decide what and how to persist across turns.

I recreated this in Pi, with a max token limit on how long the note can be, to pressure the model to be concise. Ends up being cheaper than summary compaction too.

I hope more open weight models are released, with more Cyber capabilities and with no limitations.

I despise this kind of paternalism by Antropic/OpenAI.

I don't understand why they didn't call 6-Sol just 6-Terra. It was 5.6-Terra level pricing with a perf jump.

Yes but it was underwhelming, so they seem to have rushed 6.1 Sol out. Also Opus 5.5 may have spooked them too.

What's with slop-factories forgetting what a README is supposed to be? A quick intro to the project, not a gigantic abomination with irrelevant details.

I don't know how good of a marketing ploy it is, tbh. Considering it paints them as incompetent, and that they cannot be trusted with developing this technology safely.

> It's so stupid it can't follow the basic spirit of instructions

I read this as more of a misaligned intelligence, as opposed to a lack of it entirely.

Also note that Opus 5 (High) has an index value of 62, vs Fable 5 (Max) has 61. So some strangeness going on with that index.

I have some doubts about AA-index. For example Opus 5 (High) is at the same index value as Fable 5 (Max), that doesn't seem right.

One issue I see with including something like Pi is that it's intentionally bare bones. I don't think anyone uses Pi without some basic custom extensions (subagents, check lists, etc), so this benchmark may not be representative of a realistic setup.

Would be interesting to see some sort of ablation test too. E.g. what parts of Codex contribute most to the perf, and can they be recreated more minimally in something like Pi.

I would expect the benchmark scores to be nonlinear near the top, as the easier tasks get solved and the harder ones are left over. So going from 10 to 15 would be easier than going from 60 to 65.

I only take the Intelligence Index value roughly though. Considering they put Opus 5 (High) at the same level as Fable 5 (Max), I don't trust it that much.

From Artificial Analysis cost per task, it looks like Fable 5.1 (max) is more expensive per task than Fable 5 (max)? Cache hit price went down, but the other components still add up to more.

Edit: 5.1-xhigh seems to be cheaper than 5-max, and 5.1-xhigh has a higher index score than 5-max. Also interesting that Fable 5.1 (high) is comparable to Opus 5 (max), but nearly half the price.

https://artificialanalysis.ai/models#cost-tabs

All of this stuff seems like a band-aid solution. These things need to be trained ground-up to maintain and update persistent memory (maybe outside the context window?).

Also seems like a requirement for any sort of continual learning capabilities as well.

What's going on with sol here? The note says it spends a lot of time waiting, did it just not effectively use time (i.e. something like parallel runs) so it's graph ends up being stretched in the time axis?

I'm also seeing notes like on Opus 5 saying it was a run with a older serial version of program.md, so the graphs aren't complete apples-to-apples comparisons?

Edit: the blog seems to address these https://www.primeintellect.ai/blog/measuring-autonomous-rese...

Also with Qwen 3.8 being more token hungry than Luna, using around 2.3x tokens. Which hurts for local deployment.

One line of evolution seems to be toward some form of latent-space reasoning, as in [1]. Natural language seems like a relatively low-bandwidth channel for intermediate reasoning.

[1] https://github.com/sapientinc/HRM-Text

I've never found these discussions to be all that useful, because it's hard to define what conditions are sufficient to say something is "thinking" or "reasoning". It just ends up being circular and metaphysical arguments.

That being said, current generation LLMs do have issue, it's more productive to talk about those and their impact on real tasks (long term memory, continual learning, tokenization, context rot, reversal curse, etc)

You are making two different claims, and then treating them as one.

1. Mimicking a process may not prove that the system is truly preforming the same process. In this case, a "reasoning" process.

2. A system that simulates a process cannot, by definition, be doing the process.

The first claim can be defended, but the second doesn't follow from the first.

Calling it a "simulation" is assuming the conclusion. What makes it a simulation rather than some implementation of reasoning on a different substrate? I.e. transistors rather than neurons.

Work doesn't start with a PR description though. I'm assuming most people that are using LLMs start with some sort of document (plan, spec, intent, etc) which captures intent.

I guess you could also use all the session rollouts saved to disk that were related to that task, and distill them somehow.

Their models don't consume encrypted text, it would be absurd to train them to do so. Surely they decrypt the text before feeding it into the LLM, so the contents could get leaked out by asking it.

My own experience: much less context bloat, and hidden system prompts. Can tune it to your own workflow, instead of dealing with whatever hidden system prompt Claude Code forces on you.

To be fair, Pi only had to be useful. Claude Code had to solve the difficult unsolved problem of making a terminal print text, so naturally they built a small game engine [1]

[1] https://x.com/trq212/status/2014051501786931427