highfrequency

Did they at least rule out an easy prompt fix? "Stick to the spirit of the problem and don't cheat (eg reverse engineering the test cases or source code)"

Why UK?

> idea thief. Takes credit for others’ work.

Based on what?

Thanks, I appreciate the discussion. The paper you sent is interesting. I agree it looks like for moderate values of K (on the order of 100-1000), RL models actually look a little worse at pass@k than their base models.

So perhaps the right framing of your/Sutton's claim is: RL can upweight low-probability (p) but correct outputs, but there is a limit to how small p can be, and it is on the order of 1 in a 100 or 1 in a 1000. Implicitly there must be some crossover point where you would call this discovery/creativity if it works for sufficiently small p right? Eg if RL can upweight a correct but 1 in a trillion output to 1 in 5, that's got to count as discovery given that all possible sequences are technically "in the distribution"?

In practice, it does seem like that kind of progress is happening. For example with the recent Erdos solution [0], I would wager that GPT 4's hit rate on this would have been functionally 0 (certainly less than 1 in a thousand). Curious to hear whether you'd still say this is mode-seeking within a base distribution, or if not then what is the right explanation if not iterative RL.

I'd also highlight that the paper you linked with the pass@k equivalence doesn't technically address the question of how small p can be before RL upweighting breaks down - all of the example problems were easy enough that the base model had decent hit rate with 128 tries.

[0] https://openai.com/index/model-disproves-discrete-geometry-c...

I wish it were ok for companies to bluntly say: “we made these decisions for competitive reasons, but the public backlash outweighed that so we are reversing course.”

I think it’s normal and morally fine for companies to want to protect their leadership position. I find the process of creating narratives that justify these decisions as something chosen for the good of others is a little tedious.

Sure, but I'd say that moving desirable trajectories from very low probability to high probability is characteristic of genuine human learning and discovery. Technically, quantum gravity, a bestselling novel, or a yet undiscovered proof of the Riemann Hypothesis is "in my distribution", but when we are talking about a long chain of unlikely token completions (with multiplicative probabilities), whether that trajectory lives in the tail of the distribution vs. in the mode makes all the difference.

Would you agree that it is a matter of degrees rather than a qualitative distinction? There seems to be a broad misconception in Sutton and others that output quality cannot exceed that of the base internet distribution; my point is that RL allows you to easily produce an output distribution that is better than whatever data you trained on according to some evaluation criteria. There are no clear theoretical limits on how much better it can get, rather there are many people asserting guesses that there is an upper bound and it lives below "human creativity." I just haven't seen any solid theoretical argument, and the empirical evidence has so far shown continual improvement.

Also, I would be keen to look at any sources you have of pass@k not improving much during GRPO.

The model’s distribution will certainly change from the base model’s output distribution during reinforcement learning, shifting toward outputs that score well on an external evaluation. This is very different from mode-seeking. Am I missing something?

> RLVR still does not expand beyond the base distribution though, it only mode-seeks within it.

Seems clearly false. Pretraining finds the mean/mode of the data distribution. RL can easily generate many samples around that mode, evaluate them on an external source of truth (eg compile the code and run it) and then selectively train on the good samples. This clearly can go beyond the initial data distribution.

Unless I'm missing something, this argument seems to apply only to the original pretraining era (eg GPT 1-4). The post-training and reinforcement learning paradigms are clearly doing variation, evaluation and selective retention no?

In particular, good to marinate beyond the (typically 6 month) lockup period during which investors cannot sell stock

Makes a lot of sense that Musk should do the parts of the AI stack that look more like manufacturing/regulatory bottlenecks, and rent out the compute to research-focused AI labs. Does anyone know the full accounting of how much it cost to build Colossus (plus ongoing opex) vs. the revenue it's generating now?

> now they’ve thrown down the gauntlet directly challenging frontier labs by training their own model (“much larger” than Kimi 2.5’s 1T parameters) from scratch.

To clarify, the model Composer 2.5 announced in this post is not that; it uses Kimi 2.5 as a strong starting point. This is not to discount Cursor's work or future ambitions, but one of the most striking things about the last 6 months is that multiple open-source models/labs are now within striking distance of the frontier closed-sourced labs.

See eg Kimi 2.6 benchmarks: https://www.kimi.com/blog/kimi-k2-6

> LLMs have got to the point where if a problem has an easy argument that for one reason or another human mathematicians have missed (that reason sometimes, but not always, being that the problem has not received all that much attention), then there is a good chance that the LLMs will spot it.

What’s wrong with the finance team (vibe) coding a janky prototype for planning?

Why do you prefer the laptop to be thicker and heavier?

I believe that's why 90% of the focus in these firms is on coding. There is a natural difficulty ramp-up that doesn't end anytime soon: you could imagine LLMs creating a line of code, a function, a file, a library, a codebase. The problem gets harder and harder and is still economically relevant very high into the difficulty ladder. Unlike basic natural language queries which saturate difficulty early.

This is also why I don't see the models getting commoditized anytime soon - the dimensionality of LLM output that is economically relevant keeps growing linearly for coding (therefore the possibility space of LLM outputs grows exponentially) which keeps the frontier nontrivial and thus not commoditized.

In contrast, there is not much demand for 100 page articles written by LLMs in response to basic conversational questions, therefore the models are basically commoditized at answering conversational questions because they have already saturated the difficulty/usefulness curve.

Agreed, but are you also implying that the process of iteratively "programming something that's not it, and then replacing it" multiple times is not in the scope of what LLMs can/will do?

Interestingly, non-coding improvements seem less clear. In the Virology uplift trial, Mythos does about as well as Opus 4.5, and Opus 4.6 is notably much worse than Opus 4.5 (p. 27).

"The window described above is not a metaphor. It is an arithmetic deadline...The window is not a metaphor for someone at your coordinates. It is a number."

9 to 1 bet that this is written by AI (perhaps proving the writer's point that he is being displaced).

I think AI is great, but I wish people would just post the prompt they gave instead of the full length decompressed essay. Much more efficient information transfer!

“Your objective is to find the weirdest niche possible that still has the potential to change everything.”

“Virality based social media is inherently homogenizing.”

Some nice nuggets in here!

> I’ll tell the LLM my main goal (which will be a very specific feature or bugfix e.g. “I want to add retries with exponential backoff to Stavrobot so that it can retry if the LLM provider is down”), and talk to it until I’m sure it understands what I want. This step takes the most time, sometimes even up to half an hour of back-and-forth until we finalize all the goals, limitations, and tradeoffs of the approach, and agree on what the end architecture should look like.

This sounds sensible, but also makes me wonder how much time is actually being saved if implementing a "very specific feature or bugfix" still takes an hour of back and forth with an LLM.

Can't help but think that this is still just an awkward intermediate phase of development with adolescent LLMs where we need to think about implementation choices at all.

Can you be more specific about which math results you are talking about? Looks like significant improvement on FrontierMath esp for the Pro model (most inference time compute).

Given how good Apple Silicon is these days, why not just buy a spec'd out Mac Studio (or a few) for $15k (512 GB RAM, 8 TB NVMe), maybe pay for S3 only to sync data across machines. No talent required to manage the gear. AWS EC2 costs for similar hardware would net out in something ridiculous like 4 months.

Principles aren’t tested until they bump into conflicting incentives.

Engineering is a continual lesson in axe-sharpening (if you have 6 hours to chop down a tree, spend the first 4 sharpening your axe).

My favorite framing, from Kent Beck: “first make the change easy, then make the easy change.”

> Of course, ai will continue to improve. But it is also likely to get more expensive…Microsoft’s share price took a beating last week as investors winced at its enormous spending on the data centres underpinning the technology. Eventually these companies will need to demonstrate a return on all that investment, which is bound to mean higher prices.

That’s not how it works, and I’m surprised this fallacy made it into the Economist.

Commodity producers don’t get to choose the price they charge by wishful thinking and aspirational margins on their sunk costs. Variable cost determines price. If all the cloud companies spend trillions on GPUs, GPU rental price (and model inference cost) will continue going down.

Indeed, cheaper and cheaper AI for the same level of performance has been even more consistent empirically than improvement in frontier model performance.

> you need to stay off the critical path. As soon as you start handling essential tickets, you’ll block your team when managerial work pulls you away.

Very well put

There’s a great quote from Nick Patterson of RenTech who says that the most sophisticated technique they generally used was linear regression, and the main thing was avoiding stupid mistakes:

“I joined a hedged fund, Renaissance Technologies, I'll make a comment about that. It's funny that I think the most important thing to do on data analysis is to do the simple things right. So, here's a kind of non-secret about what we did at renaissance: in my opinion, our most important statistical tool was simple regression with one target and one independent variable. It's the simplest statistical model you can imagine. Any reasonably smart high school student could do it. Now we have some of the smartest people around, working in our hedge fund, we have string theorists we recruited from Harvard, and they're doing simple regression. Is this stupid and pointless? Should we be hiring stupider people and paying them less? And the answer is no. And the reason is nobody tells you what the variables you should be regressing [are]. What's the target. Should you do a nonlinear transform before you regress? What's the source? Should you clean your data? Do you notice when your results are obviously rubbish? And so on. And the smarter you are the less likely you are to make a stupid mistake. And that's why I think you often need smart people who appear to be doing something technically very easy, but actually usually not so easy.”

http://www.thetalkingmachines.com/episodes/ai-safety-and-leg...

Good instinct. A lot of the day to day is debugging nitty things, reconciling small differences in results, trying not to make dumb mistakes. Almost all attempts to do very smart theoretical novel work fail, often because of extremely mundane engineering and data issues.

What were the main drivers?