rsfern

For me the convenience of “hey remember this fact” is outweighed by a desire not to get stuck in a search or context bubble.

It might be nice to have better UI to control which bits of history get added to the context of a chat, but then just use a coding harness instead of web UI

It should resonate here. The thesis is that good design principles are transferable between humans and agents. So focus on solving important problems and build well designed tools and documentation to get there, you don’t need special design considerations just because agents

I don’t know about the startup space, but in AI for science people are spending time building MCP wrappers around poorly designed APIs instead of redesigning the API or building an abstraction layer that humans can also use. That seems like a mis-allocation of effort

There’s no reason it has to be like that. We could change course one more time back to a stable and non-partisan science funding landscape and then stick with it if we choose to. Even some of the alarm in this article over the proposed funding cuts isn’t set in stone, the presidential budget request from last year aimed to cut NSF by a similar amount and Congress didn’t go for it

On the contrary, I think the chess comparison is on point. We’re discussing observations that even the strongest models devolve into making invalid moves without scaffolding. For me that raises the question of whether these models are learning the rules and generalizing from them, or of they’re just pattern matching and flailing on this task. Maybe the reality is somewhere in between, but the benchmarks don’t seem to directly measure conceptual generalization, they measure task completion. They can disrupt a lot of people and industries by pattern matching and flailing without being AGI.

I’m sure these models know the rules and can explain them when prompted, but that doesn’t seem to be the way they actually complete this task. Will they get there? Maybe

Maybe it depends on the person? My six year old isn’t great at strategy but they can pretty consistently make valid moves. Sometimes they ask for confirmation on a move which is also not a trait I see in language models (at least unprompted)

the discussion isn’t really about whether language models can become strong chess players though, the point is they seem to struggle to consistently make valid moves. Most humans don’t need to read two books to pick that up, just a couple lines of basic instructions

For numerical code I like einops.reduce more than numpy/pytorch sum reductions because you can reduce over named dimensions. It’s much more readable than having to reason through axis indexing again every time you come back to the code

I agree (and so does Buckmaster based on his written statement) that we are better having solved this.

But I disagree that which humans were credited is the heart of the issue in this particular controversy. The question is what do you need to bring to the table for a result like this. A pre-release frontier model trained on the open literature and $15 million of inference? Or all that plus a year of the experts finding the path to the solution for the model to run with?

I think it makes a huge difference in terms of what we think the future of mathematical research will be like, and whether we should still encourage students to go into this field, which was the original topic of this thread

That’s the prevailing narrative, but I think this controversy calls it into question to some extent. If the OpenAI result wouldn’t have been possible without experts seeding the training data with feedback on promising solution routes, there’s less reason to believe this, IMO. More information and transparency is needed

I think you’re missing an important distinction. “Major damage” to the talent pipeline because models become capable of original end-to-end mathematics is what the community has been discussing. But if the models rely on sniping nearly complete work then this damage is antisocial without a lot of upside, it would be destroying a talent pipeline that would still necessary for continued progress.

Which is it? I don’t think OpenAI is being transparent enough for us to really understand whether these results would have been possible without relying on unpublished information from the solution strategies of the experts

The session data could be cryptographically signed. Probably easier in an open harness?

The screen cap of their full correspondence in the Twitter thread, which was the subject of the preceding sentence. The Twitter post has an obviously incomplete fragment of the conversation that doesn’t resolve what the author presents it as resolving, which is the dispute over how the discussion of authorship of Alpöge actually went down

Without seeing the full correspondence it’s hard to evaluate for sure, but parent linked to a tweet from the OpenAI employee at the center of the controversy, that’s a primary source you can read and evaluate yourself

Personally I don’t find the tweet a satisfactory explanation of their behavior, it seems like a lot of deflection without directly responding to the specific claims of front-running, and the screen cap of the correspondence doesn’t include all the relevant context. If there was really no bad behavior, why not post the whole thing?

It does, yes. So designing objections functions and making sure you can afford the training rollouts becomes really important in defining which problems are tractable. It will be really interesting to see how that shapes the kinds of problems people choose to work on

Agreed, but i think this underscores my point. We have numerical simulations in materials science too, but that doesn’t mean formally verified theorems about the underlying equations automatically translate to formal (or even informal) verification of simulation results. That’s not to say you can’t make progress with agents, but I think it’s less well defined how you write the goal and progress assessment for an agent

Yes, definitely! There’s a long history of this and I think there’s tons of opportunities for more. Both for improving the exactness/physical fidelity of models and for developing new approximate theories and simulation methods

Regardless of what you think of the priority dispute issue discussed on sibling threads, I’m highly skeptical of the closing quote that this Navier Stokes result means that the same approach of casually spending a few million on agentic computation is going to solve end to end materials design or drug development.

Those problems can’t be formally verified with an automated theorem prover. We have a lot of physics based simulation tools, but they tend to focus on small subsets of the full design problem and they make limiting approximations because otherwise they’d be too computationally expensive, or we just don’t have the right data to parameterize them beyond describing qualitative behavior. Agents are helping accelerate research in these fields but I think it’s mostly a different class of problem that’s a lot harder to specify and verify

Thanks! This seems really cool. If I’ve got it right, your UI builds and displays these diffs (and implements undo/redo) by parsing the edit tool calls?

If so that seems really nice, one of the things I don’t like with coding agents is having to defensively commit changes to roll back if the agent goes off the rails. I’m not sure if that’s a problem with my workflow, but your tool seems great for exploratory stuff

How does it change the way you personally use pi? How git aware is it, do you have to commit at the end of a branching session, and does it handle manual edits?

Interesting project idea! The link seems to be 404, is the repo still private?

Why would mining chat transcripts for ideas be untenable? They already run a summarization model to auto-title the chat, and to run a bunch of safety filters, and presumably to score transcript quality for A/B testing and to collect more finetuning data. Seems like evaluating for open research questions and approaches would be pretty trivial extension of this, after all it’s kind of their core business model

The bit you quoted doesn’t capture why the reporters are upset:

> White House journalists are outraged that a threat credible enough to force Donald Trump to escape from Air Force One using an airport catering truck wasn’t relayed to them.

Right, I did specifically say that most of the DOE scientists are contractors, but I concede the phrase “government scientist” is a bit ambiguous. I appreciate the extra detail you added. I think the distinction between political appointee and scientist/researcher stands.

As an added complication, some of the DOE labs do have civil servant scientists, for example National Energy Technology Lab and National Renewable Energy Lab are like 50/50 civil servants and contractors. And most of the funding arm of DOE are career civil servants. LANL, Sandia, Livermore, Argonne are all staffed by contractors

Let’s distinguish a bit. There are political appointees (Trump’s government employees as you say) who are mostly upper management, and there are career civil servants (all the government scientists are under this category) who have a strong culture of apolitical dedication to the mission of their agency and to the American people and Constitution, regardless of who the current president is. And in the DOE labs in particular most (not all) of the scientists are actually employed as government contractors, but they have a similar non-partisan ethos.

That doesn’t necessarily mean there’s no need to be concerned with potential impact of policy and priority changes from the administration, but it does temper the threat model because the government employees you’re considering trusting have given oaths of office to protect and defend the Constitution.

I found this paper really thought provoking, but I think the conclusion of “world models are the solution” leaves something to be desired. People are already equipping agentic systems with physical simulation tools and exploring action-conditioned world models. This is cool because you can change the rules of the simulation and observe what happens, but it doesn’t address the core question of what to change the rules to, or even what the goal should be in the first place.

Or they could store the reading traces and validate the user hasn’t edited them server-side? They could sign reasoning traces so they can’t be counterfeited?

The cell DAG enforces that there’s no implicit state, which reduces cognitive load for me a lot and provides some pressure to abstract experimental code into functions. In Jupyter this is left to user discipline and restart-and-run-all workflow

The reactive components are also really nice for interactive plotting and exploratory data analysis. You can do this in Jupyter but it feels less seamless somehow. Interactive marimo workflow feels like streamlit but in a notebook interface

One thing I miss from Pluto.jl workflow is `let` for lowering friction for exploratory or plot cells. In marimo you have to name a `_` prefixed function and then call it which is better than nothing but not as clean as `let`. This is a minor complaint that’s more down to language features though

My point with the force field example wasn’t to argue against neural scaling as a valid strategy, it totally is effective and a lot of groups are doing it. But I feel like we might be talking past each other a bit.

What I’m pushing back on is what I think is a sort of one-dimensional view of Sutton’s bitter lesson. People seem to equate it with model scaling, but there are lots of general ways to leverage computation that don’t involve just scaling models and supervised training datasets up. For example Sutton’s first example is straight up search, no parameters at all.

The point of the force field example is that it seems you don’t need billions of parameters to represent the functions we’re interested in, but with small models it’s harder to find those functions by pushing harder on the standard training algorithms, and that maybe some different algorithm that leverages computation more effectively could do so.

I don’t think there’s a fundamental reason that performance has to be monotonic in model size or even training FLOPs. At least I don’t think it’s been proved to be so, so I think “misinformed” is a bit premature and sort of makes GP’s point.

There’s evidence that model size and representational capacity are not exactly the same, and that scale is maybe more important for learning than it is for representation (past a point). Consider the early work from the current neural scaling paradigm. The Chinchilla scaling study shows that smaller models can match the performance of larger models by training longer.

To GP’s point, if everyone is exploiting the scaling lever, few resources are being allocated to finding more efficient training algorithms that could let us work with right-sized models instead of pulling the scaling lever as hard as we can afford to.

I’ll end with a dramatic example from my field of materials science (which admittedly might not strictly generalize to LLMs). A lot of the field is pursuing the model scaling strategy, and it’s still paying off. But [0] recently reported competitive accuracy with much smaller models that run faster and can address much larger problems. The model architecture is pretty much the same, but they use a different training strategy and really focus on data quality

0: https://arxiv.org/abs/2504.21286

I don’t mean to pick on you in particular here, but this approach has been bothering me a lot lately, and it seems like it’s super common.

I get that it’s an early prototype and not all the design choices are made yet, but I struggle with “I can’t afford human-readable documentation yet”. Isn’t human readable documentation important for efficiently planning and deciding what you want to build? I feel like I can’t afford not to have human readable docs and plans while the project is taking shape

Related, if a user can prompt an agent to translate to a human-readable summary, wouldn’t it be better to just do this in place? Sure, models can deal with noisy LLM outputs but shouldn’t a document that’s easier for humans also be easier for bots?

I often start prompts with “please”, but I usually don’t thank the model. Framing a question or a request for help with “please” is in distribution for me, it’s a distraction from composing a thoughtful prompt about my actual question to go back and edit out politeness.

I don’t reply “thanks” like I would to a person though, I just close the chat if I have no more follow-ups