martincsweiss

They aren't on par - but people are irrational and the world has a lot of detail. Therefore, it's hard to do credit assignment on why Tesla is making fewer hit products.

Either that or the pics of him shirtless in Mykonos.

There's too many confounders to say that JB Straubel is the cause.

Examples: Covid controversies around musk in 2020, Twitter acquisition in 2022, Musk becoming the world richest person in 2021

That's a lot of money.

Found a dashboard of ICLR 2027 review scores (built on OpenReview data). The scoring is 1 = strong reject, 4 = strong accept.

- 40,458 papers with ratings - 125,048 reviews - Mean paper score: 2.06 (super low!) - Median paper score: 2.00 - Top 25% cutoff: 2.33 - Top 10% cutoff: 2.67

For what it's worth, I had a paper with 2, 2, 4 and it's in the top 10%, which is wild.

Really? I'm surprised. What do you use it for?

My read on Diffusion LLMs has been that it's cool tech, but only useful for things like like code autocomplete or editing a block of text.

They don't beat autoregressive models on quality per compute, assuming you are serving lots of users and can batch requests.

I use Codex and Claude Code with piles of project-specific skills and AGENTS.md files.

For my main work at my startup, we're very careful with how these are setup.

Everywhere else, I simmer in chaos.

I also have strong boundaries around generated prose. I don't like making other people read AI slop, but I don't mind having AIs read it.

That's useful feedback. I added introductory material in response to feedback from a colleague, but I suppose that didn't help "get to the point". I think the audience for my work is experts + experience practitioners, so the accessibility / novelty tradeoff was not a good one.

Yep, Stochastic Rounding [1]. I actually had "round-to-nearest" in the title of the piece originally. Usually people do that if they don't have the space for fp32 master weights, but it doesn't fix the second moment part of the learning process.

[1] What is New in Stochastic Rounding: a Survey on Theory, Hardware, and Applications, https://arxiv.org/pdf/2603.06060v2

Thanks for the comment - you're right, of course.

I read a bit more into the history of fp32 master copies, like in Mixed Precision Training (Micikevicius et al., ICLR 2018, https://arxiv.org/abs/1710.03740), it says that an fp16 update is lost once the weight is ~2048× larger than it. I've updated the post with a citation to that paper.

bf16's shorter mantissa moves that to ~512×, which is significant.

The point of the post is narrower: a lot of published fine-tuning code doesn't keep a master copy at all. So what does that do? It gives you a magnitude mask where only the smallest weights (those under 512xlr) are updating. Which I thought was an interesting inductive bias that wasn't really noted.

I wrote this essay out of curiosity for finetuning, and as part of my Postdoc. I thought folks here who are finetuning models might find it interesting.

And thanks for your feedback, I'll try to post more of what I'm reading, too.

Yea that's pretty related. In this case, I'm looking at full finetunes (although we don't end up actually finetuning the whole model, just the small weights!)

If you fine-tune a model with weights stored in bf16 and let torch.optim.AdamW write to them directly, you get interesting behaviour. bf16 can only represent changes to weights if the change is at least ~1/512 of the weight itself, so at small learning rates only the smallest weights can update.

As a result, you have a sparse fine-tuner that trains the small-magnitude weights and freezes the rest, with the cutoff set by your learning rate. This is a thing - I found half a dozen peer-reviewed papers (with thousands of citations) that fine-tune this way, but none of them reference this behaviour.

I dug into one called Quiet-STaR, because (a) it's a cool paper, (b) they released their weights, and (c) their code works! So I diffed their base model and saw that indeed the small-magnitude weights were the only ones updated during their training. Then I reran its recipe with an optimizer that applies every update, and watched it diverge. A higher-precision optimizer needs a 3 to 10 times smaller learning rate to get the same result, and I couldn't achieve a better result.

This started with Barron, Gnanapragasam & Tschannen's "A Fix for Softmax and Cross-Entropy" (Oct 2026). They show that the standard way of computing cross-entropy rounds the correct answer's gradient to exactly zero once the model is confident enough (past a margin of about 5.5 in bf16, which they call the precision cliff), and that this, interacting with Adam, is what produces the "slingshot" loss spikes seen during grokking. Fix the rounding and the spikes go away: something people had studied as training dynamics was rounding.

That made me wonder what else out there is weird like this. This is one, and it runs the other way: their rounding bug adds instability to training; this one hides it.

The code, logs and evaluation results are in mweiss17/quietstar-bf16-adamw, in case you want to check the receipts.

Author here, and Postdoc with Hugo Larochelle and Chris Pal.

I saw the LocalWiki last week, and it inspired me to make a learning tool based on "local" research papers. Many ML researchers write code that relies on a long history of research in ML and applied math. But which papers are you actually building on?

You can paste a GitHub repo and Standing on will read its Python code match pieces of it back to the research papers from which it arose.

nanoGPT, for example has 15 Python files and links to 13 papers including Adam, AdamW, LayerNorm, GELU, FlashAttention, etc. If you go back further, you get Adam and then AdaGrad (2011), Zinkevich's online convex programming (2003) and finally you wind up with Nash (1950) and Kakutani's fixed-point theorem (1941).

It's a very simple symbol table map with 4,800 classes and functions in torch, transformers, diffusers, timm, sklearn and others to the papers they implement. I used Claude and Codex to label the symbols and named each paper's parents, and check them against our ML research map, Lacuna.

We only support python and machine learning papers only for now. You can export to BibTeX, CSV and Markdown.

Yes I’d say so. That and a line of work on compositional generalization in deep learning.

I don’t think that’s where we are headed. Humans are the bottleneck but they aren’t replaceable by LLMs. We experience high fidelity multimodal observations all day long, generated by a 13 billion year old gigantic quantum computer. LLMs get a few quantized tokens.

The universe has too much detail for us not to contribute useful information to a human-so scientific process

Thurston wrote about this in “On Proof and Progress in Mathematics,” including how his own approach changed from proving theorems to helping a community understand the mathematics.

Nicolas Le Roux, a Lagrange Prize winner / prof / strong mathematician, turned me onto that around the time I finished my PhD in ML last year. I have been thinking about it a lot, lately, for obvious reasons.

At first, it felt silly to grapple with what science is only after finishing a PhD. This publication from OpenAI finally pushed me over the line to publishing an essay about this struggle between AI science and human science: https://news.ycombinator.com/item?id=50253759

Author here. I work on AI tools for ML researchers, and have had this essay rattling around in me for a while (since I completed my PhD last year, actually).

Seeing OpenAI's publication on "Working with academic communities to build scientific understanding" motivated me to finally post it.

I expect we’ll continue getting faster at producing results than at turning them into ideas other researchers can understand and use.

Academia already rewards new discoveries more than the work explaining and connecting them (I talk about this at length, and am working on a subsequent piece that goes deeper into the history of that).

OpenAI organizing groups to understand these results seems like a useful response, but I think that structurally the incentives and culture aren't there yet.

Yea pretty much. The picture I have in my head is of those arcade machines with the sort of snow shovel moving back and forth that sometimes pushes quarters off a shelf (called a coin pusher https://en.wikipedia.org/wiki/Coin_pusher).

You try to figure out where there’s lots of coins on the edge and drop your coin in to get it to push more coins out.

So some fields are ready to get pushed, and maybe density or patterns in concept space can show you that

yea probably - to be followed :)

edit: you can also certainly point a code agent at this paper and your library and have something interesting in a few hours. Or better yet, give your agent the Lacuna MCP and let it rip on the problem https://lacuna.tiptreesystems.com/mcp

We actually have a high-risk research idea for how to do this. It involves looking at the clustering as a vocabulary that compresses the concept space, and (one form) of significance as how much your paper / addition of concepts changes the clustering/vocabulary. I.e., if you change the optimal compression of the space, that's probably important.

The academic paper cost a few thousand bucks, max $10k in compute.

Lacuna is far more expensive and ~100x larger. If you knew what I know and started building it today (LLMs got cheaper).

Paper summaries cost ~$50k, concept extraction cost ~$25k, author profiles cost $25k, research proposals cost another $50k. So call it $150k for model calls.

It's also an online / updating daily so that costs too, plus infra, plus "spillage".

I had not, and I find it deeply hilaritragic that his paper is paywalled by JSTOR. Thinking of you, Aaron Swartz.

Hmm, interesting. Yea I think ML is a much smaller field, too. ArXiv + all the conference / applied ml stuff is ~150-250k papers per year. PubMed is 1.5-1.7 million new records per year, and like 40,000,000 total records, whereas in ML we only had to process a couple million. Plus, pretty much all of ML is open-access, whereas medical is ~50% closed.

If somebody gave us a $10-$20m grant for compute, and a few (dozen) helpful doctors showed up to help us QA / figure out how to approach it, it could probably be done.

Yes, vocab drift matters a lot. Most of the pipeline is basically compressing out any of the weird vocab/language stuff, so it doesn't have numbers and proper nouns and is in a sufficiently abstract concept space to cluster nicely. We use a density-based clustering method (HDBSCAN) which when you set min_samples=1 works as an online clustering (i.e., incrementally adding new things is the same as if you re-ran the clustering in batch mode).

It cost ballpark $250k

Thanks for digging into one! I had a project back on this back in my Probabilistic Graphical Model days with Simon Lacost-Julien (I believe this was my worst-ever grade in a course)... so I’ll defer to you on the mechanism.

I do know you're right that affinity/residence time and efficacy are separate axes. Bio is one of our sparsest regions (due to closed access papers, and our focus on core ML), so proposals there are more likely to have issues like this.

We've got some ideas on how to improve these issues, but I really appreciate you helping us by pointing this one out!

Necessity is the mother of invention.

I agree with you, though my own motivation is not centered on war, but (a long tail of) existential risks facing humanity, mostly centered around collective action problems.

Thanks for checking it out! We're working on a collaborative discussion / editing feature mediated by Althea (our AI agent). That should provide a way for domain experts like yourself to contribute to improving it. Hopefully we'll have that soon!

I'm working on a piece inspired in large part by William Thurston's "On proof and progress in mathematics"[1]. Should publish it next week.

Part of why math is effective is that its extraordinarily compressible. And you can take those ideas and apply them elsewhere (composition works well). The natural world has a lot of detail and is therefore hard to compress. That's why it's a good source of new ideas. Consider the relation between the McCulloch–Pitts neuron (MCP lol)[2], developed to model biological spiking neural networks, and an inspiration for modern artificial neural networks.

[1] https://arxiv.org/abs/math/9404236 [2] https://en.wikipedia.org/wiki/Artificial_neuron

Yann LeCun is right, sometimes