Bf16 might be freezing your weight updates
lacuna.tiptreesystems.com
lacuna.tiptreesystems.com
Every link submitted by the author seems to be self promotion...
And thanks for your feedback, I'll try to post more of what I'm reading, too.
Isn't the standard numeric fix for this to probabilistically update the weight? That is, if the desired change is current_value * 1.0014, it should have a 70% chance of going to the successor, and a 30% chance of staying unchanged. Over a number of steps, this will adjust it appropriately over time.
[1] What is New in Stochastic Rounding: a Survey on Theory, Hardware, and Applications, https://arxiv.org/pdf/2603.06060v2
As a result, you have a sparse fine-tuner that trains the small-magnitude weights and freezes the rest, with the cutoff set by your learning rate. This is a thing - I found half a dozen peer-reviewed papers (with thousands of citations) that fine-tune this way, but none of them reference this behaviour.
I dug into one called Quiet-STaR, because (a) it's a cool paper, (b) they released their weights, and (c) their code works! So I diffed their base model and saw that indeed the small-magnitude weights were the only ones updated during their training. Then I reran its recipe with an optimizer that applies every update, and watched it diverge. A higher-precision optimizer needs a 3 to 10 times smaller learning rate to get the same result, and I couldn't achieve a better result.
This started with Barron, Gnanapragasam & Tschannen's "A Fix for Softmax and Cross-Entropy" (Oct 2026). They show that the standard way of computing cross-entropy rounds the correct answer's gradient to exactly zero once the model is confident enough (past a margin of about 5.5 in bf16, which they call the precision cliff), and that this, interacting with Adam, is what produces the "slingshot" loss spikes seen during grokking. Fix the rounding and the spikes go away: something people had studied as training dynamics was rounding.
That made me wonder what else out there is weird like this. This is one, and it runs the other way: their rounding bug adds instability to training; this one hides it.
The code, logs and evaluation results are in mweiss17/quietstar-bf16-adamw, in case you want to check the receipts.
i personally do finetuning in FP16 because then the LoRA merge at the end has less rounding error.
for most models it's lossless as well (when coming from a BF16 source), it's very quick to scan the weights and make sure they're all in range if you're concerned.
This issue was recognized a decade ago when we first started training with fp16.
I read a bit more into the history of fp32 master copies, like in Mixed Precision Training (Micikevicius et al., ICLR 2018, https://arxiv.org/abs/1710.03740), it says that an fp16 update is lost once the weight is ~2048× larger than it. I've updated the post with a citation to that paper.
bf16's shorter mantissa moves that to ~512×, which is significant.
The point of the post is narrower: a lot of published fine-tuning code doesn't keep a master copy at all. So what does that do? It gives you a magnitude mask where only the smallest weights (those under 512xlr) are updating. Which I thought was an interesting inductive bias that wasn't really noted.
The title and a skim of the beginning didn't give me an idea that's where the article was going and I stopped reading further at first.