LoRA Speedrun – a public wall-clock leaderboard for fine-tuning techniques
github.com
github.com
But it is not the MBA's view of winning, it's just one potential conclusion you could draw from the bitter lesson of Machine Learning. As long as the need for more intelligence outpaces the economics of using intelligence, you'll get bigger models. This idea that small, fine-tuned models can outperform bigger models capabilities wise is mostly misinformed. They are genuinely good at other metrics, but sadly more actually means better in ML-land (most of the time at least).
There’s evidence that model size and representational capacity are not exactly the same, and that scale is maybe more important for learning than it is for representation (past a point). Consider the early work from the current neural scaling paradigm. The Chinchilla scaling study shows that smaller models can match the performance of larger models by training longer.
To GP’s point, if everyone is exploiting the scaling lever, few resources are being allocated to finding more efficient training algorithms that could let us work with right-sized models instead of pulling the scaling lever as hard as we can afford to.
I’ll end with a dramatic example from my field of materials science (which admittedly might not strictly generalize to LLMs). A lot of the field is pursuing the model scaling strategy, and it’s still paying off. But [0] recently reported competitive accuracy with much smaller models that run faster and can address much larger problems. The model architecture is pretty much the same, but they use a different training strategy and really focus on data quality
Based on a quick cursory glance at your example: Better data + better technique led to a better result with less parameters. Would you assume that then scaling both the dataset and model once again would lead to even better results? If you haven't fully encompassed the underlying distribution with datapoints, intuition says yes.
What I wanted to initially highlight was actually something slightly different: Specifically that people keep trying to "outsmart" optimizers by either fully hand-crafting solutions or skewing existing machine learning algorithms via additional tricks that are supposed to encode "human intuition" or something similar (to be fair there are ways to do it correctly). These all tend to fall short in a few years simply due to "line go up" being stupidly effective (compute getting cheaper, more training data being available, better optimization strategies, better architectures) [0]
Specifically this idea of small fine tuned LoRA models falls into the trap quite often: People assume you can beat the big, slow, general purpose LLMs with a small highly specialized model that has been fine tuned on the "good" human intuition of your special inhouse dataset.
LoRA can do great things, but it is often misunderstood what LoRA actually does.
0: http://www.incompleteideas.net/IncIdeas/BitterLesson.html
What I’m pushing back on is what I think is a sort of one-dimensional view of Sutton’s bitter lesson. People seem to equate it with model scaling, but there are lots of general ways to leverage computation that don’t involve just scaling models and supervised training datasets up. For example Sutton’s first example is straight up search, no parameters at all.
The point of the force field example is that it seems you don’t need billions of parameters to represent the functions we’re interested in, but with small models it’s harder to find those functions by pushing harder on the standard training algorithms, and that maybe some different algorithm that leverages computation more effectively could do so.
But that's exactly my claim: Smaller models can win on the computational efficiency front, but not the overall capability front. And as long as compute is getting cheaper, investing in more efficiency while there are still major capability on the table isn't a good business strategy.
Smaller, efficient models could lead to some really interesting things though especially considering it could lead to some Jevon's paradox like moment. To be honest, I feel like the biggest issue in regards to LLM usage in practice is that the patterns of use aren't really well developed. Yes we have agents, but it's still somewhat unclear what an agent can "do" - People seem to mostly focus on replacing some kind of existing process with an agent driven one, but actually coming up with AI-native processes is way harder.
Or put differently a wide exploration of long chains of obvious insights might be more valuable than a more narrow exploration of shorter chains of deeper insights. But perhaps that is a wrong sense of the difference between a small model and a large model.
A deeply ironic comment which associates <THING YOU DON'T LIKE> with <GROUP YOU DON'T LIKE> due to complete ignorance about the group. An MBA would never approve a technique with basically unlimited capex. So I hate to break it to you but "bigger weights" is 100% the computer scientist's view of winning because everything is an "abstraction".
https://www.lesswrong.com/posts/PagGF8roBJmjLunsX/competitiv...
I think the idea is interesting though, although I wonder if training time for LoRA is such a bottleneck to deserve its own, extremely narrowly scoped, leaderboard. Maybe if it was more tasks or more models we could hope that it transfers? With a single task, and a single model, I’d be afraid of this overfitting pretty heavily.
For NanoGPT, I think the idea always was that the ideas can be transferred to much larger models, or serve as stepping stones for investigations on larger models.
Have we stopped treating software as a black box? This behavior will only lead to devs moving away from OSS to avoid the AI stigma.
As an author: show me why you thought this was interesting and why you’re doing it, and why you think it’s relevant. What does it build towards? What does climbing this leaderboard mean to me?
Absent those things, this is just some thing my opus could generate as well.
This makes it impossible to really check their claims in practice.
The only way I've seen these debates settled is by using a fixed task and having someone act as a referee. That's what nanoGPT's speedrun did for optimizers.
The goal is to make a public record that shows which training tricks really save time and which ones don't work when tested again. Each record should explain how the method works, so over time, the leaderboard becomes a kind of lab notebook.
As more records are added, the leaderboard serves both as a ranking and as a detailed log of what was tried. I agree, and that's my next step: adding a second track with a different model family and task (SmolLM2 + SQuAD), so we can actually test if these methods transfer instead of just promising they do.
Honestly, your opus could probably build the framework too. But a leaderboard is more than just code. It also depends on someone being willing to review submissions for cheating, make decisions when things aren't clear, and keep adding new records even months later. That part can't be automated; it needs to be maintained by people.
When did we begin to? Why?
What is LoRA in this context? the communication protocol? Or another term appropriated by LLMs? Why the speedrun?
LoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning technique for large language models.[2]
The AI slop "why" section in the readme is completely useless. I struggle with this too. I start every project doc with a "why" then "overview" section, and AI is still horrendously bad at the "why", filled with vapid corpo/tech startup speak. I usually write the "why" myself, or at least put down the outline to be completed.