Comment by kristopolous | Hacker News Reader

Comment by kristopolous | Hacker News Reader

Parent Full thread

So let's assume a completely naive approach. modify any weight arbitrarily, run an eval if eval improves arbitrarily modify another weight and repeat ... do something like 5,000 rounds. Just brute force, zero-intelligence, random selection. how does the naive base case perform on N=20 random trials?

Unless you include a basic test like that these results are meaningless. Was there p-hacking? Why just these 3 tests? Did they try others and it failed?

I'm currently running this myself. I'll see if the naive approach improves results. Can't wait to have these in hand and have people call me stupid when I bring it up with empirical evidence, it'll be awesome.

That's exactly what I was thinking. Actually, if you look at the "Is Qwen improving" 25 and 10 forward passes, it doesn't seem to improve much.

Additionally, since it relies on using GSM8K and MATH datasets: how does it compare in terms of compute with other training methods?

What you're describing is a shitty version of gradient descent, which is the technique used to train LLMs and other neural networks. Proper gradient descent simultaneously calculates how much every weight improves the output, and edits them all simultaneously in proportion to that.

obviously. unless you have the equivalent of a coin toss, no metric carries implication. Does naive noise give you nothing? The null hypothesis needs to be established for these particular evals with this particular model.

Doing basic empirical hygiene shouldn't be too much to ask for.

For instance, on their base version of qwen if your turn thinking off you get a 52.0% on their evals. That single flag recovers ~72% of their entire claimed improvement - base model, toggle thinking off, scores go up 72% of their improvement.

I'm still investigating various wildly naive modifications. If I can hit their numbers doing inarguably stupid things then it puts their substantive claims in doubt.

Reply on news.ycombinator.com