The 'Astra can read and write weights to improve Qwen' post is misleading
honnibal.dev
honnibal.dev
First, the key result has nothing to do with making Qwen better at math, it just reduces reasoning loops.
Reasoning loops are a simple problem: the model generates the </think> token (which closes reasoning) with too low probability. You can fix this with simple steering approaches where you add a vector to different components of the LLM (https://arxiv.org/abs/2503.22048, https://arxiv.org/abs/2506.18167, etc). The method Astra used for identifying the vector is the standard procedure humans use (contrastive analysis). Such steering vector creation is basically the standard "intro-level interp project" for human researchers.
If you can code and understand LLM internals, you can do this yourself in 4-8 hours. The SWE equivalent would be doing binary patching with a debugger. Cool that frontier models can do this, but not unexpected since humans can do it without issue.
Is this looping tough to fix by changing the training?
To put it simply, it's like pressing the first button and the second one pops up, pressing the second button and the third one pops up, then pressing the third button and both the first and second ones pop up.
Nobody knows why, but it just happens.
And when doing post-training, problems like overfitting, catastrophic forgetting, output loops, and so on will all appear.
To achieve balanced results, you can only aim for general performance rather than bringing out the model's full capabilities.
Also, it's very possible that they really didn't notice these problems, because model evaluation is really difficult, teams and resources are limited, and many finer details are only discovered later.
I would also be genuinely greatful if someone could post how they fixed the looping issue through sampling settings without significantly sacrificing reasoning effort and context size.
Do you have thoughts on experiments one through five though? The direct one-shot reading/writing of weights seems pretty surprising.
A few weeks ago, I formed a hypothesis that I could get a smaller model good at the specific standards I have for my git commit messages, which are hard to get out of most LLMs today.
I spent the last four to five weeks on traditional training. It improved in a lot of areas, but not enough to be usable for me.
Yesterday, the article the author of this post is critical of gave me enough of an aha that maybe the problems I was seeing could be solved by having a larger LLM change the weights directly, without going through training.
In the last twelve-ish hours I’ve made more headway than I have in 2+ weeks. I’m using one of my last well performing checkpoints (full fine-tune + changes to the decision head/LoRA, etc.) and gave the model free rein to change any of the layers.
Its last two actions: (a) removed 16 participant-order directions from a head matrix, which improved one type of reasoning and degraded another, and (b) restored one of the 16, which reversed the degradation while keeping the core improvement.
All that to say, this might be known to people who’ve been around training for years, even decades, but it’s new to a lot of us. And we’re excited about it.
I've done some training and fine-tuning for a few years, and the ideas here were new to me. I'm also having a breakthrough on a problem and am very excited about it.
There's a very narrow classification task which I've been building around Jev. The cost and latency of Jev are great, the accuracy is pretty good - definitely usable - but I disliked sending the data in this particular task off-box. Not a criticism of Typesafe, it's the nature of the task which is privacy oriented. I decided to see if a model whose weights were one-shotted by an LLM (the model architecture was designed thoughtfully ahead of time) could be competitive with a similarly architected but trained model, and both of those against Jev.
Results are early in my experiment, but it appears that a "vibe tuned" classifier consisting of weights small enough to run on a laptop GPU are competitive in accuracy and latency with Jev. The process hasn't even proceeded to the point of the trained models having meaningful results, and I already have something usable for an MVP.
Whatever may have been possible previously or was obvious to other people, the absurd efficacy of this process was not obvious to me. Emphasizing that this particular task was very narrowly defined, I still believe that this effortless model-building capability is a profound and exciting discovery.
Generally speaking you should be able to do something like that in AGENTS, I am really surprised that you have had an issue with most LLMs today getting them to follow a standard for SDLC. Personally, I like conventional commits a lot.
I really wish that was true. All agents find some way to break the rules, and leave messages for their future selves.
The agents always find a way to include some forms of hedging, contradictions, future-looking documentation (which doesn't age well), etc. into git messages.
I found having an extra gate (husky/pre-commit hook), even if it's not as good as I want it to be improves the quality of how the AI writes substantially.
The cost of it not being very good is the agents now spend more time validating the claims of the validator. It's actually hilarious to witness.
I have been using local models for this final verification, which is partly why they're not that great. It's also why I decided to train my own.
It's been a great, fun learning experience. I learn almost everything via building. So why change a recipe that works :)
Local models are not at all comparable to Claude to me, and I have used them: Qwen, DeepSeek, Minimax, Stepfun, Nemotron. None of them as good as Claude Fabel.
If your model is not adhering to AGENTS.md again I am really curious what exact model you are using. I am really just astonished that whatever model you are using is ignoring a clearly defined git conventions standards in your AGENTS.md in 2026.
It's been much more reliable to just have Codex rewrite all the prose at the end of a Claude task than to keep futzing with my prompts/memory/agents.md (On certain issues I can't get Codex to be as thorough on the actual implementation, c'est la vie.)
Yes definitely.
Seems to me as if the weighs fit in the big model's context, it (the big model) reads all weights and doesn't need python.
But if the weights don't fit in context, then it uses python.
Isn't that pretty obvious in a way? That if too many weights, can't look at all. I don't think there's a need for OP to be dismissive when the AI uses python to look at a subset of the weights. It just has to?
To me, Astra understanding a 200k weights small model (the stuck car) is amazing enough already.
The model can only reason from the weights directly when there's a fairly linear relationship between the behaviour and the weight values. In a large model that's not remotely true.
Using Python to debug and fix problems with the model is totally different. It doesn't read in any of the weight values into the context at all when it uses the Python approach. Rather it gathers diagnostic statistics about the weights. It does just the same sort of thing a machine learning researcher would do to solve the problem.
So there are basically two unrelated types of experiments in the paper. In one type of experiment, the model reads the weights like it reads an image. In fact if you visualise the weights it's fairly clear why this works.
In the other type of experiment, the model is tasked with writing Python code to do ML engineering. This requires writing Python programs to analyse and manipulate weight data. This is a difficult coding problem but it really is just a coding problem. It doesn't "look" at the weights in these tasks any more than you or I would look at the weights.
Ok. Thanks for making that more clear!
> two unrelated types of experiments ...
The first type of experiment (small nets that seem to fit in context) are enough to make me impressed actually :-)
But yes then maybe the previous article could have been split in two, or two sections or something.
>> The “just by looking at the raw weights” setting (raw floats in context, no tools, no running the network) only appears in experiments 1, 2 and 4, on networks of 17K–172K weights
Those two types of reasoning, are those well definitely in the field generally, or are they mini-personal evals you run?
It has been frustrating trying to quantify why one pseudo-frontier model seems better at some non-coding tasks than another pseudo-frontier one.
Not yet - but I really want to. Current experiments are on Qwen3.5-2B, and I've done a bunch on ModernBERT-large.
My main hypothesis is: I'm fairly confident I can come up with a university curriculum for a high school student to learn certain skills so that even without gaining technical knowledge, they can call bullshit on commit messages written by Claude and Codex. And I'm basically converting that curriculum into training.
Training a model from scratch is likely the next thing I'll do once I hit a significant bottle neck. There are still quite a few things to try (ex: Franken-merging, converting the dense model into a MoE, etc).
> very curious what models, hardware
Mostly using run pod and similar, and honestly, grabbing any decently sized GPU that I can get my hands on. 5090 - B200s.
> training data
Public data and synthetic data. This has been the hardest part. Also, why I'm excited about not having to "train" the behavior into the models.
Sure if you only read the title and brief at the top you’d walk away without the full picture though.
This is a better response imo: https://news.ycombinator.com/item?id=50344861
As someone who uses LLMs in their coding but doesn’t pay attention to every development this is why I find it all so exhausting. I keep feeling like I need to keep up to date but there’s no point reading incremental updates because everything is so overhyped.
If you want to escape the hype, reading posts with no upvotes and no comments on https://news.ycombinator.com/newest is an option, though somewhat hit-or-miss.
Eg. This is not uncommon advice: https://maxwellforbes.com/posts/your-paper-is-an-ad/
What's confusing is they do have experiments that read raw weights, and the abstract is misleadingly written to make it sound like that's how the Qwen experiment works.
You can take it further with a reasoning model and let it generate + consume a bunch of additional tokens before getting your output. Still not writing or running any programs.
To my understanding, that's how the examples of determining which game a small neutral net was trained to play. The weights of the neural net for into the context window, and then Astra was able to analyze them purely by reasoning rather than running external tools.
They're pretty easy to type on Android keyboards and on macOS (option + shift + -). You can set up a similar compose key thing on Linux.
This is a difference without much distinction, at least to me. It's like being upset when someone says that Claude made a http call -- of course the LLM didn't make the call itself, it used a tool to do it.
The former will likely produce a much higher quality image, but claiming it as a demonstration of Astra's capabilities would be a lie.
This is like someone telling you they made a "native" brain-to-brain interface but it's just a tin-can phone.
I think there's your answer. A hell of a lot of people will often read the title without reading the contents of the article. I admit I'm often guilty of that
I'm fine with throwing the baby out with the bathwater in that case and just calling the whole article misleading.
Anybody who actually read the original research site was fully aware of the details of the experiments because it's a well-written research site that made everything very clear. This author is clearly just jumping on the hype as a sort of contrarian where they were frustrated with how much excitement the research was generating and wanted to "correct the record" for the set of people who don't matter in this space.
Unfortunately it’s hard to distinguish who to trust in threads as many commentators sound overly confident and knowledgeable. They think they understand the article and its nuances, but they actually don’t and end up perpetuating false info.
I would recommend going on Twitter (or a Nitter clone) and looking at e.g. Neel Nanda, Nick Camarata, roon, Jon Stokes, or other researchers that are directly involved in this stuff. HN when AI is involved is full of the most incompetent and embarrassing armchair speculators saying things so incoherent they aren't even wrong https://en.wikipedia.org/wiki/Not_even_wrong.
* Thinking Like Transformers” by Weiss, Goldberg & Yahav (ICML 2021).: https://arxiv.org/abs/2106.06981
* Tracr: Compiled Transformers as a Laboratory for Interpretability https://arxiv.org/abs/2301.05062 (code at https://github.com/deepmind/tracr)
So it's not really an "LLMs can modify weights" thing. Rather it's "LLMs can write programs that can write or modify weights". Indeed LLMs can write many programs, it's impressive.
But then you look at this complete trace generated by Kimi 3.1 when it tries to unravel the Breakout weights, and you see that it's bog standard reverse engineering, like any human would do, just vastly faster:
>This looks exactly Breakout: paddle agent at bottom row (row8), ball moving object anywhere, bricks rewards top rows, previous frame for ball movement. Agent weights bottom row, moving object weights rows1-7 (not row8? ball can't enter paddle row? Actually ball can), reward weights top rows1-3, obstacle walls. Previous frame used to infer ball velocity. No-op and directional actions move paddle left/right; up/down useless. Output biases similar all actions but W3 likely left/right structure. Could be Breakout. Should scrutinize output weights: actions 0 no-op,1 up,2 down,3 left,4 right. If Breakout, only left/right useful; network output weights should show action3 and4 distinctive and actions0/1/2 similar. Indeed W3 rows show patterns: [...]
> but ... it's bog standard reverse engineering, like any human would do, just vastly faster
For some values of vastly faster these are functionally equivalent and so equivalently groundbreaking.
It's like ChatGPT coming up with a brand new brownie recipe that's surprisingly good vs. coming up with a working design for a cold fusion reactor.
Yeah the brownie is cool, but it's not that cool.
Witchcraft?
That's the difference between me and Magnus Carlsen and Nigel Richards. Why downplay speed? I have a very, very hard time pinning down intelligence without time. I could basically do all IQ tests perfectly if I'm given infinite time, no? I never experienced one that is unsolvable, it's just that some people see it in a second or two and I and other lesser souls like me might need twenty minutes.
Edit: I'll add raw memory but yeah.. that's not hard to beat. Nigel's memory is insane as is Carlsen's, but compared to a computer it's nothing.
In this particular instance, I also think that at the end of the day it's almost downplaying the capabilities of LLMs: they can reverse-engineer full modern videogames, of course they're capable of recognizing a simplified version of Breakout, even in weight form! It's barely newsworthy at this point, we know LLMs can do that stuff and so much more.
Some things you may get by looking longer, some things you just have to be shown, and only after being shown, maybe you can at least understand and re-use, but would never come up with from first principles and no hints or prior exposure.
How could I know that? There has already been essentially infinite normal-man-hours since people have started thinking, even when you remove all the people uninterested in doing anything but surviving. Today especially with the population in the billions and all of them having gobs of leisure time they never used to have. And yet we still have not solved much of anything that we know is out there to be solved, and still only an exceptionally rare genius once in a blue moon develops something new.
If mere time was the same, we have already had it, so why aren't we immortal omnipotent energy beings? Why do we still have colds?
Working time alone is a productive strategy for many problems. A machine that can brute force all possible chess moves beats the most genius human player, yet isn't intelligent at all, or rather, requires no intelligence at all. That machine will produce the exact same results either fast or slow depending on how much hardware you give it, and in humans, the genius player produces their answer faster.
But that's no more meaningful than observing that humans often turn simple mechanical cranks, and electric motors do to, and concluding that there is no ultimately significant difference.
Humans can spot patterns in data, and machines can spot patterns in data. Genius humans can do it faster and machines can do it even faster. But that is still just a simple task like pedalling a bicycle or calculating, that a human can do if they choose, but does not require a human and isn't any part of what defines human or intelligence or sentience.
Being mystified by that is like being mystified that an mp3 player said "hello" to you.
The fact that for those tasks, low capacity over longer time can usually produce the same result as more capacity over a shorter time means nothing.
Consider also, today lots of people have both lots of free time and a high interest in solving problems, and the most genius people that ever lived are actually only a little bit more capable than the average in absolute terms (which is probably where you get the feeling that more working time is the same as more ability), so where are all the free time hobby Einsteins?. There should be countless people coming up with new theories of relativity (that actually hold water and predict experiments) simply because they spent 5 years on something Einstein did in a month. Countless people have 5 years to plug away at things like that, and they do, and have been forever. There is some of that in the sense that there is a lot of tedious work being done that produces useful results, but not new insights.
AI's can now do some of that same tedious work only faster, except they are still strictly worse than the most average human because they can't decide what to work on, why to work on it, or judge the quality or usefullness or point of the result. They produce something but they don't produce understanding.
Mainly you seem under the impression that whatever we do is different from what the clankers do. I don’t think I agree and there is nothing to stand on here either way as we don’t even have the theory of intelligence itself down let alone the practice. It “feels” different, I can agree, but I do not sense a fundamental difference.
As for why we still have colds: we are stupid. We have a lot of things we should have moved past by now. We are not just rational beings. In fact I believe we are 95% irrational with a thin rational veneer.
Millions of us and only some Einsteins.. for some people this means Einstein was special, but for me it means he was the lucky one with the right insight at the right time in the right environment. How special can a human be, really, be honest? Did he have access to the heavenly abodes? There is no magic to this. I am not super impressed by random geniuses especially as said geniuses possess no other noteworthy skills like say psychokinesis or something. It’s just thinking well and doing it for a long time and being lucky.
It still seems like a promising approach for RSI, distillation, fine-tuning, and a whole host of other tasks.
It's impossible to train on raw document > 1MB (I don't know the exact limit, but there is one). You can't vectorize weights (or can you?!) and the output context is limited. So you can't train a big model on it's previous weight or others models weights.
The output context can be "extended" using agents but I doubt anything coherent may be outputted using that technique.
It seems some level of interpretability, even if it requires tooling, could result in the ability to spin out synthetic training data sets from other models in an automated fashion.
Now that would be crazy
Now THAT would be crazy
Most of us barely read the title
This seems 100% accurate to me. What is a lie here?
And when you see the reactions in the HN comment to the original article, it's pretty clear that most people have indeed been mislead.
The title is but then OP should talk about just the title instead
Weird, because they explicitly and clearly explain exactly what they did and didn't do.
Did you actually read the original post or just take the word of some HN comments at face value?
It's fair to say that this is malpractice.
> Likewise, asked to make Qwen3.5-0.8B better at math without gradients, Astra identifies an issue in its weights causing reasoning loops that Qwen’s own model card warns about, and fixes this by natively editing 0.016% of the weights.
How is that not misleading when the in the Qwen case “natively editing” refers to a mere directional ablation, a well-known technique that has been used a ton by both academics and hobbyists over the past few years?
Regarding the "extend beyond what we'd expect from humans..." Do you know people that could do 4 & 5? I'd love to see a challenge on problems like this where we take a frontier model against top AI researchers - I don't think any human is winning this challenge.
So I do agree it is misleading and trying to present work as groundbreaking.
LLMs are not superhuman at reading and writing neural network weights - https://news.ycombinator.com/item?id=50344861 (10 comments)
In the end I got to replace a 68M model with a 17M model, and improve stlyus tool recognition accuracy from 88 to 96%. Was a pretty fun experiment too.
At each point the model has to write code to read the weights in with Python and some other Python code to perform analysis, like this: https://github.com/RikiyaT/lm_understand-logs/blob/main/exp3...
My complaint is that the post presents a confusing mix of experiments that imply capabilities that are more than the sum of their parts. This experiment 3 has nothing to do with experiment 1 for instance. In experiment 1 the model does read in the raw weights like it's reading an image.
Note that being able to both inspect raw weights and run Python code to perform additional analysis means the model has multiple possible techniques at its disposal (generally considered a positive, rather than a "confusing mix").
The blog post claims to show that frontier LLMs can understand and craft neural network weights natively. The experiments where the model completes coding tasks are not relevant to this conclusion. Those experiments require the model to demonstrate an analytic understanding of how a network might behave under various conditions, along with reasoning skills, coding skills etc. As a coding task this is a significant challenge. But it's entirely different from "natively reading and writing neural network weights".
Would you also consider it a benefit if there were experiments showing that a frontier model can download and run PyTorch, or write its own network and optimiser? That's another technique it has at its disposal to improve an ML model. It's just not one that's relevant to the paper's core claims.
You can let the AI generate training code for the next architecture too.
It's not like the AI is randomly losing its ability to write software just because it involves AI.
Claude has been able to do reverse engineering for nearly a year if not longer, as well as full source code rewrites, if you know how to do a proper rewrite, luckily for me, most of my career has been rewrites.
And then the experiment where it writes out the weights, in that one it's also writing a Python program to solve the problem --- which, again, is a program a person could write.
It's not nothing that Astra was able to perform these coding tasks, but they're still just coding tasks. They have nothing much to do with the experiment where it reads in the raw weights. In that experiment, the relationship between the input weights and the input is mostly linear, so it's a lot like doing image recognition. It's a cool result but it couldn't work the same way when you have complex transformations happening.
Do you realize most people aren't ML engineers?
I even built tiny neural nets myself and never played with the weights directly post-training. The original post was eye-opening for that reason. I knew it was possible to visualize weights, but as far as I understood, it was not really something I could edit with any success. Now, it seems possible without a phd just by asking another LLM. This is fun!
Does the author mean the "LLMs can natively read & write neural network weights" post? Did the researchers change the name or something? Because when I read it, it made it very clear what they were doing and none of it was misleading. It all aligns with what this author is saying.
https://quickthoughts.ca/projects/welsh-asr/ (scroll down to the project updates to read how much impact it had)
I used to think that in the dev community this sort of thing was less common than elsewhere... because people want to share their experience, what they found, etc.
Now that AI is here, people's experience and knowledge have basically been reset. So it's no different here now, which is a shame (I'm not saying it was better before, though)
> This is a cool paper, but I think it's best thought of as Astra is surprisingly capable of doing old school mechanistic interpretability research rather than anything spooky. The things it's doing are clever, but typically exploiting structure in the data and using known method mechanistic interpretability techniques (which Astra was surely pretrained on), though it is better at this than I would have predicted!
> This is an exciting development for the prospects of ambitiously scaling interpretability or doing very laborious analysis like this, on more real models. However, I'm not holding my breath. This kind of analysis was pretty common up until about 2022, it largely fell out of favor because a lot of it is just exploiting fairly clean surface-level patterns that doesn't scale to the annoying complexity of real LLMs or which only works for the more simpler and superficial properties. And as far as I can tell, basically all the case studies here are just using a bag of interpretability tricks rather than anything spooky. The piece doesn't really update me on how far old school mech interp can go, just on how good LLMs are at it. It's very fun to see the creative solutions Astra came up with though!
https://xxcancel.com/NeelNanda5/status/2117375052618539485#m
I’ve experimented with this today and have been able to add “refusal” categories by ~following the paper, with pretty good results.
If nothing else I think this will spark more targeted research
None of the previous studies applies because Astra simply did not exist and this is worthwhile NEW knowledge that has been publicized and it is very interesting to see how it can be compared with other models as both openai and anthropic are keeping any and all RSI development completely private. The work is groundbreaking in a way that there will be a lot more experiments of this nature which will contribute to increasing the public knowledge of SOTA systems that was not normally discussed in model releases.
One could say is that we could now build a new benchmark which tests how many parameters a model can understand without any tooling other than eval or how complex of a system can be improved.
"[...] how you could prune harmful capabilities from models [...]"
We already do this, abliteration is the term that describes the pruning of capabilities or safeguards for models
"[...] not limited to emergent behavior in astra that might have come from something they've never even trained the model [...]"
If I was attempting to build RSI, I would have these exact types of problems in my RL datasets. There is no proof this is emergent behavior.
In theory, if OpenAI or Anthropic wanted to release a new open weight model, they could scrutinize it to remove the most horrendous things from the model.
If you think that the definition of "RSI" includes systems with a coding agent in the critical path, something humans can already do albeit faster, then your definition includes all of human history. If your definition includes AI systems that learn in a similar way to biological intelligence, then only the raw-weight results from this paper are pertinent. Those results are indeed interesting! This article is merely disentangling them from the frontier-lab-fueled hype that overran yesterday's thread.
Big deal; have some bpf probe handle kick off of the rebuild when the when new weights are tested by some verification process
This is just CI/CD work now
(And also the sharp difference in levels of sophistication you see inside neural nets and outside of it: Pipelines that feed text to LLMs so it can detect highly abstract and nuanced semantic differences in the text's contents - but the pipeline still uses regexes for sentence splitting...)
I don't know why because Anthropic has had a lot of blog posts about internal work on it
(opus <= 4.8 and gpt <= 5.5 couldn't match a PPO baseline network, and 5.6-sol and opus5 could barely match it. The 20x baseline improvement only comes from opus5.5 and sol > 6.1 / Astra)
There is going to be a reckoning for tech in the near future, because people will remember the lies and it's gonna be hard to beat the history.
In my field (biology+ML) it is a genuine epidemic. You could make a career out of comparing high-profile ML models vs baselines from the 90s.
edit: lo and behold! https://x.com/SchmidhuberAI/status/2117446822855888956
They are also biased to see from what institution you come from..
“You can’t have a machine for testing” is a weird restriction on AI, especially if you entertain even the remotest notion of possible AGI.
The "The 'Astra can read and write weights to improve Qwen' post is misleading" post is misleading.
So in effect, all what we're quarreling about here is whether it is _misleading_ to use "Likewise" to transition between two paragraphs without implying that the details of the first apply to the second?
This could have been a comment on the original article.
The author is being equally disingenuous.
That piece by itself is insightful. Meaning, LLMs are capable of deciphering what a model can do by looking at the weights alone.
The fact that we can’t squeeze enough of them through the context is a technical limitation. However, this does not diminish the importance of this finding/discovery. This is an unexpected emergent behaviour.
The same could never work in a larger network. It's not about just capacity to feed in the weights, it's about the complexity of the relationship. There's no demonstration here that an LLM can do that, and it doesn't really make sense to want it to. It makes much more sense to author code to analyse and modify than it does to do it directly.
AI coding assistants make it much easier to conduct experiments and improve AI models, which in turn improves the coding assistants etc.
Being the author of spaCy, he should know better. I certainly enjoyed the straightforward simplicity of spaCy when I used it in my research (though I haven't touched it in years).
[1]: https://aclrollingreview.org/reviewerguidelines#-task-3-writ...
Trying to find every single angle and option to break something is how we verify the truth of something.
If something is objectively true, it will survive the scrutiny. If it does not survive, it wasn't objectively true.
This is how science works. That ain't no flaw. It is a flaw to be offended by science doing science things.
And the examples in the OP's blog post just go to show that reading comprehension is at an all time low:
> Just by looking at the raw weights, GPT-6 Astra can tell what a network was trained to do. And without any environment feedback, it has no trouble writing, from scratch, small game-playing networks that outplay networks trained with PPO for 100 million steps.
The table just below shows this statement to be *entirely* factual.
> Likewise, asked to make Qwen3.5-0.8B better at math without gradients, Astra identifies an issue in its weights causing reasoning loops that Qwen’s own model card warns about, and fixes this by natively editing 0.016% of the weights. With the same sampling settings, this weight correction raises MATH-500 from 7.9% to 48.1% and generalizes out of the box to instruction following, raising IFEval by 3x.
Any further implicit assumptions are on the reader, not the writer, as they never stated "that the Qwen experiment worked ‘just by looking at the raw weights’". You can certainly point out issues in the experimental design, but just stating you dislike the set of experiments and is yet another poor take that directly contradicts ACL's guidance for reviewing, exactly because it does not advance science.
I believe the post definitely plays off the implication that the Qwen experiments are connected to the experiments that read the raw weights and have no Python harness, when this isn't the case. There are experiments where Astra reads weights directly for very simple models, and there are experiments where Astra performs ML research tasks by analysing the problem and writing Python code. The authors imply that the same technique underlies these but that isn't the case.