leogao

> The purpose here is not to responsibly warn us of a real threat. If that were the aim there would be a lot more shutting down of data centres and a lot less selling of nuclear-weapon-level-dangerous chatbots.

you're lumping together two very different groups of people and pointing out that their beliefs are incompatible. of course they are! the people who think there is a real threat are generally different people from the ones who want to push AI progress as fast as possible! the people who say both do so generally out of a need to compromise rather than there existing many people who simultaneously hold both views.

39% just doesn't pass basic muster. in the past year, my rent hasn't near-doubled. it doesn't cost anywhere near twice as much as last year to buy food or clothing or transportation. 39% inflation over the past year would mean the economy is rapidly shrinking in real terms.

the converse is that good hiring managers who can spot actual undervalued talent have a huge advantage.

in the grand scheme of things, this is a very small amount of plastic waste, and as far as resources go, one of the less scarce ones. at some point, the cost of the hand wringing to avoid waste is more of a drag on society than the actual wasted material itself.

where are you getting this 39% figure? inflation in 2025 was only 2.7%.

this kind of cynicism hurts young people who actually do good work on their own though.

I'm not an expert at hardware, so take this with a grain of salt, but there are two main reasons:

- Discrete optimisation is always going to be harder than continuous optimization. Learning the right sparsity mask is fundamentally a very discrete operation. So even just matching fully continuous dense models in optimization efficiency is likely to be difficult. Though perhaps we can get some hope from the fact that MoE is also similarly fundamentally discrete, and it works in practice (we can think of MoE as incurring some penalty from imperfect gating, which is more than offset by the systems benefits of not having to run all the experts on every forward pass). Also, the optimization problem gets harder when the backwards pass needs to be entirely sparsified computation (see appendix B).

- Dense matmuls are just fundamentally nicer to implement in hardware. Systolic arrays have nice predictable data flows that are very local. Sparse matmuls with the same number of flops nominally only need (up to a multiplicative factor) the same memory bandwidth as an equivalent dense matmul, but they need to be able to route data from any memory unit to any vector compute unit - the locality of dense matmuls means that the computation of each tile only requires a small slice of both input matrices, so we only need to load those slices into shared memory; on the other hand, because GPU-to-GPU transfers are way slower, when we op-shard matmuls, we replicate the data that is needed. Sparse matmuls would need either more replication within each compute die, or more all-to-all internal bandwidth. This means spending way more die space on huge crossbars and routing. This would cost a lot of die space, though thankfully, the crossbars consume much less power than actual compute, so perhaps this could match dense in energy efficiency and not make thermals worse.

It also seems very likely that once we create the interpretable GPT-1 (or 2, or 3) we will find that making everything unstructured sparse was overkill, and there are much more efficient pretraining constraints we can apply to models to 80/20 the interpretability. In general, a lot of my hope routes through learning things like this from the intermediate artifact (interpretable GPT-n).

To be clear, it doesn't seem literally impossible that with great effort, we could create custom hardware, and vastly improve the optimization algorithms, etc, such that weight-sparse models could be vaguely close in performance to weight-dense models. It's plausible that with better optimization the win from arbitrary connectivity patterns might offset the hardware difficulties, and I could be overlooking something that would make the cost less than I expect. But this would require immense effort and investment to merely match current models, so it seems quite unrealistic compared to learning something from interpretable GPT-3 that helps us understand GPT-5.

I'm also very excited about SAE/Transcoder based approaches! I think the big tradeoff is that our approach (circuit sparsity) is aiming for a full complete understanding at any cost, whereas Anthropic's Attribution Graph approach is more immediately applicable to frontier models, but gives handwavier circuits. It turns out "any cost" is really quite a lot of cost - we think this cost can be reduced a lot with further research, but it means our main results are on very small models, and the path to applying any of this to frontier models involves a lot more research risk. So if accepting a bit of handwaviness lets us immediately do useful things on frontier models, this seems like a worthwhile direction to explore.

See also some work we've done on scaling SAEs: https://arxiv.org/abs/2406.04093

There's been some work (e.g RASP - https://arxiv.org/abs/2106.06981) on taking logical computations and compiling them into transformer weights.

Mixture of experts sparsity is very different from weight sparsity. In a mixture of experts, all weights are nonzero, but only a small fraction get used on each input. On the other hand, weight sparsity means only very few weights are nonzero, but every weight is used on every input. Of course, the two techniques can also be combined.

For what it's worth, we think it's unfortunately quite unlikely that frontier models will ever be trained with extreme unstructured sparsity, even with custom sparsity optimized hardware. Our main hope is that understanding sub-frontier models can still help a lot with ensuring safety of frontier models; an interpretable GPT-3 would be a very valuable object to have. It may also be possible to adapt our method to only explaining very small but important subsets of the model.

you can always have an LVM layer for atomic snapshots

btrfs has eaten my data within the last decade. (not even because of the broken erasure coding, which I was careful to avoid!) not sure I'm willing to give it another chance. I'd much rather use zfs.

> Sentences like the opening line of the Declaration of Independence simply do not occur in conversation.

maybe not in your conversations

The GPT-4 pretraining set included chess games in PGN notation from 1800+ ELO players. I can't comment on any other models.

Nvidia SASS has a similar instruction too (LOP3.LUT)

I looked into the question of what counts as reasonable care and wrote up my conclusions here: https://www.lesswrong.com/posts/kBg5eoXvLxQYyxD6R/my-takes-o...

In domains like ML, people care way more about the half precision FLOPs than single precision.

Is there a better test than the one that is normally prescribed, to distinguish the false positive cases you describe?

The paper cites Anthropic's work extensively.

Note that we focus on random positive activations, which are less susceptible to interpretability illusions than top activations (but also look less impressive as a result). We also provide access to random uncherrypicked features, whereas Anthropic does not. We made these choices deliberately to give as accurate an impression of autoencoder feature quality as possible.

Also note that GPT-4 is a more powerful model than Sonnet, which makes it harder to train autoencoders with the same quality features.

The paper introduces substantial improvements over the methodology in the Anthropic SAE paper, and the research was done concurrently.

We were planning to release the paper around this time independent of the other events you mention.

I think it is still predominantly accurate to say that we have no idea how LLMs work. SAEs might eventually change that, but there's still a long way to go.

This project has been in the works for about a year. The initial commit to the public repo was not really closely related to this project, it was part of the release of the Transformer debugger, and the repo was just reused for this release.

Thankfully, there already exist evaluation tasks like that, and Eleuther actually has a project collecting a handful of them together; see https://github.com/EleutherAI/lm-evaluation-harness/

To date, EleutherAI as an "organization" (read: basically a Discord server) has not really attempted any kind of marketing. It has no PR dept, just individuals tweeting about the work that Eleuther does.

Could the title of this post be change to emphasize that the model sizes released were 1.3B and 2.7B? Something like "EleutherAI releases 1.3B and 2.7B parameter GPT-like language models". The current title implies that a full sized GPT-3 model is currently available, which is not the case.

edit: the title has been changed, seems good enough

My impression from reading the paper is most of the other optimizations (custom kernels, contiguous memory, checkpointing, etc) are orthogonal to the partitioning stuff. That seems to imply that ZeRO is model+pipeline parallel plus a bunch of miscellaneous bits. But they seem to emphasize that this isn't what their partitioning is, and that's the part that perplexes me the most. To be specific, I'd like someone to explain how their magical zero-redundancy data parallel (termed ZeRO-DP in the paper) works and how it's different from from model+pipeline parallel, and their paper is awfully sparse on that.

I have a reasonable amount of experience with distributed machine learning (and transformers in particular, too) and I have to 100% agree that this blog post (and even the ZeRO paper) is largely technobabble. I don't doubt that this might really work, but how it works is not elucidated very well, and I'm still not 100% sure I understand what they actually did.

For anyone who still thinks the blog post has substance: Saying that they partitioned the optimizer state, params, etc to have no redundancies is kind of "duh," sort of like saying "we solved the problem using coding and algorithms." It's obvious that we want to eliminate redundancies to maximize the effective VRAM; it's not like nobody thought of not having redundancies before. The problem is that in general, training models distributed is a weird balancing act between redundancy, network usage, compute, etc. The existing methods, model/pipeline/data parallel, gradient checkpoint/accum, etc all have their pros and cons. Unless ZeRO3 is doing something crazy, it has to be giving something up to get to zero redundancy, and knowing what that is would be very important.

If someone could ELI5 how ZeRO actually works, that would be nice.

Handling heterogenous (and potentially untrustworthy) systems also adds overhead, not to mention that buying hardware in bulk is cheaper, so it makes the most sense just to raise the money and buy the hardware.