Can we reverse engineer transformer models into human-understandable programs?

transformer-circuits.pub

33 pointsapsec1124 comments

Still reading, but an early highlight for me is their rewriting of the attention mechanism in terms of tensor products. (Section “Attention Heads as Information Movement”) I always found the original formulation in terms of query/key/value/output matrices quite hard to operationally make sense of.

It’s extremely exciting to see more “mechanistic interpretability”/“circuit analysis”-style work on ML architectures that are not convolutional neural networks!

If they are human understandable then maybe they can be written that way in the first place?

The papers states that human understandable and computationally efficient are often very opposed goals.

Yes but in other areas we compile human understandable to computationally efficient.

Reply on news.ycombinator.com