Hacker News Reader

How does this compare to or improve on applying something like SHAP[0][1] on a model? The idea in the first line that "we currently don't understand how to make sense of the neural activity within language models." is..straight up false?

[0] https://github.com/shap/shap

[1] https://en.wikipedia.org/wiki/Shapley_value#In_machine_learn...

View on HN

SHAP is pretty separate IMO. Shapley analysis is really a game theoretical methodology that is model agnostic and is only about determining how individual sections of the input contribute to a given prediction, not about how the model actually works internally to produce an output.

As long as you have a callable black box, you can compute Shapley values (or approximations); it does not speak to how or why the model actually works internally.

Reply on news.ycombinator.com