Mechanistic interpretability is a field that studies exactly this.

Please, go on. How would you do it, specifically?

They do, there was a paper that claimed a major breakthrough in that field just a few days ago. See the "Neural networks are clean" discussion:

https://news.ycombinator.com/item?id=50105647

Are you saying no one is trying to do exactly that?