> As far as I'm concerned the gold standard of understanding something is being able to construct a program that replicates it which is exactly what we can do with LLMs.

But we actually can't! We can build a program that can build a program that is the LLM, which is not the same! I'd argue that you're right insofar as training is concerned. We understand training very well. But the actual model, how it operates, what it actually knows, we don't know how to build that, we don't know what weights to put where.