Architecturally, LLMs are very simple compared to many software projects.

The crux of their behavior comes from their learned weights which are gigabytes and can cost millions to obtain via training.

The magic is in the billions of learned weights (~synapses). This is just the scaffolding that runs them.

The magic is the structure of the model, and the real magic is the billions of weights.