The two fast.ai courses end up implementing Stable Diffusion from scratch if that’s the sort of thing you are looking for.