>Why isn't there a framework that does all this automatically for you?

Check MosaicML if it might help in your case. I haven’t tried myself but they’ve most customizations and speed up optimizations I came across in the recent times

https://www.mosaicml.com/blog/supercharge-training-composer

Also worth checking out their “training from scratch” blog posts.

Training StableDiffusion: https://www.mosaicml.com/blog/training-stable-diffusion-from...

Training GPT-3: https://www.mosaicml.com/blog/billion-parameter-gpt-training...