> At what point does it become more feasible to rewrite your architecture or use less GPU

They are but it takes a lot of time.

Most of the big players - Google, Meta, OpenAI, Amazon, and Microsoft all are actively developing TPU/NPUs that would be used instead of the H100/A100's everyone is using for machine learning.

Google (tensorflow/jax), Meta(pytorch), Microsoft(onyx), Openai(triton) and Apple (mlx) each have software stacks for optimizing models for multiple platforms.

It takes a lot of time to develop the silicon and software stack. As a result everyone is using H100's in the interum until the hardware/software catches up. Google has been using their TPUs already.

There's other companies like Groq that are also developing NPU/TPU like devices.