If they were to release a design with the same architecture, but with all compute larger than f16 completely gone, they could probably get identical performance from a chip that is significantly smaller. I think both AMD and Nvidia will be releasing this kind of design (with more tweaks I'd guess) in the future.
And I definitely see some future for FP16/BF16 or INT8 devices for inference, but I don’t know how widespread such devices will become for training. For many types of models, they simply won’t converge if they don’t at least use a mixed precision scheme. And there are certain problems where even for inference you get a significant benefit from using FP32 - for instance, I work on building DL surrogate models for physical simulations. We get better results with the increased dynamic range and precision.
On the other hand, work being done in the big three DL packages for making distributed training easier has been quite nice. I know the least about TF, but their dtensor looks promising. JAX’s entire distributed paradigm using pmap/xmap etc make certain classes of models very easy to distribute. The one I’m following most closely though is Pytorch and their sharded tensor, and it looks like they’re planning on implementing native distributed ops powered by their RPC framework, which should make full tensor- and model-parallelism significantly easier.