What might be needed from NN accelerator designers is not so much perf per watt as fast links between chips, between chips and memory, and between nodes in a cluster, so that a large distributed cluster of chips, each with its own memory, would appear to a user as a single chip with a huge amount of memory. Imagine 10TB/s links everywhere in such a cluster - not only training a large model would be vastly simpler, it would also be a lot more efficient (e.g. no need for data parallel model replication). In theory you could have a truly unified and shared memory space, with little need for synchronization.