How much opportunity is there for this level of datacenter design to reduce power consumption? What are the trade-offs?
Having full cross-sectional bandwidth between any pair of hosts means the bin packing problem is a lot easier. You don't need to (say) make sure your map reduce job is scheduled with one shard per rack because racks only have so much bandwidth. Any host on any rack will do. You can forget racks even exist.
This makes overall utilization of clusters more efficient (tighter bin packing), and the corollary is you don't need as many clusters and machines. (Not that has ever stopped Google from building more :)