Hacker News Reader

Comment by dekhn | Hacker News Reader

Google absolutely runs workloads that need supercomputer tight scaling. That's why they built TPUs with ICI networks.

You got me on a technicality, but nothing of practicality.

This isn't exposed to the larger Google, nor to customers. Google itself as an organism doesn't HPC. It is an ad company that runs ad company systems.

Why are there no top-k super computers running on Google?

ICI is basically nvlink, not like IB.

no, ICI is a fiber optic (or electrical) network between TPUs and it doesn't have any switch functionality. I used to work for Google (on TPUs and ICI). Anyway, the comparisons don't matter that much. The long and short of it is that Google built their own supercomputers (finally) that can do allreduce and alltoall patterns, as well as low-latency broadcast.

medium isn't the point, and duh, torus networks are distributed-switching.

Do you have a point? There's nothing technically wrong with saying that TPUs with ICI are a supercomputer- they are very similar in design to the T3E I used decades ago.

Please make your point instead of trying to negate whatt I said.

Medium matters a lot- because supercomputers are fundamentally constrained by physics in terms of power dissipation, transistor density, and speed of light, all-optical networks have slightly lower latencies than electrical, and also let you build larger systems (longer cables).

what is ICI networks?

ICI is interchip interconnect in TPUv4 pods.

You can read more on paper that my collegues in Google platforms recently published https://arxiv.org/abs/2304.01433 https://cloud.google.com/blog/topics/systems/tpu-v4-enables-...

I am so happy this paper is finally out :-) I led the SRE work during their NPI. It's a pain when you have to wait 3 years to discuss what you worked on.

They are based on OCS, which powers Google datacenter networks https://arxiv.org/abs/2208.10041

https://dl.acm.org/doi/pdf/10.1145/3360307 Very fast (hundreds of gigabits per connection) networks that attach TPUs on the same board, as well as between boards. It's wired up point to point, forming a logical grid (technically a 2D or 3D torus with wraparound links).

Reply on news.ycombinator.com