> Or is there a theoretical maximum a single chip can do?

There's a point where you won't be able to pack more transistors on a chip, but if Cerebras' work is an indication, chips themselves can get quite large.

But remember these chips are not designed with desktop users in mind. These chips are more similar to the ones inside very large computers that are built to host hundreds of simultaneous virtual machines on behalf of hundreds of users.

Sadly, we won't see desktops based on these unless we custom order them.