You're assuming your app has to be served as one large chunk of stuff. There are many JS libraries for asynchronously loading additional content, and it is simple enough to use one of these (or implement your own) so that your app is served as one small bootstrap and a bunch of additional content that is loaded as needed.

You say "static sites can spit back pages in like 200ms (across the network!!)". This time is entirely dominated by network latency. There is no way you're getting data to the client in 200ms if they're based in Australia and you're in, say, east coast US. With the client-side model you can push your app's bootstrap to a CDN to get ~200ms latency just about anywhere. You can render the UI and show a "Loading data" spinner to hide the latency of the initial request to your application server. This gives the user the appearance of a snappy site even though network latency might be 1s or more. You can't do this if you go the server-side route.