I think in most cases it will be more than 100ms. Much more than that. Just latency to basecamphq.com is 350ms from here (Europe.)
1. Controlling how long page render takes at the server is under the developer control.
2. The size of network load is under developer control.
3. Client side optimization are under developer control.
OP is saying 1 is low; 2 is low(not much difference between html load and json load); and 3 is low as there isn't much going on at client side.
Other than that:
1. User network is slow, hence leading to large load times.
Go bake potatoes - can't help.
2. User is located at a distant location from the server.
If the user is part of a market which the company sees as profitable, the server can be mirrored.
Or else you can go bake potatoes.