If you connect to a satellite at an oblique angle, it is also much further away. A satellite in a 500km orbit is 500km away from you when it's directly above you. But when it's on the horizon, it can be more like 2500km away.
This adds a little bit of latency (maybe 10-20 ms). It also hurts the signal, since going through atmosphere attenuates signals more than going through vacuum. Especially for Ka band signals, which particularly weakened by water vapor.
On top of that, SpaceX needs to have redundant coverage over an area. You don't want your customers' internet to stop working because a bit got flipped on one of your satellites or a thruster blew up unexpectedly. Also, a single satellite may be able to handle Alaska or Siberia but have bandwidth issues handling the state of California.
I am genuinely curious how you arrived at that number, FWIW – I don't know how to go about calculating something like that.
If you run the calculation the other way around, every satellite has to service a ground patch with about 380 miles radius. And your antenna has to swivel to a point that might be more than 60 degrees away from zenith. So every obstruction of more than 30 degrees above the horizon will lead to at least intermittent disconnects.
With a minimal satellite system, your tracking needs to follow satellites all the way across the sky, and probably also needs to be able to rapidly make large changes in where it is aimed.
With a very large number of satellites, a ground station might be able to get by with only having to be able to track within a small part of the sky.
If you have enough that there are many of these small areas that always have a satellite in them distributed all across the sky, then even places that have a lot of buildings or trees blocking much of the sky should still have some patch they can view that works.
Throw in realistic angle limitations (not horizon) and coverage per satellite rapidly decreases