SpaceX does not have cross links. They will need to have several hops up and back to different satellites before the signal arrives in many cases. It's not unlikely that when processing time is included, it's well over 100ms.
If conventional geosynchronous satellite internet can achieve real world latencies between 600ms and 1200ms, can you describe a scenario where having a hop time that is 60x faster could not manage to improve this by a factor of 12x?
Think of it this way: traditional geo satellite has a theoretical latency of 430-500ms in most cases. That means there's AT LEAST another 100ms purely due to processing delay and other hops, but typically much more.