Picture the rendering process like a conveyor belt factory. This is a bit of a simplified model, GPUs are incredibly complex and things aren't quite this straightforward, but it's good enough way of reasoning about how they operate by imagining something like Factorio.
First you load up a scene, and then it goes into one machine that does the first part of the rendering, then the next machine does the next part, and so on and so forth, until it arrives at the other end and is drawn on the screen.
It's easy to see you can start working on the next frame while the first frame is in the second machine and that's probably in general a good idea.
Reducing how many items are on the belt doesn't reduce the time it takes to go from start to finish (i.e. the latency), it still has to go through the same steps. Loading up more items on the belt at the same (probably) improves the stability of the framerate, but again it does nothing for latency. If you load up too much on the belt may cause problems, but such a condition is easy enough to determine and avoid programmatically. The frames still take the same time to go from start to finish.
Bottom line is that the time between frames and the time to render a frame are disconnected, if they seem connected it's because getting a better GPU increases them both. But you could (hypothetically) render at 120 Hz refresh rate with a 30 second rendering latency.