I'd love to know the answer here too!
I've ran a few tests on pg and retrieving 100 random indices from a billion-scale table -- without vectors, just a vanilla table with an int64 primary key -- easily took 700ms on beefy GCP instances. And that was without a vector index.
Entirely possibly my take was too cursory, would love to know what latencies you're getting bryan0!