With people talking about pgvectors current scaling issues, one thing I'm not sure about is whether it's if the Postgres DB table simply contains a lot of vectors - e.g. 500k vectors - or if you are searching over 500k vectors?

E.g. If the DB table had 500k vectors, but you were pre-filtering by WHERE client_id = X (returning only 200 rows) then with an AND <embedding search> (returning only 6 rows) - would this still have the same performance issue?

Or is it literally if the embedding search is over 500k rows?