Such a great question! I think vector search is designed to solve a specific task, like finding K nearest neighbors for your given vector embedding. If you have a few hundred thousand, a million, it might work fine with a linear scan. Again, this depends on your use case. But if you have tens of millions, hundreds of millions or even a billion of vectors, you will need something more robust to store, index and search over this data.

An example in NLP world is BERT-like NNs, that allow you to embed your text into a dense vector representation.