Hacker News Reader

What I've done for most of my research projects is just pickle a PyTorch dataset object that contains all my embeddings. The pkl file can then just be uploaded anywhere and becomes plug and play with any Torch model.

What advantages would this bring for a user like me? I guess it might make more sense for people working closer to production?

View on HN

Exactly, once you have latency-sensitive applications that read-write embeddings, then you have to rethink your vector storage system. Furthermore, when your dataset grows to millions of datapoints, having a system that scales according avoids downtime and costly query lookup operations.

How would you handle an online ML application where the set of embeddings is changing such as an image recognition app where images are constantly being added and need to be deduplicated?

Reply on news.ycombinator.com