Hacker News Reader

Comment by koeng | Hacker News Reader

Very cool! I’ve been looking at using SQLite-vss for fast vector sorting, but it requires CGO. I imagine if I do WASM compilation directly from C, I might be able to get around using CGO. Do you have any idea able the feasibility of that using your package? Thanks!

View on HN

One problem with WASM is that, at this point (2023), the only easy way of building that, is to statically link everything C into one WASM blob (so you can't use my blob).

But you can try to use my build scripts, and tweak my code, to build SQLite+sqlite-vss+Faiss into one large WASM. If you use my patches/imports/exports/config for SQLite, my binding will gladly take your WASM blob, without modification to the Go code.

All you need to do is set this global variable (and not import package embed, which embeds my WASM and sets this variable for you at init): https://pkg.go.dev/github.com/ncruces/go-sqlite3#pkg-variabl...

Ping me on GitHub if you need help.

Vector search is on my meters long todo list, but I'd rather do it in Go if at all possible.

How would you do it in Go? It is my understanding that FAISS is integrated into the SQL database itself (using embedding in the table), so writing it in Go wouldn't integrate it in the same way.

> But you can try to use my build scripts, and tweak my code, to build SQLite+sqlite-vss+Faiss into one large WASM

That makes perfect sense to me! I'm looking into your build scripts right now. I'm pretty proficient in Go, but haven't touched wasm or C much, so. I may reach out with more questions.

You need a virtual table (same as sqlite-vss) and a library that does cosine similarity indexing (Faiss here)… which I know little about.

My plan was to evaluate Go for cosine similarity indexing, and if I found something promising, port the virtual table.

Reply on news.ycombinator.com