Hacker News Reader

Comment by jchw | Hacker News Reader

Hey, that's awesome. I think if I wanted a CGo-less solution, I would've probably picked the modernc variation, but honestly this looks very polished + the modernc version is known to be slow in some cases (though, the benchmarks actually made it look alright, realistically.)

One question I have is, is there a strategy to stop the performance from accidentally regressing in the future? Seems like it might not be a bad idea to include some benchmarks in the repository if you don't already.

Also, personally, I am interested in the nitty gritty of why it was slow and how you fixed it, so if you write it anywhere in this thread at least I will definitely read it through.

View on HN

> I would've probably picked the modernc variation

Heads up about the modernc library, it has been stuck on an old version of sqlite for several months [1]. It seems like maintainer time is the limiting factor [2]. There has been a call to arms on that issue page, the maintainer is looking for help, but it looks like not much has arrived. It seems like it might trace back to blockers in the C-to-Go compiler.

It's a major undertaking and a very impressive piece of work, but I'm not surprised it's a struggle when big roadblocks get hit. I hope they find a way to progress, but I'm very relieved to be seeing some CGo-free alternatives like ncruces/go-sqlite3 emerging. I'm going to give it a try for sure and see if I can live with the compromises.

Squinn-go looks very compelling too, but I don't like that it requires the squinn binary to already be installed on a user's machine, I think that gives with one hand and takes with the other: sure, I get to avoid CGo, but I also lose the turnkey, single-command install, static build benefits Go brings out of the box.

Seconding the point about nitty gritty, I'd read it for sure too!

  [1]: https://gitlab.com/cznic/sqlite/-/issues/154
  [2]: https://gitlab.com/cznic/sqlite/-/issues/164

The nitty gritty is here, mostly: https://news.ycombinator.com/item?id=38652566

About my driver, the major compromise to be aware of is that WAL mode support is limited. No tiptoeing here: I call that caveat out in the front page.

My driver reimplements the entire SQLite OS Interface (VFS) in pure Go. This makes it very portable, but it's also a risk: it's definitely not as extensively tested as SQLite's. And the WASM sandboxing model makes shared memory a hard problem to solve, which prevents WAL mode from working.

I have some ideas for work arounds, but: they're hard to implement, and hard to "prove" correct. Idea 1 is a VFS for WAL where SQLite believes it's using memory journal mode, but the wrapper uses the standard WAL format to persist disk. It's a lot of work, but standard SQLite tools can recover a crashed DB. Idea 2 is for the WASM host to simulate shared memory through copying. This should work (there are explicit locks and memory barriers to use as copy points), but it's hard to "prove" correct (without bothering SQLite developers to death with details).

I had some hope that the work being done in libsql to virtualize the WAL would help, but their API for this is just terrible. They simply extracted and exposed internal SQLite stuff, but have no documentation, no usage examples, or even any public VFS implementation using these APIs. And I honestly don't trust their ability to maintain a quality fork. An API that's not dogfooded nor documented is not an API.

Thanks for the info, I think I'll be using this SQLite library next time this comes up, then. I do think the modernc library is really cool, but there are some aspects of it that I already disliked, so I'm more than willing to see if the web assembly approach is better (it looks surprisingly tractable.)

One thing I tried to make sure, to avoid the pitfall modernc is having, is to make sure building "the WASM BLOB" is easily reproducible with widely available tools:

https://github.com/ncruces/go-sqlite3/blob/main/.github/work...

I do apply some light patches to SQLite, but so far they've always cleanly applied, and I can produce a new release within hours of being notified of SQLite releases.

The nitty gritty is in another answer.

The commits implementing the new function cache are these two: https://github.com/ncruces/go-sqlite3/compare/0b093b7c...986...

Creating good, representative, benchmarks is hard. I use speedtest1 [1] but this tests WASM/SQLite and my VFSes, not the Go driver that wraps it.

I already forked this benchmark and will run it regularly, but I may also adapt it into a Go benchmark (easier to run, benchstat compatible, etc).

[1] https://sqlite.org/src/file/test/speedtest1.c

Reply on news.ycombinator.com