Hacker News Reader

Comment by eatonphil | Hacker News Reader

It's a really neat project. I did some simple benchmarks of it and it was between 10% slower to twice as slow. Which depending on your use case isn't that bad!

https://datastation.multiprocess.io/blog/2022-05-12-sqlite-i...

View on HN

at segment we benchmarked https://github.com/segmentio/ctlstore against this driver. We saw about a 50% hit to read performance, so we didn't move forward with it, but the improvements in service build times were really appealing.

Really tho, how much is build time vs runtime? Maybe is just work small projects but I rarely care about build time. Am I missing something?

Honestly, on a non-toy project, build times with cgo are _brutal_. I agree with you usually, but when a build time on a beefy computer switches from under a second to >1min you notice it.

Linters and IDEs get slow when they check for errors, tests run slow, feedback drags, and all your workflows that took advantage of Go's fast compile times are now long enough that your flow and mental context disappear.

I'm way more lenient with other languages since the tooling and ecosystem are built around long build times. Your workflows compensate. But Go's tooling and ecosystem assume it compiles fast and treat things more like a scripting language. When that expectation is violated it hurts and everything feels like it's broken.

In my experience, encapsulating the access to sqlite in a go package helps a lot with avoiding recompilation of the c source, which indeed is brutally slow. It acutally seems to be way slower than compiling with gcc from the command line. Anyone knows why this is the case?

Why would you have to recompile sqlite every time?

I guess you just need to compile the .a once and then just reuse it?

If you're rebuilding it every single time, your build is set up wrong.

If you have a lot of people and a lot of different builds it can become increasingly significant. It wasn't in our case, but it felt like it could have become so. And a lot of services wouldn't really notice too much if the occasional read from disk was slower, given that the nature of the data was control data.

I raised this question on twitter awhile back and the response that I got was that (for read-only tasks) the connection string might ameliorate this. I haven't been able to test it, though:

https://twitter.com/frioux/status/1483235674228596739

I’d go so far as to say it’s completely acceptable for 90+% of use cases.

What makes it slower?

It’s hard for Go to beat C in a straight benchmark - Go is garbage collected and allocations often have extra overhead due to the memory layout of interfaces.

Also, Go barely optimizes code, that’s how it compiles as fast.

It is a matter of skill, Go can do manual memory management and stack allocation just as well.

In addition, a lot of optimizations (like loop unrolling) are left on the table to preserve Go's fast compile times.

I was surprised with the announcement that Go's standard compiler had register allocation for function arguments[0] added to it. This seemed like table stakes for a very long time but it really illustrated just how Go's compiler is much, much simpler than most in many areas.

On one hand it makes you wonder how much could be squeezed out of Go and how many basic things they're still leaving out of the compiler. On the other it tells part of the story of why Go compiles fast with the backdrop that despite this lack of work by the compiler people use Go productively all day and deliver services they are happy with from a performance perspective.

[0] - https://go.dev/doc/go1.17#compiler (2021-08)

It goes to show the impact of optimizing from top-down instead of bottom-up can have. Go was designed with performance in mind ("mechanical sympathy") as opposed to most other GC languages for which performance was an afterthought.

If you get high-level language constructs right, you'll get great performance by default, even with a simple compiler:

- structs instead of objects

- value types

- exposed pointers

- straightforward allocation semantics

Most compilers out there are busy fighting bad language designs. There's only so much a compiler can do when faced with fat objects hidden behind a spider web of pointers. So they have to do these microoptimizations to claw back at least some performance.

I wonder if they'll ever switch to a "normal = fast compile, low optimisation; release = slow compile, high optimisation" model like Rust. Although I suppose other languages already have the "high optimisation" parts covered (rust, julia, python+numpy, etc.) which might make it a fool's errand.

Gccgo is the answer here.

Gccgo at its core may be fast but when it comes to how the go frontend interpret it to gcc ir that pales in comparison than just straight up optimizating the Go compiler

There’ll more opportunities for more optimizations since they move to SSA ir since 1.5 and in turn compile time will definitely go down

However, it is absolutely plausible for Go-calling-Go to beat Go-calling-C-via-CGo, since CGo has plenty of overhead.

Not only Go compiler is not as good at optimisation as C/C++/Rust/Zig/D compilers, but also Go FFI to C has significant overhead, which means there is often not much point in using low-level Linux APIs for performance advantage. Same problem like with Java. And in a database system you quite likely want a fine degree of control with non-standard stuff.

Other than what the other responders said, since it uses a C to Go compiler, I'd imagine the Go code that's generated isn't as fast as handwritten Go code could be.

C -> GPT -> Go -> Edit -> commit?

-> fail to compile / crash at startup

My guess is that over the years many optimizations have gone into SQLite that would take a lot of effort to catch up to.

Reply on news.ycombinator.com