A MySQL compatible database engine written in pure Go
github.com
github.com
For us this package is most important as the query engine that powers Dolt:
https://github.com/dolthub/dolt
We aren't the original authors but have contributed the vast majority of its code at this point. Here's the origin story if you're interested:
https://www.dolthub.com/blog/2020-05-04-adopting-go-mysql-se...
Dolt can do the same two directions of MySQL binlog replication, and also has its own native replication options:
> If you have an existing MySQL or MariaDB server, you can configure Dolt as a read-replica. As the Dolt read-replica consumes data changes from the primary server, it creates Dolt commits, giving you a read-replica with a versioned history of your data changes.
This is really cool.
Let us know if you try it out, we're always interested in feedback.
Not sure how GC works in Golang but if you see 20k writes/s sustained that's what I'd be nervous about. If every write is 4 kB I think it would be something like a quarter of a TB per hour, probably a full TB at edge due to HTTP overhead, so, yeah, a lot to handle on a single node.
Maybe there are performance tricks I don't know about that makes 20k sustained a breeze, I just know that I had to spend time tuning RAM usage and whatnot for peaks quite a bit earlier and already at that load planned for sharding the traffic.
- Don't use "mysql" in the name, this is a trademark of Oracle corporation and they can very easily sue you personally if they want to, especially since you're using it to develop a competing database product. Other products getting away with it doesn't mean they won't set their sights on you. This is just my suggestion and you can ignore it if you want to.
- Postgres wire/sql compatibility. Postgres is for some reason becoming the relational king so implementing some support sooner rather than later increases your projects relevance.
https://github.com/dolthub/doltgresql
Background and architecture discussion here
Not enticing enough to build a business around, due to it being that bit too different and the persistence layer being that bit too important. But the sort of thing that I'd love it if the mainstream DBs would adopt.
I didn't realise the engine was written in Go, and honestly the first place my mind wonders is to performance.
[1] https://docs.dolthub.com/architecture/storage-engine/prolly-...
We needed to build a custom storage engine to make querying and diffing work at scale:
https://docs.dolthub.com/architecture/storage-engine
It based on the work of Noms including the data structure they invented, Prolly Trees.
https://docs.dolthub.com/architecture/storage-engine/prolly-...
The default proxied database is dolt. I'm guessing this is extracted from dolt itself as that claims to be wire-compatible with mysql. Which all makes total sense.
We didn't extract go-mysql-server from Dolt. We found it sitting around as abandonware, adopted it, and used it to build Dolt's SQL engine on top of the existing storage engine and command line [2]. We decided to keep it a separate package, and implementation agnostic, in the hopes of getting contributions from other people building their own database implementations on top of it.
[1] https://github.com/dolthub/dolt [2] https://www.dolthub.com/blog/2020-05-04-adopting-go-mysql-se...
> No transaction support. Statements like START TRANSACTION, ROLLBACK, and COMMIT are no-ops.
> Non-performant index implementation. Indexed lookups and joins perform full table scans on the underlying tables.
I actually wonder if they support triggers, stored procedures etc.
The bundled in-memory database implementation is mostly for use in testing, for people who run against mysql in prod and want a fast compatible go library to test against.
For a production-ready database that uses this engine, see Dolt:
Also I suspect this project isn't for holding hundreds of GB of stuff in memory all the time, but I could be wrong.
Writing no-alloc is oftentimes done by reducing complexity and not doing "stupid" tricks that work against JIT and CoreLib features.
For databases specifically, .NET might actually be positioned very well with its low-level features (intrisics incl. SIMD, FFI, struct generics though not entirely low-level) and high-throughput GC.
Interesting example of this applied in practice is Garnet[0]/FASTER[1]. Keep in mind that its codebase still has many instances of un-idiomatic C# and you can do way better by further simplification, but it already does the job well enough.
Borrow checker based .drop/deallocation is very, very convenient for data with linear or otherwise trivial lifetime, but for complex cases you still end up paying with either your effort, Arc/Arc<Mutex, or both. Where it does help is knowing that your code is thread-safe, something Rust is unmatched at.
But otherwise, C# is a quite unique GC-based language since it offers full set of tools for low level control when you do need that. You don't have to fight GC because once you use e.g. struct generics for abstractions and stack allocated/pooled buffers or native memory for data neatly wrapped into spans, you get something close to C but in a much nicer looking and convenient package (GC is also an optimization since it acts like an arena allocator, and each individual object has less cost than pure reference counted approach).
Another language, albeit with small adoption, is D which too has GC when you want an escape hatch while offering features to compete with C++ (can't attest to its GC performance however).
In .NET, this does not seem to be a performance issue, in fact, it is used by almost all performance-sensitive code to match or outperform C++ when e.g. writing SIMD code.
There are roughly three ways to pass data by reference:
- Object references (which work the same way as in JVM, compressed pointers notwithstanding)
- Unsafe pointers
- Byref pointers
While object references don't need explanation, the way the last two work is important.
Unsafe pointers (T*) are plain C pointers and are ignored by GC. They can originate from unmanaged memory (FFI, NativeMemory.Alloc, stack, etc.) or objects (fixed statement and/or other unsafe means). On an occasion a pointer into object interior (e.g. byte* from byte[], or MyStruct* from MyStruct field in an object) is required, such object is pinned by setting the bit in object header which is coincidentally used by GC during mark phase (concurrent or otherwise) to indicate live objects. When an object is pinned in this way, GC will not move it during relocation and/or compaction GC phase (again, concurrent or otherwise, not every GC is stop-the-world). In fact, objects are moved when GC deems so to be profitable, either by moving to a heap belonging to a different generation or by performing heap compaction when a combination of heuristics prompts it to reduce fragmentation. Over the years, numerous improvements to GC have been made to reduce the cost of object pinning to the point that it is never a concern today. Part because of those improvements, part because of the next feature.
Byref pointers (ref T) are like regular unsafe pointers except they are specially tracked by GC to allow to point to object interiors without writing unsafe code or having to pin the object. You can still write unsafe code with them by doing "byref arithmetics", which CoreLib and other performance sensitive code does (I'm writing a UTF-8 string library which heavily relies on that for performance), and they also allow to point to arbitrary non-object memory like regular unsafe pointers do - stack, FFI, NativeMemory.Alloc, etc. They are what Span<T> is based on (internally it is ref T + int length) allowing to wrap arbitrary memory ranges in a slice-like type which can then be safely consumed by all standard library APIs (you can int.Parse a Span<byte> slice from stack-allocated buffer, FFI call or a byte[] array without any overhead). Byref pointers can also be pinned by being stored in a stack frame location for pinned addresses, which GC is aware of. For stack and unmanaged memory this is a no-op and for object memory this only matters during GC itself. Of course nothing is ever free in software engineering but the overhead of byrefs is considered to be insignificant compared to object pinning.
It is still possible to end up with fragmentation challenges with both the jvm and .net under specific circumstances- but there are also a lot of tunable parameters and so on to help address the problem without substantially modifying the program design to fit an under specified contract with the memory subsystem. In go there are few tunables, and an even less specified behavior of the gc and its interaction with application code at this level of concern.
Odds are if you need one written in Go then you’re requirements are somewhat different. For example the need to stub for testing.
On the JVM you have things like Cassandra, Elasticsearch, Kafka, etc. each of which offer performance and scale. There are lots more examples. As far as I know they don't do any of the things you mention; at least not a lot. And you can use memory mapped files on the JVM, which helps as well. Elasticsearch uses this a lot. And I imagine Kafka and Cassandra do similar things.
As for skillset, you definitely need to know what you are doing if you are going to write a database. But that would be true regardless of the language.
It is also true that the JVM and the GC are a bottleneck in what they are able to offer. Scylla and Redpanda's pitch is "we are like this essential piece of software, but without the JVM and GC".
Of course, having a database written in Go still has its pros and cons, so each to their own.
There are many valid reasons to pick other products but Elasticsearch is pretty good and fast at what it does. I've seen it ingest content at half a million documents per second. No stutters. Nothing. Just throw data at it and watch it keep up with that for an hour sustained (i.e. a bit over a billion documents). CPUs maxed out. This was about as fast as it went. We threw more data at it and it slowed down but it didn't die.
That data of course came from kafka being passed through a bunch of docker processes (Scala). All JVM based. Zero GC tuning needed. There was lots of other tuning we did. But the JVM wasn't a concern.
In Go, I’ve found that with a little bit of awareness, and a small bag of tricks, you can get very low allocations on hot paths (where they matter). This comes down to using sync.Pool and being clever with slices to avoid copying. That’s footgun-performance tradeoff that’s well worth it, and can get you really far quickly.
Then we can finally have multiple database engine support for WordPress and others.
To clarify, the wire protocol is the easy part, the semantic differences how each database does things is a whole other can of worms. Such emulation will never be 100%, quirks and all.
Somewhat relatedly, StarRocks is also MySQL compatible, written in Java and C++, but it's tackling OLAP use-cases. https://github.com/StarRocks/starrocks
But maybe this project is tackling a different angle. Vitess MySQL library is kind of hard to use. Maybe this can be used to build ORM-like abstraction layer?
We here at DoltHub use it to provide SQL to Dolt.
Given how important the DB layer is I would be careful to use something like this in production, but if it allows speeding up the test suite it could be really interesting.
TLDR; Because go-mysql-server existed.
https://www.dolthub.com/blog/2022-03-28-have-postgres-want-d...
We have a Postgres version of Dolt in the works called Doltgres.
https://github.com/dolthub/doltgresql
We might have a go-postgres-server package factored out eventually.
Could I swap storage engine with own key value storage e.g. rocksdb or similar?
... it would be a wild future if Vitess replaced the underlying MySQL engine with this (assuming the performance is good enough for Vitess).
if someone is considering running it, they're probably considering it against the actual thing. and I would think the main decision criteria is: _how much faster tho?_
We expect that it's faster than MySQL for small scale. Dolt is only 2x slower than MySQL and that includes disk access.
In dynamic language land, we tend to use real DBs for test runs. So having a faster DB wouldn't hurt!