https://github.com/rqlite/rqlite/blob/master/auto/backup/sum*
It took a few minutes, might have taken me 1+ hour of some research, and mechanically typing it all in.
5,215 karma · joined April 3, 2014
https://www.philipotoole.com
[1] https://www.rqlite.io
https://github.com/rqlite/rqlite/blob/master/auto/backup/sum*
It took a few minutes, might have taken me 1+ hour of some research, and mechanically typing it all in.
Maybe if it continues I'll do as you suggest.
"Why would I use this, versus some other distributed database?
rqlite is very simple to deploy, run, and manage – in fact, simplicity-of-operation is a key design goal. It’s also lightweight and easy to query. It’s a single binary you can drop anywhere on a machine, and just start it, which makes it very convenient. It takes literally seconds to configure and form a cluster, which provides you with fault-tolerance and high-availability. With rqlite you have complete control over your database infrastructure, and the data it stores.
That said, it’s always possible it’s too simple for your needs."
[1]: https://rqlite.io/docs/faq/#why-would-i-use-this-versus-some...
Yes, you still need the data at every node. rqlite is not about edge-compute specifically (though many solutions have come along after rqlite was created to address that need, I guess I should have responded to that demand much earlier instead :-)). One of my primary goals with rqlite has always been ease-of-deployment and ease-of-use. I wanted to build a quality distributed database that was both really reliable and super easy-to-use.
Check out the FAQ, which might answer some more of your questions: https://rqlite.io/docs/faq/
The GPT transcript is spoken by someone with an Irish accent (sounds like it's from Northern Ireland). It's something else to hear it.
Whether it’s worth the cost is a decision each company makes. Also, you don’t need to keep the log forever. Max of a few weeks retention would be common.
https://rqlite.io/docs/faq/#how-can-i-change-my-multi-node-c...
https://rqlite.io/docs/clustering/general-guidelines/#recove...
While this talks about "dealing with failure" using the config file described above basically allows you to force the cluster into any configuration you want (including a single-node cluster).
However if you create a 3-node cluster (or add 2 nodes to an existing single-node cluster) then at least a quorum of nodes (N/2 +1) must be up and running for the cluster to be available. That's the way Raft works -- Cockroach, etcd, Consul -- all of them will work the same way.
So in your case a 3-node cluster with 2 nodes subsequently going down (or deliberately taken offline) is a cluster which is offline.
There is extensive documentation available on clustering, which you should review:
From the Litestream page: https://litestream.io/alternatives/#raft-based-consensus
https://www.philipotoole.com/9-years-of-open-source-database...
https://www.philipotoole.com/replicating-sqlite-using-raft-c...
https://www.philipotoole.com/rqlite-v3-0-1-globally-replicat...
The post makes a fairly big point of the shortcomings of statement-based replication (like what rqlite does).
What they state is correct -- non-deterministic SQL will be a problem, but it's not a difficult problem to solve. Just parse and rewrite the SQL statement before sending it to Raft. It's exactly what rqlite does[2] (though there is more to do). However they claim that there are two concerns with this approach.
>This solution introduces a runtime delay because each SQL statement must be parsed and analyzed.
The delay is insignificant compared with the delay introduced by Raft consensus. This is not a valid concern. I'm surprised they even mentioned it.
>More importantly, it increases the difficulty of staying up-to-date with the new versions of the underline SQL engine (SQLite in this specific case) because the custom parser must be updated each time a new SQL function/feature is introduced.
Theoretically somewhat true, but not true in practice. Any competent programmer can update any parsing code in a matter of hours. Also just because a new feature is introduced does not mean that the parser needs to be rewritten. That's because SQLite will not introduce breaking changes into their SQL specification. Any parser needs to be updated only if you want to support any new, non-deterministic feature.
I will say that one misconception people often have about rqlite is that it's a drop-in replacement for SQLite. You see that a bit even in the comments section here. That was never its goal. Its goal is a lightweight and super-simple to use distributed relational database. Using SQLite as the database engine was the perfect way to meet that requirement.
[2] https://rqlite.io/docs/faq/#is-it-a-drop-in-replacement-for-...
The Effective Executive by Peter Drucker.
[1] https://books.google.com/books/about/High_Output_Management....
rqlite offers horizontal and vertical real scalability, but only vertical for writes. See https://rqlite.io/docs/guides/performance/
I'm not developing libraries, I'm developing an entire RDBMS. In my experience -- and this is broader than rqlite -- integration and end-to-end tests seem like they are great - at the start. But as you rely more and more on them they become costly to maintain and really hard to debug. A single test failure (often due to a flaky, timing-sensitive issue) means wading through layers and layers of code to identify the issue.
Overly relying on integration and end-to-end testing (note I said over-reliance, there is absolutely a need for them) becomes exponentially more costly over time (measured in development velocity and time to debug) as the test suite grows. If you find you're having difficulty identifying a place for them it may that you're not decomposing your software properly in the first place. All this is probably manageable if you're a solo developer, but when a team is building the software it can become really painful.
For more details see the talk I gave to CMU[1] on my testing strategy.
The fundamental cause AFAIK was a SQLite connection was attempting to make a state transition (from one type of locking state to another) which shouldn't be allowed under certain circumstances, but the implementation didn't actually enforce this rule. So the added test really does test the root cause.
This bug can affect anybody using an in-memory version of a SQLite database. That was the point of writing the C unit test.