The section of the talk at the timestamp above is titled "Should I use Raft"?
5,215 karma · joined April 3, 2014
https://www.philipotoole.com
[1] https://www.rqlite.io
The section of the talk at the timestamp above is titled "Should I use Raft"?
You can enable BasicAuth on the HTTP endpoints. You can also require that any client presenting BasicAuth credentials has the right permission for the operation the client is trying to perform. Finally you can also enable Mutual TLS, requiring that any client that connects must first present a cert that is signed by an acceptable CA.
For full details on security check out https://rqlite.io/docs/guides/security/
https://github.com/wildarch/jepsen.rqlite/blob/main/doc/blog...
It's still in active use by them. https://www.textgroove.com/ is another company that uses it, though I know less about their use case.
https://github.com/rqlite/rqlite/blob/master/CHANGELOG.md#10...
I really admire the testing that the SQLite team does[1], and the way it allows them to stand behind the statements they make about quality. It's inspiring.
IMO there have only been two really big improvements to software development relative to when I started programming professionally 25 years ago: 1) code reviews becoming mainstream, and 2) unit testing. (Perhaps Gen AI will be the third). I believe extensive testing is the only reason that rqlite continues to be developed to this day. It's not just that it helps keep the quality high, it's a key design guide. If a new module cannot be unit tested during development, in a straightforward manner, it's a strong sign one's decomposition of the problem is wrong.
"But if you will use it, keep in mind that [InfluxDB] uses Bolt as its data backend."
Simply not true. The author seems to have confused the storage that Raft consensus uses for metadata with that used for the time series data. InfluxDB has its own custom data storage layer for time series data, and has had so for many years. A simple glance at the InfluxDB docs would make this clear.
(I was once part of the core database team at InfluxDB and have edited my comment for clarity.)
That said, I do think it depends on what you consider important, and what your experience has been in the past. I used to value simplicity above all, so reducing the number of moving pieces was important to my designs. For the purpose of this discussion let's count a service as a single moving piece.
But over time I've decided that I also value reliability. Operators don't necessarily want simplicity. What they want is reliability and ease-of-use. Simplicity sometimes helps you get there, but not always.
So, yes, rqlite means another service. But I put a lot of emphasis on reliability when it comes to rqlite, and ease-of-operation. Because often when folks want something "simple" what they really want is "something that just works, works really well, and which I don't have to think about". SQLite certainly meets that requirement, that is true.
That said, it may not be practical to have hundreds of read-only nodes, but for moderate-size needs, should work fine.
Disclaimer: I'm the creator of rqlite.
I originally provided these musl-based builds so I could provide rqlite Docker images based on Alpine[1]. But now the Docker release process simply builds rqlite from the source during the image-creation process[2].
[1] https://www.philipotoole.com/building-a-highly-available-sea...
Disclaimer: I'm the creator of rqlite, and it's not the only piece of software to make SQLite available over the network.
Of course, it's easy for me to do this. I am not developing the database for commercial reasons, so can just say "no" to an idea if I want. That said, I've found that many ideas which didn't seem interesting to me when an end-user first proposed them become compelling once I think more about the operational pain (and it's almost always operational) they are experiencing.
Automatic backups to S3[1] was such a feature. I was sceptical -- "just run a script, call the backup API, and upload yourself" was my attitude. But the built-in support has become popular.
[1] https://www.philipotoole.com/adding-automatic-s3-backups-to-...
Hugely important, and I would add "backup-and-restore" to that list. At risk of sounding conceited, ease of use is a primary goal of rqlite[1] -- because in the real world databases must be operated[2]. I never add a feature if it's going to measurably decrease how easy it is to operate the database.
[2] https://docs.google.com/presentation/d/1Q8lQgCaODlecHa2hS-Oe...
Disclaimer: I'm the creator of rqlite.
I mean, yes, the more disk IO rqlite has to make to more write performance will be affected. However the advantages of running with an on-disk SQLite database are worth it I believe. In addition rqlite supports storing the SQLite database file on a memory-backed filed system if users really want that[1]. That can help squeeze more write throughput out of rqlite.
>My understanding is that rqlite Raft entries are mostly SQL statements (is that right?).
That's right, rqlite does statement-based replication, though I'm currently looking into extending it so it also does changeset[2] replication where it makes sense.
[1] https://rqlite.io/docs/guides/performance/#use-a-memory-back...
That's the basic design that rqlite[1] had for its first ~7 years. :-) But rqlite moved to on-disk SQLite, since with WAL mode, and with 'PRAGMA synchronous=OFF' [2], it is about as fast as writing to RAM. Or at least close enough, and I avoid all the limitations that come with :memory: SQLite databases (max size of 2GB being one). I should have just used on-disk mode from the start, but only now know better.
(I'm guessing you may know some of this because rqlite uses the same Raft library [3] as Nomad.)
As for the upgrade issue you mention, yes, it's real. Do you find it in the field much with Nomad? I've managed to introduce new Raft Entry types very infrequently during rqlite's 10-years of development, only once did someone hit it in the field with rqlite. Of course, one way to deal with it is to release a version of one's software first that understands the new types but doesn't ever write the new types. And once that version is fully deployed, upgrade to the version that actually writes new types too. I've never bothered to do this in practise however, and it requires discipline on the part of the end-users too.
[2] This might sound dangerous but in the current design of rqlite, the underlying SQLite database is completely rebuilt from the Raft log on startup (which is fsync'ed on every write). So any corruption of the SQLite database due power loss, etc is moot since the SQLite database is not the authoritative store of data in rqlite.
[1] https://rqlite.io/docs/clustering/general-guidelines/#cluste...
[2] https://rqlite.io/docs/faq/#can-any-node-execute-a-write-req...
If you need to ensure that a given API request (which can contain multiple SQL statements) is atomically processed (all SQL statements succeed or none do) that is supported however [1]. That's why I think of rqlite as closer to the kind of use cases that etcd and Consul support, rather than something like Postgres -- though some people have replaced their use of Postgres with rqlite! [2]
You may wish to read this:
https://github.com/wildarch/jepsen.rqlite/blob/main/doc/blog...
rqlite -- to the best of my knowledge and as a result of extensive testing -- offers strict linearizability due to its use of the Raft protocol. Each write request to rqlite is atomic because it's encapsulated in a single Raft log entry -- this is distinct from the other form of transactions offered by rqlite[1], but that second form of transaction functionality has zero effect on the guarantees offered by Raft and rqlite (they are completely different things, operating at different levels in the design). If you know otherwise I'd very much like to know precisely why and how.
"Evidence Dump Fallacy." This fallacy occurs when a person claims that a certain proposition is true but, instead of providing clear and specific evidence to support the claim, directs the questioner to a large amount of information, asserting that the evidence is contained within.
>One of the problems is if you're working with developers, the log replication contents is the queries, instead of the sqlite WAL like in dqlite.
I think you mean rqlite does "statement-based replication"? Yes, that is correct, it has its drawbacks, and is clearly called out in the docs[1].
>Another issue is if I want to architect a system around rqlite, it wont be "consistent" with rqlite alone. The client must operate the transaction and get feedback from the system, which you can not do with an HTTP API the way you've implemented it.
I don't understand this statement. rqlite docs are quite clear about the types of transactions it supports. It doesn't support traditional transactions because of the nature of the HTTP API (though that could be addressed).
>Furthermore to this point, you can't even design a consistent system around rqlite alone because you can't use it as a locking service. If I want locks, I end up deploying etcd, consul, or zookeeper anyways.
rqlite is not about allowing developers build consistent systems on top of it. That's not its use case. It's highly-available, fault-tolerant store, the aims for ease-of-use and ease-of-operation -- and aims to do what it does do very well.
>If I had to choose a distributed database with schema support right now for a small scale operation, it would probably be yugabyte or cockroachdb. They're simply better at doing what rqlite is trying to do.
https://rqlite.io/docs/faq/#why-would-i-use-this-versus-some...
Of course, you should always pick the database that meets your needs.
>If building a reliable database was as easy as integrating sqlite with a raft library, I would have shipped nearly 10 years ago.
Who said it was easy? It's taken almost 10 years of programming to get to the level of maturity it's at today.
>They need a more robust design and better safety guarantees than rqlite can offer today.
That is an assertion without any evidence. What are the safety issues with rqlite within the context of its design goals and scope? I would very much like to know so I can address them. Quality is very important to me.
That's clearly a mistaken attitude because both Consul and etcd also use a single "Raft group" and they are production-grade software.
Ruling out a piece of software simply because it doesn't "scale horizontally" (and only writes don't scale horizontally in practice) is a naive attitude.
Can you expand a bit more on your concern? What scenario do you have in mind?
Also rqlite doesn't support seamless downgrades between major versions, only seamless upgrades. I want to communicate that too (I've put a lot of work into the backup-and-restore system[1] so users can protect themselves if they are concerned about the seamless upgrade failing on them).
So by bumping the major version it helps people understand that they are upgrading to a substantially different version of rqlite, even if their client code doesn't have to change at all.