HNHacker News
TopNewBestAskShowJobs

otoolep

5,215 karma · joined April 3, 2014

Software Engineer based in Pittsburgh, PA. Engineering Manager at Google, building large-scale data systems. Creator of rqlite[1], the lightweight, distributed database built on SQLite.

https://www.philipotoole.com

[1] https://www.rqlite.io

submissionscomments
otoolep··on Crsql – Multi-writer and CRDT support for SQLite
Sure, would be happy to consider.
otoolep··on Crsql – Multi-writer and CRDT support for SQLite
No idea.
otoolep··on Crsql – Multi-writer and CRDT support for SQLite
rqlite author here, happy to answer any questions.
otoolep··on Replacing Postgres with rqlite, the lightweight, distributed DB built on SQLite
Many thanks to the team at Replicated[1] -- their feedback and input has made rqlite much better.

[1] https://www.replicated.com/

otoolep··on Ntfy.sh – Send push notifications to your phone via PUT/POST
rqlite[1] author here, happy to answer any questions about it if it helps.

rqlite will give you HA, but won't help with load (rqlite replicates for reliability, not for write performance). But I like to think the it'll give you solid -- and simple to use -- HA. Replicated just replaced their use of Postgres with rqlite for this reason[2].

[1] https://github.com/rqlite/rqlite

[2] https://www.replicated.com/blog/app-manager-with-rqlite/

otoolep··on Embedded programming is like web development
A system made of multiple distinct components, while it can be complicated, is not a distributed system. A SoC is not a distributed system.

For the system you describe to have the practical challenges of a distributed system it would also need experience some of the following:

- some of the "different processors" would need to fail at seemingly random times, only to come back again in a few minutes.

- the speed at which those "different processors" respond would need to vary widely, and different times of the day.

otoolep··on Embedded programming is like web development
Basically agree, yeah.
otoolep··on Embedded programming is like web development
I've written about the same comparison, but came to a somewhat different conclusion. In my experience Embedded Software development was poor preparation for my later distributed-systems work (I wrote Embedded Software for 10 years, then distributed databases and logging analysis systems for the next 10). Embedded software is so deterministic compared to large-scale systems.

https://www.philipotoole.com/a-prayer-for-distributed-system...

"The idea of monitoring the responsiveness of a read-modify-write cycle is obviously nonsense from a programmer’s point of view (though not so for the microprocessor designer!)."

otoolep··on IOx: InfluxData’s New Storage Engine
Congrats to the team at InfluxDB - great to see this released.
otoolep··on Litestream live replication has been moved to the LiteFS project
And benbjohnson wrote the SQL parser[1] rqlite uses to do all this. So you see, the man is everywhere. :-)

[1] https://github.com/rqlite/sql

otoolep··on Litestream live replication has been moved to the LiteFS project
Actually rqlite release 7.7.0[1] adds support for RANDOM(). Timestamp functions support will be added to an upcoming release. It does this by statement-rewriting[2] before sending the SQL to the Raft log.

[1] https://www.philipotoole.com/rqlite-7-7-0-released/

[2] https://github.com/rqlite/rqlite/blob/master/DOC/NON_DETERMI...

otoolep··on SQLite: QEMU All over Again?
As for rqlite.... see this FAQ entry: https://github.com/rqlite/rqlite/blob/master/DOC/FAQ.md#why-...
otoolep··on SQLite: QEMU All over Again?
Nope, as explained in my other comment. It is, as you say, built on top of SQLite.
otoolep··on SQLite: QEMU All over Again?
rqlite[1] author here. I wouldn't consider rqlite a fork in any sense, just so we're clear. That rqlite uses plain vanilla SQLite is one of its key advantages IMHO. Users have no concerns they're not running real SQLite source.

That said, there are some things that would be much easier to do with some changes to the SQLite source. But I think the message that rqlite sits on top of pure SQLite makes is still the right choice.

[1]: https://github.com/rqlite/rqlite

otoolep··on Turning SQLite into a Distributed Database
And good luck with your project, welcome to the distributed SQLite space. :-)
otoolep··on Turning SQLite into a Distributed Database
Can only speak for rqlite: https://github.com/rqlite/rqlite/blob/master/DOC/FAQ.md#why-...
otoolep··on Turning SQLite into a Distributed Database
Correct. What I meant by the reference to Consul is that rqlite uses the Raft implementation at its center and repurposes for use by rqlite.

I can make that clearer.

otoolep··on Turning SQLite into a Distributed Database
Exactly. Providing write-scalability is big deal, don't get me wrong, and this project looks interesting. The biggest misconception folks have about rqlite is that it increases SQLite write performance, when it's about providing fault-tolerance and reliability.

So projects like mvsqlite are solving a real need. I just disagree with some of its definitions.

otoolep··on Turning SQLite into a Distributed Database
rqlite author here.

There are a couple of statements on that page I disagree with.

"mvsqlite is a distributed database, while dqlite and rqlite are replicated databases"

I disagree with this statement, and consider its definition of "distributed system" to be incorrect. rqlite[1] is a distributed database. A "distributed system" is simply a system that splits a problem over multiple machines, solving it in a way that is better, more efficient, possible etc than a single machine. rqlite uses distribution to provide fault-tolerance and high-availability. It uses distributed systems technology i.e. Raft, to make rqlite appear up-and-running, even in the face of node failures. That it replicates a full copy of the SQLite database to every node is correct, but that doesn't mean it's not a distributed system. Is Consul a distributed key-value store? etcd? By the definition quoted on that page they are not, but no one would actually agree with that.

rqlite is not just about replicating a SQLite database. I understand what the page is trying to say, but the point is that rqlite is distributed, just for fault tolerance and high-availability.[2]

"(+): mvsqlite runs on a production-grade distributed key-value store, FoundationDB, instead of implementing its own consensus subsystem."

rqlite doesn't implement its own consensus system either. It uses the same Raft consensus code that powers Hashicorp Consul. I don't see how this is any different, in principle, than using Foundation DB's consensus system.

[1] https://github.com/rqlite/rqlite

[2] https://github.com/rqlite/rqlite/blob/master/DOC/FAQ.md#rqli...

otoolep··on SQLite is not a toy database (2021)
rqlite author here, thanks for the mention, happy to answer any questions.
otoolep··on SQLite is not a toy database (2021)
rqlite author here, happy to answer any questions.
otoolep··on Ask HN: Who do you talk to about system architecture and design?
rqlite creator here, happy to answer any questions about it.
otoolep··on SQLite 3.39.2
Agreed. In the real world actually operating the database is just as important as its intrinsic query functionality.
otoolep··on SQLite 3.39.2
rqlite author here, happy to answer any questions.
otoolep··on SQLite 3 Fiddle
rqlite author here. I'm actually working to address that. But it does require parsing the SQLite statement before it's sent to the Raft log, and replacing the non-deterministic value with the value evaluated on the leader. It's all solvable (in principle).
otoolep··on Let's build a distributed Postgres proof of concept
Every node in the cluster "knows" the network address of the Leader. It knows because a) it's part of the Raft cluster configuration, and b) every follower "heartbeats" to the Leader periodically.

So it's actually pretty simple for a given node to contact the Leader. If a node receives a request which must be performed on the Leader, and that node is not itself the Leader, it can do one of the following things:

1) reject the request with an error, but this isn't really a production-viable option.

2) reject the request with an error, but tell the client where the Leader can be found, so the client can retry the request, this time sending the request to the Leader.

3) transparently forward the request to the leader, wait for the Leader to execute the request, get the response, and return the response to the client. In this case the client doesn't even know the forwarding to the Leader happened.

rqlite supports mode 2 and 3, client can choose which behavior it wants, on a request-by-request basis. Option 3 is the default.

https://github.com/rqlite/rqlite/blob/master/DOC/DATA_API.md...

otoolep··on Let's build a distributed Postgres proof of concept
You can wrap a single request (containing multiple SQL statements) in a transaction, and either all the statements will be successful, or none will. That's what rqlite currently supports.
otoolep··on Let's build a distributed Postgres proof of concept
Yes, embedding any kind of store can work. For example, here is an in-memory KV store embedded in Raft:

https://github.com/otoolep/hraftd

This is kind of a reference use of Hashicorp's Raft.

otoolep··on Let's build a distributed Postgres proof of concept
It's all handled by the Raft libraries (I know this because rqlite uses the same Raft library at this blog post). When integrating with Hashicorp's Raft code, your code must implement Apply()[1]. The Hashicorp code promises not to call Apply() until a quorum of nodes have been persisted the data to the Raft log. That way when your code is called to modify your state machine (FSM), quorum has already been achieved by the Raft system.

Architecturally (in rqlite's case) a node looks something like this: https://docs.google.com/presentation/d/1Q8lQgCaODlecHa2hS-Oe...

[1] https://pkg.go.dev/github.com/hashicorp/raft#FSM

otoolep··on Let's build a distributed Postgres proof of concept
I was half-thinking about taking the Postgres wire protocol code[1] that CockroachDB uses, and adding it to rqlite[2]. Then you could talk to rqlite using Postgres tooling (perhaps). The only thing is that rqlite doesn't support distributed transactions in the usual sense, so there might be some difficulty mapping Postgres transaction control to rqlite. But still, could be interesting.

[1] https://github.com/jackc/pgproto3

[2] https://github.com/rqlite/rqlite

Disclaimer: I am the creator of rqlite.

← PreviousPage 5 of 12Next →