HNHacker News
TopNewBestAskShowJobs

otoolep

5,215 karma · joined April 3, 2014

Software Engineer based in Pittsburgh, PA. Engineering Manager at Google, building large-scale data systems. Creator of rqlite[1], the lightweight, distributed database built on SQLite.

https://www.philipotoole.com

[1] https://www.rqlite.io

submissionscomments
otoolep··on Single dependency stacks
rqlite author here.

It's not quite what rqlite does. rqlite is a distributed database, and it uses SQLite as its storage engine. But it's not a drop-in, distributed, version of SQLite (but it's close, as SQLite is basically fully exposed).

https://github.com/rqlite/rqlite/blob/master/DOC/FAQ.md

otoolep··on Embeddable, replicated and fault tolerant SQL engine based on Sqlite
Yes, that's correct. rqlite does statement-based replication, which means it has the limitation you outlined. This could be solved by parsing the entire SQLite statement passed into rqlite before it's written to the Raft log, but I haven't got around to that yet.
otoolep··on Embeddable, replicated and fault tolerant SQL engine based on Sqlite
rqlite author here.

Good summary, I generally agree. rqlite installation is also very easy, thanks to Go. Single, statically-linked binary you can just drop on your machine. rqlite also has a really simple build process (another explicit goal -- just execute 'go build') making it easy to build it yourself -- giving you a high-level of control over the actual software you run.

https://github.com/rqlite/rqlite/blob/master/CONTRIBUTING.md...

otoolep··on Embeddable, replicated and fault tolerant SQL engine based on Sqlite
rqlite author here. It's not a stupid question. :-) In fact, I answer it head-on in the rqlite FAQ. To quote:

"rqlite is very simple to deploy, run, and manage. It's lightweight. It's a single binary you can drop anywhere on a machine, and just start it. This makes it very convenient. It takes literally seconds to configure and form a cluster. With rqlite you have complete control over your database infrastructure, and the data it stores. That said, it's always possible it's too simple for your needs."

https://github.com/rqlite/rqlite/blob/master/DOC/FAQ.md#why-...

otoolep··on Embeddable, replicated and fault tolerant SQL engine based on Sqlite
rqlite author here, no, they are quite different. To quote the rqlite FAQ:

"dqlite is library, written in C, that you need to Integrate with your own software. That requires programming. rqlite is a standalone application -- it's a full RDBMS (albeit a relatively simple one). rqlite has everything you need to read and write data, and backup, maintain, and monitor the database itself.

rqlite and dqlite are completely separate projects, and rqlite does not use dqlite. In fact, rqlite was created before dqlite."

https://github.com/rqlite/rqlite/blob/master/DOC/FAQ.md#how-...

otoolep··on Docker Is Down
Back up.
otoolep··on Docker Is Down
Second outage in about a week.
otoolep··on Ask HN: What do you use SQLite for?
I use it as the storage engine for rqlite, a distributed relational database.

https://github.com/rqlite/rqlite

otoolep··on Store SQLite in Cloudflare Durable Objects
https://github.com/rqlite/rqlite provides a form of horizontal scaling for reads, but it's not a drop in replacement for SQLite.

https://github.com/rqlite/rqlite/blob/master/DOC/READ_ONLY_N...

(I'm the author of rqlite)

otoolep··on rqlite, the distributed SQLite DB: Designing Autoclustering with Consul and Etcd
I believe using a headless service on Kubernetes is the right way to build node discovery and auto-clustering on that system, but I'm interested in learning about other approaches.
otoolep··on Why are we still tailing log files?
Yeah, agreed, if the logging agent (or whatever) isn't installed or running yet, it's going to be hard to get the logs into the centralized system.

> That requires intermediate storage as logs are generated, which can fill up (disk or ram, it's all finite).

That's also an argument for centralized logging. Don't store the logs anywhere near your production systems. We've all had the experience where a machine goes down because the disk is full.....due to log files.

otoolep··on Why are we still tailing log files?
Exactly, I didn't mention Elastic either, another hugely important player.
otoolep··on Why are we still tailing log files?
What is about your deployment or setup that you don't use your log search system instead? Assuming you have one? You could search all your logs files at once (for errors, for strack traces), instead of manually tailing each one, one after the other. Is that you don't even know what to search for?
otoolep··on Why are we still tailing log files?
Well, now that I look it, maybe it's not exactly a tailing tool. Interesting nonetheless.
otoolep··on Why are we still tailing log files?
Ha, hadn't seen lnav before. Looks interesting. Proves my point, tailing will never die. :-)
otoolep··on Why are we still tailing log files?
I agree about the centralized logging -- it's my last point. My bigger question is why is anyone tailing log files anymore when the industry has built such a huge number of tools. It's curious, but tailing still has its place, at least emotionally.
otoolep··on Chiselstore: an embeddable, distributed SQLite for Rust
Cheers, glad you like it -- great to hear.
otoolep··on Chiselstore: an embeddable, distributed SQLite for Rust
Yes, that's exactly how rqlite works. If I ever get around to adding proper transaction support it will have to overcome the challenge you identify. I believe it can be done, but it will require significant changes to the current design. But an explicit goal for rqlite has always been to do a certain set of things very well, before doing something new. So I'm cautious.

Systems like Chiselstore and dqlite operate at a different architectural layer, so can offer transaction support. But require more programming to use. It's a trade off.

otoolep··on Chiselstore: an embeddable, distributed SQLite for Rust
Sure, but rqlite tries to give you the best of both worlds -- a distributed database that is also lightweight and super-easy to run.
otoolep··on Chiselstore: an embeddable, distributed SQLite for Rust
Yeah, I need to get around to creating a Go sql.DB layer. Been too busy adding to the database itself. :-) Soon!
otoolep··on Chiselstore: an embeddable, distributed SQLite for Rust
rqlite author here, happy to answer any questions about it. It seems some folks have questions about its design, and the design of systems that combine SQLite and Raft in general. Checking out the rqlite design docs might be helpful:

https://github.com/rqlite/rqlite/blob/master/DOC/DESIGN.md

otoolep··on The Session Extension
What inconsistencies do you mean, precisely? I think you need to provide a clear example of what you mean, to understand what you're getting at.

In my experience, any inconsistencies would be due to misuse of Raft, not something intrinsic to Raft itself.

otoolep··on The Session Extension
rqlite creator here.

rqlite (and Raft) doesn't assume this. All it states is that for the cluster to make progress i.e. apply a change in a consistent manner, at least a quorum of nodes must be online and in contact with the Leader (which is one of the online nodes). Every node in a rqlite cluster knows what size the cluster is and therefore requires that (N/2)+1 nodes acknowledge the change before that change is committed. The Leader node performs this coordination.

The scenario you outlined above is obviously possible. But in that event the cluster is down -- the Leader (let's say it's node A) cannot contact a quorum of nodes. No changes can be made to it.

otoolep··on Mentoring junior programmers: What I tell them is job number one
Yeah, thanks, silly typo -- fixed.
otoolep··on Durable Objects now Generally Available
rqlite author here, happy to answer any questions about it.
otoolep··on Ask HN: Companies of one, what is your tech stack?
rqlite author here, happy to answer questions about it. I should note that rqlite is a distributed database that uses SQLite as its database engine. This means in practice it does replicate a SQLite database to each node on the rqlite cluster -- but it's not a SQLite replication system per-se.

https://github.com/rqlite/rqlite/blob/master/DOC/FAQ.md

otoolep··on IceFireDB: Distributed disk storage database based on Raft and Redis protocol
rqlite author here, happy to answer any questions.
otoolep··on State of netbooting Raspberry Pi in 2021
rqlite author here, happy to answer any questions about it.

https://github.com/rqlite/rqlite

otoolep··on The database ruins all good ideas
rqlite author here. Happy to answer any questions about it.
otoolep··on Do you really need Redis? How to get away with just PostgreSQL
rqlite author here. Happy to answer any questions about it.

https://github.com/rqlite/rqlite

← PreviousPage 7 of 12Next →