HNHacker News
TopNewBestAskShowJobs

otoolep

5,215 karma · joined April 3, 2014

Software Engineer based in Pittsburgh, PA. Engineering Manager at Google, building large-scale data systems. Creator of rqlite[1], the lightweight, distributed database built on SQLite.

https://www.philipotoole.com

[1] https://www.rqlite.io

submissionscomments
otoolep··on Latest Chat-GPT4 release (May 12th 2023) sure is buggy
Yes, it's saved me hours of time -- with certain narrow tasks. For example, it generated these files for me:

https://github.com/rqlite/rqlite/blob/master/auto/backup/sum*

It took a few minutes, might have taken me 1+ hour of some research, and mechanically typing it all in.

otoolep··on Latest Chat-GPT4 release (May 12th 2023) sure is buggy
Could be, but I've been using 4 up until tonight without a single problem. 4 was magic until this evening, TBH.

Maybe if it continues I'll do as you suggest.

otoolep··on Latest Chat-GPT4 release (May 12th 2023) sure is buggy
The release prior to this one was quite good, no issues at all. Very noticeable problems in the last few hours.
otoolep··on Ask HN: It's 2023, how do you choose between MySQL and Postgres?
rqlite[1] creator here, happy to answer any questions.

[1] https://www.rqlite.io

otoolep··on Why US presidents skip British coronations
I thought it would be something more interesting than "never did it, don't intend to start now".
otoolep··on Show HN: rqlite 7.15, the distributed DB built on SQLite, now with backups to S3
To quote from the rqlite FAQ[1]:

"Why would I use this, versus some other distributed database?

rqlite is very simple to deploy, run, and manage – in fact, simplicity-of-operation is a key design goal. It’s also lightweight and easy to query. It’s a single binary you can drop anywhere on a machine, and just start it, which makes it very convenient. It takes literally seconds to configure and form a cluster, which provides you with fault-tolerance and high-availability. With rqlite you have complete control over your database infrastructure, and the data it stores.

That said, it’s always possible it’s too simple for your needs."

[1]: https://rqlite.io/docs/faq/#why-would-i-use-this-versus-some...

otoolep··on Show HN: rqlite 7.15, the distributed DB built on SQLite, now with backups to S3
rqlite creator here.

Yes, you still need the data at every node. rqlite is not about edge-compute specifically (though many solutions have come along after rqlite was created to address that need, I guess I should have responded to that demand much earlier instead :-)). One of my primary goals with rqlite has always been ease-of-deployment and ease-of-use. I wanted to build a quality distributed database that was both really reliable and super easy-to-use.

Check out the FAQ, which might answer some more of your questions: https://rqlite.io/docs/faq/

otoolep··on Tyler Cowen Interviews Jonathan “GPT” Swift
"In this conversation, Tyler uses ChatGPT to interview Jonathan Swift about his views on religion, politics, economics, and literature."

The GPT transcript is spoken by someone with an Irish accent (sounds like it's from Northern Ireland). It's something else to hear it.

otoolep··on AI could replace equivalent of 300M jobs – report
Suitably dramatic image headlines the piece. :-)
otoolep··on We updated our RSA SSH host key
No, it wouldn’t be - assuming by “insane” you mean “silly to do”. I build systems at Google that do exactly that.

Whether it’s worth the cost is a decision each company makes. Also, you don’t need to keep the log forever. Max of a few weeks retention would be common.

otoolep··on Strong Consistency with Raft and SQLite
I added a FAQ entry which explains a little bit more what you can do:

https://rqlite.io/docs/faq/#how-can-i-change-my-multi-node-c...

otoolep··on Strong Consistency with Raft and SQLite
Yeah, going from 3->1 simply by taking down nodes won't work -- that's breaking quorum. If you really want to do this, you can explicitly override the configuration stored in Raft. Check out these directions:

https://rqlite.io/docs/clustering/general-guidelines/#recove...

While this talks about "dealing with failure" using the config file described above basically allows you to force the cluster into any configuration you want (including a single-node cluster).

otoolep··on Strong Consistency with Raft and SQLite
A single node should work perfectly fine when it's the only node in the cluster i.e. you're running a single-node cluster. If you couldn't bring up a single node from scratch, you're doing something wrong. :-) Show me your launch command, or join the rqlite Slack channel for help (https://www.philipotoole.com/join-rqlite-slack)

However if you create a 3-node cluster (or add 2 nodes to an existing single-node cluster) then at least a quorum of nodes (N/2 +1) must be up and running for the cluster to be available. That's the way Raft works -- Cockroach, etcd, Consul -- all of them will work the same way.

So in your case a 3-node cluster with 2 nodes subsequently going down (or deliberately taken offline) is a cluster which is offline.

There is extensive documentation available on clustering, which you should review:

https://rqlite.io/docs/clustering/

otoolep··on Strong Consistency with Raft and SQLite
This is my take: https://rqlite.io/docs/faq/#how-is-it-different-than-litestr...

From the Litestream page: https://litestream.io/alternatives/#raft-based-consensus

otoolep··on Strong Consistency with Raft and SQLite
rqlite creator here. It's not related to my work, best of luck to them.

[1] https://rqlite.io

otoolep··on Strong Consistency with Raft and SQLite
rqlite[1] creator here. This offering is completely new to me. Welcome to the party, that's what I say. I started doing what they are doing about 10 years ago. :-)

https://www.philipotoole.com/9-years-of-open-source-database...

https://www.philipotoole.com/replicating-sqlite-using-raft-c...

https://www.philipotoole.com/rqlite-v3-0-1-globally-replicat...

The post makes a fairly big point of the shortcomings of statement-based replication (like what rqlite does).

What they state is correct -- non-deterministic SQL will be a problem, but it's not a difficult problem to solve. Just parse and rewrite the SQL statement before sending it to Raft. It's exactly what rqlite does[2] (though there is more to do). However they claim that there are two concerns with this approach.

>This solution introduces a runtime delay because each SQL statement must be parsed and analyzed.

The delay is insignificant compared with the delay introduced by Raft consensus. This is not a valid concern. I'm surprised they even mentioned it.

>More importantly, it increases the difficulty of staying up-to-date with the new versions of the underline SQL engine (SQLite in this specific case) because the custom parser must be updated each time a new SQL function/feature is introduced.

Theoretically somewhat true, but not true in practice. Any competent programmer can update any parsing code in a matter of hours. Also just because a new feature is introduced does not mean that the parser needs to be rewritten. That's because SQLite will not introduce breaking changes into their SQL specification. Any parser needs to be updated only if you want to support any new, non-deterministic feature.

[1] https://rqlite.io

[2] https://rqlite.io/docs/api/non-deterministic/

otoolep··on What Technologies Are You Future Proofing With?
rqlite[1] author here. I may not follow your point, but one of rqlite's design goals is simplicity-of-operation. I like to think it's one of the easiest-to-operate distributed databases out there.

[1]https://www.rqlite.io

otoolep··on HC-tree is an experimental high-concurrency database back end for SQLite
I will say the promise of "improved concurrency" is very interesting to me. If nothing else, I would definitely pull that into rqlite (probably as an option). That said, working with Raft log has always been the real bottleneck, not SQLite.
otoolep··on HC-tree is an experimental high-concurrency database back end for SQLite
rqlite[1] author here. It's very interesting see this. Not sure whether it can bring more functionality to rqlite, but it's definitely cool to see the SQLite database evolve with more client-server applications in mind.

I will say that one misconception people often have about rqlite is that it's a drop-in replacement for SQLite. You see that a bit even in the comments section here. That was never its goal. Its goal is a lightweight and super-simple to use distributed relational database. Using SQLite as the database engine was the perfect way to meet that requirement.

[1] https://www.rqlite.io

otoolep··on LiteSync – Easy synchronization of SQLite databases
PostgREST is new to me. Interesting things in common, now that I see it, but much that is not.
otoolep··on LiteSync – Easy synchronization of SQLite databases
rqlite author here -- happy to answer any questions. While rqlite[1] does replicate a SQLite database, it's not simply a replication system (just to be clear). It's a distributed database that uses SQLite as its database engine, and replicates for fault tolerance. That said, it mostly exposes SQLite via its HTTP API, so can work for many SQLite applications[2].

[1] https://rqlite.io

[2] https://rqlite.io/docs/faq/#is-it-a-drop-in-replacement-for-...

otoolep··on Ask HN: Recommendation for Management Books
High Output Management by Andy Grove.

The Effective Executive by Peter Drucker.

[1] https://books.google.com/books/about/High_Output_Management....

otoolep··on SQLite-loadable-rs: A framework for building SQLite Extensions in Rust
rqlite[1] author here, happy to answer any questions.

rqlite offers horizontal and vertical real scalability, but only vertical for writes. See https://rqlite.io/docs/guides/performance/

[1] https://rqlite.io/

otoolep··on I found a bug in SQLite
Glad you enjoyed it -- thanks.
otoolep··on I found a bug in SQLite
rqlite creator here.

I'm not developing libraries, I'm developing an entire RDBMS. In my experience -- and this is broader than rqlite -- integration and end-to-end tests seem like they are great - at the start. But as you rely more and more on them they become costly to maintain and really hard to debug. A single test failure (often due to a flaky, timing-sensitive issue) means wading through layers and layers of code to identify the issue.

Overly relying on integration and end-to-end testing (note I said over-reliance, there is absolutely a need for them) becomes exponentially more costly over time (measured in development velocity and time to debug) as the test suite grows. If you find you're having difficulty identifying a place for them it may that you're not decomposing your software properly in the first place. All this is probably manageable if you're a solo developer, but when a team is building the software it can become really painful.

For more details see the talk I gave to CMU[1] on my testing strategy.

[1]https://youtu.be/JLlIAWjvHxM?t=2067

otoolep··on I found a bug in SQLite
https://quoteinvestigator.com/2015/03/02/eureka-funny/
otoolep··on I found a bug in SQLite
I may not be following your point, but I don't believe the root cause of this issue was a race condition, or anything that might be related to concurrency.

The fundamental cause AFAIK was a SQLite connection was attempting to make a state transition (from one type of locking state to another) which shouldn't be allowed under certain circumstances, but the implementation didn't actually enforce this rule. So the added test really does test the root cause.

otoolep··on I found a bug in SQLite
Actually, they added a unit test in a follow-on change:

https://www.sqlite.org/src/info/dc7dd2d3e50e7cc4

otoolep··on I found a bug in SQLite
rqlite[1] author here. To be clear rqlite is using SQLite in a completely conventional manner. Nothing about the distributed nature of rqlite impacts on SQLite, since each rqlite node runs its own complete copy of SQLite.

This bug can affect anybody using an in-memory version of a SQLite database. That was the point of writing the C unit test.

[1] https://github.com/rqlite/rqlite

otoolep··on Go and SQLite in the Cloud
rqlite author here, happy to answer any questions. The rqlite FAQ[1] might be useful to you.

[1] https://github.com/rqlite/rqlite/blob/master/DOC/FAQ.md

← PreviousPage 4 of 12Next →