Ah thank you! (and for posting!)
That’s the dream. To serve the world’s transactions and data (with a whole lotta beetles!). We’re working to make it reality.
Appreciate the congrats! You make today a double whammy! :P
5,021 karma · joined November 18, 2009
https://github.com/tigerbeetle/tigerbeetle
joran@tigerbeetle.com
That’s the dream. To serve the world’s transactions and data (with a whole lotta beetles!). We’re working to make it reality.
Appreciate the congrats! You make today a double whammy! :P
We still write, read (and have an independent engineer review) each line of code by hand.
We go faster like that, but, most of all, it’s the guarantee we make to our users, also to continue to invest in our own understanding, because second order that’s valuable for the kind of high performance safety work we do.
Long term, I’m sure LLMs will improve, but right now they’re just not there.
I'd check out tigerstyle.dev, pick up Zig, and then make an HTTP server or file format parser. Those are great ways to learn and experience this kind of programming. At some point, you start to realize that it's just easier to build API services in this way.
But for sure, you can learn so much in JVM-based languages. They make you appreciate low-level techniques all the more!
But if your application then creates another transfer against the client, and another, while the first request is inflight, then the client will autobatch under the hood and send these off as a batch when the first request returns.
You get this sweetspot then between latency and throughput. And your latency is not spiking as your load increases, since your throughput is now able to keep up.
For example, if you take a look at our LSM compaction, regardless of the table size, we compact at the 512 KiB block granularity, and everything is streaming.
The same principle applies everywhere.
In our experience writing TigerStyle (and for all our internal code and tooling, not only TB as DBMS), we’ve never had a scenario where static allocation was not applicable or didn’t produce a better design.
You also tend to become more memory efficient, not less. Again, since you’re streaming. (You’re not allocating a massive buffer, just because a file is multi-GiB.)
I did correct you where you are wrong (“appeal to popularity” as a logical fallacy).
> I can't help and wonder where you could be by now if your language was lifting you up, instead of you having to lift up your language.
Did you know we’ve had on the order of 3 memory bugs in 6 years of TigerBeetle?
We also reached production in 3.5 years, bringing not only a global consensus implementation, but also a local storage engine to market. (Each of these typically take 5-10 years elsewhere to reach maturity).
Zig does lift us up.
In fact, Zig’s memory model has always been the perfect expression of TigerStyle. And TB could not have been designed the way it is today in any other language (including Rust). Implicit allocation, global allocator… you automatically lose OOM-safety. But the zero-copy intrusive memory techniques we use… Zig is perfect for TigerBeetle.
> I wonder if Tigerbeetle will also have problems arguing for their solution now that the other project they can point to for customer assurance is gone.
In general, we never like to appeal to popularity (a logical fallacy), but why would you assume here that we would point to Bun specifically (or any project for that matter) [1] as an example of Zig’s quality?
We prefer to judge Zig’s quality on its own intrinsic merit:
For example, we subject the language through TigerBeetle to inordinate amounts of fuzzing, perhaps more than any other language (you could say Zig is lucky to have TB’s test suite aimed against it!).
Literally 1,024 dedicated CPU cores, 24/7.
Zig holds up remarkably well.
We also recently pledged $512K to the ZSF, together with Synadia.
These are the kinds of things we prefer to point to. Not hype, but real end-to-end systems engineering, and long term financial support, regardless of the language we choose to use.
[1] I picked Zig back in July 2020. At the time, the largest project was River, but already Zig was a phenomenal choice, and the years have only shown that Zig was probably one of the best design decisions in the development of TigerBeetle. It turned out better than I imagined.
In general, we never like to appeal to popularity (a logical fallacy), but why would you assume here that we would point to Bun specifically (or any project for that matter) [1] as an example of Zig’s quality?
We prefer to judge Zig’s quality on its own intrinsic merit:
For example, we subject the language through TigerBeetle to inordinate amounts of fuzzing, perhaps more than any other language (you could say Zig is lucky to have TB’s test suite aimed against it!).
Literally 1,024 dedicated CPU cores, 24/7.
Zig holds up remarkably well.
We also recently pledged $512K to the ZSF, together with Synadia.
These are the kinds of things we prefer to point to. Not hype, but real end-to-end systems engineering, and long term financial support, regardless of the language we choose to use.
[1] I picked Zig back in July 2020. At the time, the largest project was River, but already Zig was a phenomenal choice, and the years have only shown that Zig was probably one of the best design decisions in the development of TigerBeetle. It turned out better than I imagined.
However, TB here is also providing a Replicated State Machine with consensus and strict serializability, in front of object storage, to provide remote object storage capacity and recovery, but with local NVMe latency and without sacrificing consistency or durability.
TB navigates the entire design space, specializing for both hot and cold (transactional) data.
The more you zoom, it’s a stronger set of guarantees in terms of safety and performance.
This was a team effort: the object storage connector, the scale test, the visualization, the slides, even provisioning the hardware had its challenges!
I don’t think we reviewed your Go benchmarking code at the time—and that there were no technical critiques probably should not have been taken as explicit sign off.
IIRC we were more concerned at the deeper conceptual misunderstanding, that one could “roll your own” TB over PG with safety/performance parity, and that this would somehow be better than just using open source TB, hence the discussion focused on that.
In other words, to count not only the money changing hands, but also the corresponding goods/services being exchanged.
These are all transactions: goods/services and the corresponding money.
I searched the recent history of our community Slack but it seems it may have been an older conversation.
We typically do code review work only for our customers so I’m not sure if there was some misunderstanding.
Perhaps the assumption that because we didn’t say anything when you pasted the code, therefore we must have reviewed the code?
Per my other comment, your benchmarking environment is also a factor. For example, were you running on EBS?
These are all things that our team would typically work with you on to accelerate you, so that you get it right the first time!
For example, if you have 8K transactions through 2 accounts, a naive system might read the 2 accounts, update their balances, then write the 2 accounts… for all 8K (!) transactions.
Whereas TB does vectorized concurrency control: read the 2 accounts, update them 8K times, write the 2 accounts.
This is why stored procedures only get you typically about a 10x win, you don’t see the same 1000x as with TB, especially at power law contention.
We’d love to roll up our sleeves and help you get it right. Please drop me an email.
Do you not think if someone can remember those four words, they’re less likely to get it wrong?
If you could contribute some better suggestions we could consider them!
Or, can you suggest something pithier?
(This is similar also to how chain replication preserves consistency.)